ETL software service on monitors

ETL Software

The project was originated to build a unique toolset that will simplify the reporting process and increase the database general performance by the technically adjusted flow of big data extraction, processing, and uploading it to the database.

  • Industry - Data Mining & Processing
  • Services - Web Development / IT Consulting
  • Main Technologies - Angular / Django / Python

Precisely 162%

of the database productivity increase reached.

More than 500

projects can be processed simultaneously.

Faster & Easier Reporting

The solution we have provided in terms of this project aimed at fixing few problems. First, we had to make the system process hundreds of projects simultaneously. Second, the work of managers on one project should be synchronized and more efficient. Third, the time for the entire data processing flow should be as short as possible. It took us 6 months to create the system that would optimize all these actions and processes to increase the overall business efficiency profit for ETL Software.

Introduction to the Problem

The main goal of this project was to fix lack of automation in the data gathering, processing, and delivering. The core of the problem was the necessity to manually extract information and statistics from many independent databases and data sources. In addition to the excel files manual extraction optimization, the company used Python for writing code that would help to set up the data collection and convert this data into final reports for the company's clients. Managers had to deal with the same data each time, but they had to make adjusted uploading and configurations for each case. Our task was to create a unique system that would be easy and convenient to use, and most important, will enhance the general efficiency of database performance and reporting.

Challenges & Solutions

Collecting and Reporting within Minutes

We were asked to build a fully-functional and convenient system that can automatically collect, process, and deliver data in a form of structured and easy to read reports within minutes. Using Django, DRF and other technologies, we’ve managed to write code that allows the system to extract data from multiple sources and transform it into hundreds of excel files with the understandable information. The system should have high capacity and give our client the ability to simultaneously provide up to 500 different reports with a freshly collected data.

And, as a result…

We created a highly effective and convenient system that allows configuring all the data required into one wholesome report using the variety of data blocks; thus, managers can create reports and send them to clients fast and easily. With this system it takes just a few hours to gather the information from different databases to create a report instead of 10 days as before. Also, this system allows working on 500 projects simultaneously, while in the past only 1 project can be processed at a time. And last but not least, the speed of creating reports increased significantly, because the system allows managers to synchronize their actions on one project instead of doing their work consequently as before.

Team & Time

  • 1 PM
  • 1 TL
  • 1 QA
  • 2 DEVs
  • 4 Months

Technologies

David

ETL Software

I was excited working with the Light IT guys on the development of our ETL software from scratch. I came to them with just an idea and they helped to bring it to life. After the project was successfully completed it was extended with additional features and is being supported properly.

Make your idea a profitable reality!