The project was originated to build a unique toolset that will increase the common database performance by the technically adjusted flow of large volume data extraction, processing, and its further uploading to the data warehouse as well as simplify the reporting process.
of the database productivity increase reached.
More than 500
projects are now being processed at the same time.
The solution we have provided in terms of this project had to fix a few problems. First, we had to make processing hundreds of projects simultaneously real. Second, the work of managers on one project should be synchronized and more efficient. Third, the entire data processing flow should take time as short as possible. It took us 6 months to create the system that would optimize all these actions and processes to increase overall business efficiency profit for ETL Software.
The main goal of this project was to fix lack of automation in the data gathering, processing and delivering. The core of the problem was the necessity to manually extract information and statistics from many independent databases and sources. In addition to the exel files manual extraction, the company had to use Python for writing a code that will help to set up the data collection and convert this data into final reports for the company's clients. Managers had to deal with the same data each time, but they had to make adjusted uploading and configurations for each case. Our task was to create a unique system that would be easy and convenient to use, and, most important, that will enhance the general efficiency.
In terms of this project, we had to create fully-functional and convenient system that will automatically collect, process, and deliver data in a form of structured and easy to read reports within minutes. Using Django, DRF and additional technologies, we’ve managed to write a code that allows the system to extract data from multiple sources and transform it into hundreds of excel files with the understandable information. The system had to be of a significant capacity to give our client the ability to provide up to 500 different reports with a freshly collected data simultaneously.
We had created a highly effective and convenient system that allows configuring all the data required into one wholesome report using the variety of data blocks, thus, managers can create reports and send them to clients fast and easy. With this system it takes just a few hours to gather the information from different databases to create a report instead of 10 days as before. Also, this system allows working on 500 projects simultaneously, while in the past only 1 project can be processed at a time. And the last, but not least, the velocity of creating reports had vastly increased, because now managers can synchronize their actions on one project, and don't have to do their work one after another as before.
I was excited working with the Light IT guys on the development of our ETL software from scratch. I came to them with just an idea and they helped to bring it to life. After the project was successfully completed it was extended with additional features and is being supported properly.