UtiliStats

Our client wanted to give Sweden an advanced platform that would help people to optimize their energy consumption; thus, save natural resources and their money at the same time. The system is presented as a website where everyone dealing with public-service utilities can purchase data on weather conditions for specified, location, year, day, hour, and many other parameters. Based on the selected parameters, the platform provides users with a report that includes air temperature, precipitation, wind speed, etc. Our Big Data processing and cloud solutions made the system fully automated, so new user validation, data scraping, processing, reporting, and emailing users don’t involve manual work.

The platform is monetized with the subscription business model. The quality and the extensiveness of weather reports ensure a high client retention rate because both private and governmental companies that deal with providing utilities to citizens will always need such data to plan their budget on natural energy resources purchasing. Also, the clients f the platform can use this data to see whether the need for energy among citizens has dropped or increased.

To meet the client’s Big Data requirements, we used Python programming language with its stack of libraries and frameworks. We made the platform from scratch, and to optimize its functioning, we separated the online store where users can purchase required data on weather and API that collects data from public sources and puts in into the cloud database (Amazon S3 Bucket). HTTP request processing is implemented with Django REST, Redis, Celery, and other technologies. To ensure good-quality and fast Big Data processing, we split large requests into smaller tasks and used AWS Lambda for each small task. One Lambda function is responsible for getting, processing, and saving data to the database on one date, which eliminates timeouts and many other system malfunctions. We also used the ElasticSearch tool Kibana to track possible inconsistencies and malfunctions of the data scraping process. The website is written with Django Oscar because this framework is perfect for eCommerce stores.