Our development team built a Big Data processing platform that provides companies and governmental institutions with comprehensive information on weather conditions from 1974 to the present day to calculate the current need in utilities for one building or the entire city.
Big Data processing platform 1

Big Data Solution to Optimize Energy Consumption

Our client wanted to give Sweden an advanced platform that would help people to optimize their energy consumption; thus, save natural resources and their money at the same time. The system is presented as a website where everyone dealing with public-service utilities can purchase data on weather conditions for specified, location, year, day, hour, and many other parameters. Based on the selected parameters, the platform provides users with a report that includes air temperature, precipitation, wind speed, etc. Our Big Data processing and cloud solutions made the system fully automated, so new user validation, data scraping, processing, reporting, and emailing users don’t involve manual work.
Big Data processing platform 2

Advantages for Business

The platform is monetized with the subscription business model. The quality and the extensiveness of weather reports ensure a high client retention rate because both private and governmental companies that deal with providing utilities to citizens will always need such data to plan their budget on natural energy resources purchasing. Also, the clients f the platform can use this data to see whether the need for energy among citizens has dropped or increased.

1974 – Till Present

The platform provides detailed information on this period and offers to filter it by location, year, day, hour, temperature, precipitation, wind speed, precipitation, etc.

1 Lambda Function for 1 Day

This approach ensures immediate processing of small requests and three days maximum to process the largest ones.

Highly Beneficial for 3 Industries

Fintech, Entertainment, Travel

Technical Implementation

To meet the client’s Big Data requirements, we used Python programming language with its stack of libraries and frameworks. We made the platform from scratch, and to optimize its functioning, we separated the online store where users can purchase required data on weather and API that collects data from public sources and puts in into the cloud database (Amazon S3 Bucket). HTTP request processing is implemented with Django REST, Redis, Celery, and other technologies. To ensure good-quality and fast Big Data processing, we split large requests into smaller tasks and used AWS Lambda for each small task. One Lambda function is responsible for getting, processing, and saving data to the database on one date, which eliminates timeouts and many other system malfunctions. We also used the ElasticSearch tool Kibana to track possible inconsistencies and malfunctions of the data scraping process. The website is written with Django Oscar because this framework is perfect for eCommerce stores.
Statista indicates that the global Big Data and business analytics market is forecast to grow to $274.3B by 2022. Become a business giant with our Big Data, BI, and cloud solutions!

Why You Need a Similar Solution

Such Big Data platforms are highly useful for the analytical and statistical needs of large enterprises and smaller companies. Depending on your industry and market, you can apply a similar solution to track changes in customer behavior, see the history of trends to predict the upcoming ones, and many others. This platform deals with weather conditions, but your system can collect, process, and save data on sports, cryptocurrencies, stock trading, vehicle sales, and many more.
Want to know what we can do for you?
Technologies used:
Have a project in mind?Book a free consultation with tech experts
White checkmark in a blue box icon Your privacy is protected
Expertise proven by
ISO 9001:2015 certification badgeThe Professional Scrum Master certification badgeISO 27001 certification badgeCisco certification badgeIBM professional certification badgeAWS certification badgeISTQB testing certification badgeAgile fundamentals (ICP) certification badge
Light IT Global's Clutch badge
4.9 - 36 Reviews
Light IT Global's GoodFirms badge
4.8 - 19 Reviews