Highly Loaded Data System

The system functions as data storage and a billing system for all clients’ spendings and actions.

Challenge

The telecom company had an outdated billing system that could no longer cope with some critical tasks and meet modern requirements. So, they decided to design a new system from scratch. One of the main goals of the system is to store data on all the consumptions and actions of clients. Since the product is also a billing system, it also serves to process and calculate data.

This is a Big Data project. A key feature of the system is the storage and processing of large data arrays. The scale of the system can be compared with Spotify, Yandex.Music, and so on. The system processes billions of files per day, and receives approximately 600 thousand records per second.

Technologies

Team Composition

Team Leader
4 Developers
4 QA Specialists

600K

records per second

20

microservices

4

years project duration

Solution

In the conditions of such data volumes, the distributed Cassandra database was used as storage. Several data centers located in different regions were used to ensure data integrity and load balancing.

As for billing, the system is not able to issue invoices itself, but it processes the data that is used for this purpose. For processing and calculating the distributed data, the development team used Apache Spark.

The system functions as data storage and a billing system for all clients’ spendings and actions.. In addition to the fact that all data is stored in a distributed manner, it is processed and stored using microservices. Currently, there are about 20 microservices within the system.

To ensure system security, the SFTP protocol, Single Sign-on (SSO) technology, and data encryption are used.

To ensure system security, the SFTP protocol, Single Sign-on (SSO) technology, and data encryption are used.

Methodology & Transparency

At the initial stage, the project was developed according to the Scrum methodology with two-week sprints. At the end of each sprint, the whole dev team including business analysts discussed the demo versions. Once a month the development team presented demo versions directly to the client.

During the project, the team was growing. For convenience, a special Slack chat was created for daily meetings. Also, once a week the dev team arranged a general Zoom meeting in order to discuss both the current situation of the project and news in the IT-sphere.

At the stage of active project implementation, the development team switched to the Kanban methodology to optimize work processes.

Features

Information management

Data collection, processing and aggregation, data reports
Calculations

Accounting of payments and expenses, recalculation of expenses for issuing a new invoice based on the data updated

Feedback from the client

“We needed specialists with solid teamwork skills and enterprise-level app development expertise. Choosing from an enormous number of positive aspects, we’ve been especially impressed by IntexSoft specialists’ professionalism, responsibility, and flexibility. When circumstances called for deadlines to be shifted, IntexSoft’s specialists adjusted and continued to work overtime.The main result is that we’ve agreed to embark on a long-term cooperation with IntexSoft. They always make valuable contributions to our projects.”

Procurement Manager form the Client’s side