The main goal was to improve the existing prototype of the framework. Eventually, the prototype was completely redeveloped. Our team used Selenium to improve the collection of information from SPA (single-page application) stores.
IntexSoft specialists have added an API to force the product information update and display status information about crawlers/scrapers.
We managed to get the most out of the framework by scaling through processes and using proxies.
It was essential to implement the storage of all collected information in Azure Blob Storage, which was done successfully. Besides, IntexSoft specialists have added telemetry to Azure App Insights.
While working on the project, 38 crawlers and scrapers of online stores were updated and implemented.
The work was carried out using an iterative methodology. We arranged daily meetings with the client, where IntexSoft specialists demonstrated reports on completed tasks and resolved issues.