About the project
Around early 2021 in Indonesia, a nation-wide vaccination program was started, and institutions and companies in Indonesia are encouraged by the government to register their employees/people to the vaccination program.
By using this method, the people inside the institutions or companies are automatically registered to the national vaccination program, the goal is to make vaccination easier, faster, and more widespread.
Challenges to solve
Collecting and managing hundreds of thousands to millions of data on a nationwide scale is a challenge.
During the development we were faced by two important challenges involving huge amount of data, which are:
a. How to process and validate vaccine candidate uploaded data when each file could contain upto one million data?
b. How to maintain the website loading speed when it’s processing 20+ million data for most of the time?
Solutions
First of all, the client chose the Extended Team collaboration method, and our developer helped to develop the Content Management System (CMS) using PHP and Laravel. This CMS was the core of the vaccination program, a hub where institutions uploaded their data and where the client analyzed and managed the data.
To solve the first challenge, we chose to create an upload queue system. When multiple files—which can contain one million of data each—are uploaded, the system puts them into a queue, so the system can process and validate each uploaded file easier.
The result is an efficient big data processing system and less constraint for the databases and CMS.
Another extra solution we chose is a log activity system, it is used to record and monitor users activity and help the tracing process in case errors happen.
For the CMS loading speed problem of challenge number two, other than optimizing the data query processing we put various limitations on how the data is displayed, and one example is to limit the data loaded and displayed in each page.
When the admin only needs the first 10.000 data out of 20 million, the system only loads and displays that 10.000 data (up to 100.000 data), thus decreasing the CMS loading speed compared to if the system loads all 20 million data at once.
Project summary
Closing
Being able to process and validate hundreds of thousands of data everyday is definitely one strong point of this software.
In the first month after release, we approximately processed and validated 1-2 million vaccination candidate data, and after that the data just increased rapidly.
Extended Team collaboration method and Web Apps development service is just one of the combinations you can choose when working together with us.
Follow or contact us to know more about our collaboration methods, expertise, and other case studies.
