2023
First, I researched conventions and best-practices within web scraping. Most web scrape applications use python, for example in combination with Jupyter notebook and Beautifulsoup. Since I will have to handle Python a lot in my internship and for final work, this was the perfect opportunity for me to gain some experience in this as well and become more comfortable with Python's syntax and working environment. I did the non-web scraping stuff in PHP Laravel, which I already had experience with. Within this, however, I also experimented by using a new library for Laravel; Livewire Volt. Since the python web scraper functionality is completely distinguishable from the rest, it was also a good opportunity to apply Docker; among other things, I learned more about partitioning an application into different containers. I also tried to build the python web scraper functionality as a microservice, so that the web scraper could also function as an API within the docker network. In the end, I still had to integrate the python functionality into the Laravel application itself so that it could more easily use the output from the web scraping endpoints. One of the technical problems was scrapping lazyloaded data. To solve this problem, I was introduced to concepts such as XHR and some browser tools that make analyzing the bn incoming data easier.
