The Controversial practice of web scraping

What's the most important thing you need to do Data Analysis? That's right... you guessed it. It's DATA! If your answer was somehow different, maybe you should go back to the bases, pause, and re-start the journey. Again... Data Analysis is not a sprint... it's a marathon.

Now... going back to data, the first problem we are going to face is: where do we get data from? Yes, it's true, there are plenty of data sources around (Kaggle.com is my go-to website when it comes to get some "ready to use" data), but, especially when you want to get insights about real world problems that are not "mainstream", you need to dig a little deeper and, sometimes, you may need to gather data by yourself.

When that happens, one thing I think it's very useful is WEB SCRAPING. With Web Scraping we refer to the practice of analysing automatically the information on a website to copy them in a table or dataframe and be utilised for other purposes. There are many tools that help you doing that, some of them are free. I like to use a Python package called BeautifulSoup that, after an initial phase of familiarisation, it's super easy to use once you interpret correctly the website you are trying to scrape. It gives me the ability to get only the info I need and, if I do it in a smart way, I can re-use the same code for many other pages with, maybe, only few small tweaks and adjustments.

I personally think this practice can give you a lot of advantages, especially when you want to gather many data from a website like Amazon, Ebay, AirBnB. This simply because if you were to copy few info from a website into a table you were better off doing it manually as you would not need to code, review he code for bugs, check applicability of your code in the whole website, etc. My last web scraping exercise has been done on the website www.immobiliare.it as I wanted to determine what a fair rental price could have been for an apartment in a certain area, with certain features in the current post-Covid scenario in Florence. (more on that later... stay tuned)

In that case, the process was complex as the scraping had to be done for a specific listing, then repeated for all the listings in a page and then repeated for all the pages in the search results. It worked beautifully to get the initial data even though, as it always happen when real world examples are taken, data need a lot of cleansing before being used and analysed.

On the web there are a lot of different opinions bout web scraping. Some people say it is (or it should be) illegal, some others state that it is perfectly legal and some others are, in between, and say that it's"immoral'. Let me make this clear. I think it is absolutely legal and there is no reason for it not to be. You are not breaking into anyone's home, you are not stealing any information. You are just collecting some info about something that you could have searched and it is on that specific website for that specific reason.. .being easily reachable by everyone in the world. Look at it this way... if you can take notes of these information on a piece of paper, than you may as well scrape it from the web. You are just doing it more efficiently, incredibly faster, and you are eliminating the human factor from the equation.

To do it efficiently you need time and practice but I think it's absolutely worth spending some time mastering this practice. You will be inspired to take on the next challenge as you won't need anymore to worry about how to get the data, but only about what do I need o get out of them.

How do I use it (or plan to use it) in my daily job? Well... very simple. When I focus on a specific Account/Company or a specific Project, I can scrape the web to get some information automatically and report only the links that report useful information so I won't have to read all the articles and the news that may be repetitive or, worse, useless. Of course this is not only web scraping, but web scraping is the beginning.

What do you think? Do you think Web Scraping can be useful for your daily job too? Let me know your thoughts.