python Webscraping

Writing a web scraper with python is easy, if you allow yourself a little help by two beautiful libraries

  • BeautifulSoup
  • Requests

If you look at it in reality, all the good stuff is happening from line 15 to 21

On line 15+16 we let Requests makes it easy to get the content of our request (in this case html)

On line 17 we let BeauitfulSoup make things super easy.  BeautifulSoup(html)  transforms all that content into something that is easy to parse and the rest is just that, using find functions to find what we want and going through it.

It truly couldn’t have been easier and you can find good documentation on BeautifulSoup here

Please note that BeautifulSoup 4 has a lot going for it and you should investigate it, in lieu of BeautifulSoup 3…

You Might Also Like