From the course: Using Python for Automation

Unlock this course with a free trial

Join today to access over 24,800 courses taught by industry experts.

Prepare for scraping multiple pages

Prepare for scraping multiple pages - Python Tutorial

From the course: Using Python for Automation

Start my 1-month free trial Buy for my team

Prepare for scraping multiple pages

“

- Alright, data enthusiasts. It's time to challenge yourself to create a scraper that efficiently navigates and extracts information from multiple pages. Many websites are structured with pages that divide information, and you might need to scrape more than one page to get the data you want. In this lesson, you'll examine a website with opportunities for multi-page web scraping, and plan a technical workflow for scraping data from multiple pages. This contains the first page of results when browsing for bestselling books in English from this public catalog. It shows the first 50 books out of 3,477 total. Now, replace the page=1 with page=2 in the URL. You can now view the second page of the results. The URL structure is consistent, so if you wanted to scrape multiple pages, here's how you would approach that. Create a for loop to iterate over a range of the page numbers that you want to scrape from. Inside the for loop, define the URL to be the first part of this URL, which is…

Contents