Part one: Data Scraping and Cleaning
Scrape the reviews from bestbuy.com, using Selenium in Python
Step one: Import packages & modules
Step two: Set up the Chrome WebDriver
WebDriver is an open-source tool for automated testing of web apps across many browsers. Without that, it is not possible to execute Selenium test scripts in Google Chrome as well as automate any web application. After executing my code, a google browser will be opened and controlled by python.
Step three: Scrape data from bestbuy.com
I first created a new list "reviews_one_store" to store all the data I scraped from the website. In the for loop, I created a dictionary "one_review" to store raw HTML, review text, review date, and review star. Then I append the dictionary to my list. My code will scrape one page at a time, and stop when will the pages are scraped. The total number of reviews I collected is 5054.
Step four: Clean data
Scraped data may not be as decent as we want. We need to make sure that the data is pretty clean and organized for later analysis.
Step five: Export to Excel
Part of my data looks like:
