How to download web pages rendered by JavaScript

For Web pages that the final contents are rendered by JavaScript, using requests will not be able to get the final web page. This is a code snip using Selenium to do the job.

from selenium import webdriver
import chromedriver_autoinstaller

url='URL of web page rendered with JavaScript'

driver_directory = chromedriver_autoinstaller.install() ## This needs to run as local admin

options = webdriver.ChromeOptions()

## hide the web browser in background
options.add_argument("headless") 

browser = webdriver.Chrome(driver_directory, options=options)
browser.get(url)

## Wait for a few seconds for JAVAScript to finish rendering
time.sleep(5) 

## Find the elements you are interested in and process them
belement = browser.find_element_by_xpath('/html/body')

browser.quit()

Back To Top