When I was trying to scrape a Javascript heavy website with my Raspberry using Python, I ran into some interesting issues that needed to be solved.<\/p>\n
I found that modules like request,request_html, urlllib did not deliver the complete content with Javascripts websites containing shadow-dom (#shadowroot). When searching for solution i found some, like the use of PhantomJS or other discontinued modules.<\/p>\n
The solution I found was using Chromedriver in headless mode. But the version I got my hands on kept throwing errors on the version of the browser.<\/p>\n
After extensive searches I found the solution in:<\/p>\n
https:\/\/github.com\/electron\/electron\/releases<\/a><\/p>\n (get the arvmv7 version)<\/p>\n https:\/\/www.raspberrypi.org\/forums\/viewtopic.php?t=194176<\/a><\/p>\n In your code add these two arguments, when you start the driver: When trying to execute the script I still got the error on Chromium version.I was able to solve that using:<\/p>\n IT WORKS<\/p>\n now the script finally worked<\/p>\n from selenium import webdriver # Define the site to be opened # Set Chrome Options With the content of the page in driver it is possible to further decompose the page.<\/p>\n content1= driver.find_element_by_tag_name(‘…..’) To get access to the shadow element the function below needs to be used:<\/p>\n # function to expand a shadow element to useable content When I was trying to scrape a Javascript heavy website with my Raspberry using Python, I ran into some interesting issues that needed to be solved. I found that modules like request,request_html, urlllib did not deliver the complete content with Javascripts websites containing shadow-dom (#shadowroot). When searching for solution i found some, like the use … Continue reading “Scrape Javascript heavy website on RaspberryPi3B+ using Python with Selenium”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[71,72,13],"tags":[73,53],"class_list":["post-1276","post","type-post","status-publish","format-standard","hentry","category-programming","category-python","category-raspberry","tag-python","tag-raspberry"],"_links":{"self":[{"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/posts\/1276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1276"}],"version-history":[{"count":8,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/posts\/1276\/revisions"}],"predecessor-version":[{"id":1287,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=\/wp\/v2\/posts\/1276\/revisions\/1287"}],"wp:attachment":[{"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/edwinmichielsen.nl\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}2. Install this using the instructions i found on:<\/h3>\n
\n
\nsudo apt-get install libminizip1
\nsudo apt-get install libwebpmux2<\/li>\n
\n-headless
\n-disable-gpu<\/p>\n3 Update the Chromium bowser<\/h3>\n
\n
The Python Script to get the page content<\/h3>\n
\nimport time
\nfrom selenium.webdriver.common.by import By
\nfrom selenium.webdriver.support.ui import WebDriverWait
\nfrom selenium.webdriver.support import expected_conditions as EC
\nfrom selenium.webdriver.common.keys import Keys
\nfrom selenium.webdriver.chrome.options import Options<\/p>\n
\nsite = “http:\/\/….”<\/p>\n
\nchrome_options = Options()
\nchrome_options.add_argument(“–headless”)
\n# Open Chrome Headless
\ndriver = webdriver.Chrome(chrome_options=chrome_options)
\ndriver.set_page_load_timeout(20)
\ndriver.get(site)<\/p>\n4. Analyze the content of the page<\/h3>\n
\nshadow_content1 = expand_shadow_element(content1)<\/p>\n
\ndef expand_shadow_element(element):
\nshadow_root = driver.execute_script(‘return arguments[0].shadowRoot’, element)
\nreturn shadow_root<\/p>\n","protected":false},"excerpt":{"rendered":"