Dfyjkt

I'm trying to scrape https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro and i want to retrieve every paint store in every district from every state. That is a dynamic element on the website, and i can't get any data out of it.

You have to select a State from a dropdown list, and then a district from another dropdown list. Then you have to submit the options with the button down bellow. After all that the data is displayed in a list beside a "map" (that it doesn't apear, don't know why). I need to get that data. But i have no idea what to do. The code i have at this moment is:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Chrome()
driver.get('https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro')

provincias = driver.find_elements_by_xpath('//*[@id="ddlState"]/option')
provinciaSelect = Select(driver.find_element_by_xpath('//*[@id="ddlState"]'))
localidadSelect = Select(driver.find_element_by_xpath('//*[@id="ddlLocation"]'))
driver.implicitly_wait(2)

buscarPinturerias = driver.find_element_by_css_selector('#search_paint_shops')

for i in range(1,len(provincias)):
 provinciaSelect.select_by_index(i)
 driver.implicitly_wait(2)
 for j in range(1, len(localidadSelect.options)):
 localidadSelect.select_by_index(j)
 print(i,j)

I am getting this error

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

That's how far i could get. I know i have to Submit the result and wait until the data apear. I don't know how to do that

UPDATE

I've updated the code to this:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('')
driver.get('https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro')

# all options of the dropdown list
provincias = driver.find_elements_by_xpath('//*[@id="ddlState"]/option')
provinciaSelect = Select(driver.find_element_by_xpath('//*[@id="ddlState"]'))
localidadSelect = Select(driver.find_element_by_xpath('//*[@id="ddlLocation"]'))
driver.implicitly_wait(2)

buscarPinturerias = driver.find_element_by_css_selector('#search_paint_shops')

for i in range(1, len(provincias)):

 provinciaSelect.select_by_index(i)
 elemLoc = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="ddlLocation"]/option[2]' )))
 # to know in which state i am
 print(i)

 for j in range(1, len(localidadSelect.options)):

 localidadSelect.select_by_index(j)
 buscarPinturerias.submit()
 elemRes = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="map_info"]/div' )))
 resBlock = driver.find_element_by_xpath('//*[@id="map_info"]/div')
 dataBlock = resBlock.find_elements_by_tag_name('li')

 for l in dataBlock:
 # I have the error here now
 print(l.text.encode('utf-8'))

Now the problem is that if i make

print(l.text.encode('utf-8'))

it raises a

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

is it because i don't wait enough?