Script fails to keep clicking on load more button
Script fails to keep clicking on load more button
I've written a script in Python in association with selenium to keep clicking on MORE button to load more items until there are no new items left to load from a webpage. However, my below script can click once on that MORE button available on the bottom of that page.
MORE
MORE
Link to that site
This is my try so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://angel.co/companies?company_types=Startup&company_types=Private+Company&company_types=Mobile+App&locations=1688-United+States"
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
print(elems.get_attribute("href"))
try:
loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
driver.execute_script("arguments[0].scrollIntoView();", loadmore)
loadmore.click()
except Exception:break
driver.quit()
How can I keep clicking on that MORE button until there are no such button left to click and parse the links as I've already tried using for loop.
MORE
for loop
.more
If I use
.more instead of [class='more'], the loop will be infinite and will never break.– SIM
Sep 7 '18 at 8:02
.more
[class='more']
2 Answers
2
I've managed to solve the problem pursuing sir Andersson's logic within my exising script. This is what the modified script look like.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://angel.co/companies?company_types=Startup&company_types=Private+Company&company_types=Mobile+App&locations=1688-United+States"
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)
while True:
try:
loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"[class='more']")))
driver.execute_script("arguments[0].click();", loadmore)
wait.until(EC.staleness_of(loadmore))
except Exception:break
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))):
print(elems.get_attribute("href"))
driver.quit()
Just move
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))): print(elems.get_attribute("href")) out of while loop and execute it in the end of script– Andersson
Sep 6 '18 at 10:10
for elems in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".results .name a.startup-link"))): print(elems.get_attribute("href"))
while
Yep, It did the trick perfectly @sir Andersson. I would gladly take out my answer and accept yours if you post one. Thanks.
– SIM
Sep 7 '18 at 8:11
It's not some kind of major improvement, so there is no need in new answer. You can just update your own solution
– Andersson
Sep 7 '18 at 8:15
why not just?
while (driver.FindElements(By.ClassName("more")).Count > 0)
driver.FindElement(By.ClassName("more")).Click();
//Some delay to wait lazyload to complete
c# example. pretty sure that it can be done with python as well
Thanks for contributing an answer to Stack Overflow!
But avoid …
To learn more, see our tips on writing great answers.
Required, but never shown
Required, but never shown
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
Remove try catch block you provide error you're getting. Change visibility wait to clickable and try again. Also change your locator to
.more- just simplier– Sers
Sep 3 '18 at 22:15