Python web scraping query

I have written my first ever python code to scrape a dividend history table from the web but soup.select statement doesn't seem to select anything and gives rise to an index error.

Any advice on how to resolve please?

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

driver = webdriver.Chrome(executable_path='F:PythonAppsChromeDriverChromeDriver.exe')
driver.get("https://www.dividendchannel.com/history/?symbol=ibm")
soup = BeautifulSoup(driver.page_source,"lxml")
driver.quit()
table = soup.select("table#Dividend History")[0]
print(table)
list_row =[[tab_d.text.strip().replace("n","") for tab_d in
item.select('th,td')] for item in table.select('tr')]

for data in list_row[:2]:
print(' '.join(data))


File "F:/System/Python/dividend.py", line 9, in
table = soup.select("table#Dividend History")[0]

IndexError: list index out of range

"table#Dividend History"

"div#divvytable"

2 Answers
2

this is not a direct answer, but a recommendation. Depending on what you need it for, the website you have referenced has a limited usage based on IP, only can be accessed 6 times.
Take a look at the dividend api which is FREE(not advertising)->
IEX API

If you choose to use it, it might make your application that much more efficient. It is much easier playing with JSON data then converting to dataframe(PANDAS) or post to a front end via JavaScript.

here is a sample call for apply for last 5 years->

https://api.iextrading.com/1.0/stock/aapl/dividends/5y

You would use requests.get(url, params).json() and traverse it through a simple for loop.

It seems that layout of this page is based on tables, lots of tables. Your code is trying to find table with id of "Dividend", which does not exist.

Here is your code after some tweaks. It finds the rows with data, and then extracts data from the rows:

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

driver = webdriver.Chrome()
driver.get("https://www.dividendchannel.com/history/?symbol=ibm")

soup = BeautifulSoup(driver.page_source,"lxml")
driver.quit()

dividend_rows = soup.select("div#divvytable")[0].find_all("tr")

for row in dividend_rows:
columns = list(row.stripped_strings)
if len(columns) != 2:
continue
print("date: amount: ".format(columns[0], columns[1]))


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.