How to scrape a specific table form website using python (beautifulsoup4 and requests or any other library)?

https://en.wikipedia.org/wiki/Economy_of_the_European_Union

Above is the link to website and I want to scrape table: Fortune top 10 E.U. corporations by revenue (2016).

Fortune top 10 E.U. corporations by revenue (2016)

Please, share the code for the same:

import requests
from bs4 import BeautifulSoup

def web_crawler(url):

page = requests.get(url)
plain_text = page.text
soup = BeautifulSoup(plain_text,"html.parser")
tables = soup.findAll("tbody")[1]
print(tables)

soup = web_crawler("https://en.wikipedia.org/wiki/Economy_of_the_European_Union")


2 Answers
2

following what @FanMan said , this is simple code to help you get started, keep in mind that you will need to clean it and also perform the rest of the work on your own.

import requests
from bs4 import BeautifulSoup
url='https://en.wikipedia.org/wiki/Economy_of_the_European_Union'
r=requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
temp_datastore=list()
for text in soup.findAll('p'):
w=text.findAll(text=True)
if(len(w)>0):
temp_datastore.append(w)


Some documentation

beautiful soup:https://www.crummy.com/software/BeautifulSoup/bs4/doc/

requests: http://docs.python-requests.org/en/master/user/intro/

urllib: https://docs.python.org/2/library/urllib.html

You're first issue is that your url is not properly defined. After that you need to find the table to extract and it's class. In this case the class was "wikitable" and it was a the first table. I have started your code for you so it gives you the extracted data from the table. Web-scraping is good to learn but if your are just starting to program, practice with some simpler stuff first.

import requests
from bs4 import BeautifulSoup

def webcrawler():

url = "https://en.wikipedia.org/wiki/Economy_of_the_European_Union"
page = requests.get(url)
soup = BeautifulSoup(page.text,"html.parser")
tables = soup.findAll("table", class_='wikitable')[0]
print(tables)

webcrawler()


Thanks for contributing an answer to Stack Overflow!

But avoid …

To learn more, see our tips on writing great answers.

Required, but never shown

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.