Looping through pages always gets the same result
1
I'm trying to loop through pages and save a specific image
import urllib.request
from bs4 import BeautifulSoup as bs
frontstring = 'http://www.haz.de/'
for i in range(1, 50):
url = 'http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/'
+ 'Digitales-Daumenkino-So-waechst-das-Parkhaus#p' + str(i)
with urllib.request.urlopen(url) as page:
soup = bs(page)
galleryimage = soup.findAll('img', 'class': 'pda-fullgallery-large photo')
for imgtag in galleryimage:
try:
imgurl = frontstring + imgtag['src']
imgname = 'img/fullgallery-large' + str(i) + '.jpg'
urllib.request.urlretrieve(imgurl, imgname)
print('saving image from ' + imgurl + ' to ' + imgname)
except Exception as e:
raise
else:
pass
However the image is always the same. I don't know where it went wrong. If I open the url in the browser it's the correct page and image but the soup seems to be always the same code. Probably something really stupid and simple but I'm not seeing it after trying really long to find the mistake.
python python-3.x http beautifulsoup urllib
edited Nov 13 '18 at 5:53
Cœur
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
add a comment |
1
I'm trying to loop through pages and save a specific image
import urllib.request
from bs4 import BeautifulSoup as bs
frontstring = 'http://www.haz.de/'
for i in range(1, 50):
url = 'http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/'
+ 'Digitales-Daumenkino-So-waechst-das-Parkhaus#p' + str(i)
with urllib.request.urlopen(url) as page:
soup = bs(page)
galleryimage = soup.findAll('img', 'class': 'pda-fullgallery-large photo')
for imgtag in galleryimage:
try:
imgurl = frontstring + imgtag['src']
imgname = 'img/fullgallery-large' + str(i) + '.jpg'
urllib.request.urlretrieve(imgurl, imgname)
print('saving image from ' + imgurl + ' to ' + imgname)
except Exception as e:
raise
else:
pass
However the image is always the same. I don't know where it went wrong. If I open the url in the browser it's the correct page and image but the soup seems to be always the same code. Probably something really stupid and simple but I'm not seeing it after trying really long to find the mistake.
python python-3.x http beautifulsoup urllib
edited Nov 13 '18 at 5:53
Cœur
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20
add a comment |
1
1
1
I'm trying to loop through pages and save a specific image
import urllib.request
from bs4 import BeautifulSoup as bs
frontstring = 'http://www.haz.de/'
for i in range(1, 50):
url = 'http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/'
+ 'Digitales-Daumenkino-So-waechst-das-Parkhaus#p' + str(i)
with urllib.request.urlopen(url) as page:
soup = bs(page)
galleryimage = soup.findAll('img', 'class': 'pda-fullgallery-large photo')
for imgtag in galleryimage:
try:
imgurl = frontstring + imgtag['src']
imgname = 'img/fullgallery-large' + str(i) + '.jpg'
urllib.request.urlretrieve(imgurl, imgname)
print('saving image from ' + imgurl + ' to ' + imgname)
except Exception as e:
raise
else:
pass
However the image is always the same. I don't know where it went wrong. If I open the url in the browser it's the correct page and image but the soup seems to be always the same code. Probably something really stupid and simple but I'm not seeing it after trying really long to find the mistake.
python python-3.x http beautifulsoup urllib
edited Nov 13 '18 at 5:53
Cœur
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
I'm trying to loop through pages and save a specific image
import urllib.request
from bs4 import BeautifulSoup as bs
frontstring = 'http://www.haz.de/'
for i in range(1, 50):
url = 'http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/'
+ 'Digitales-Daumenkino-So-waechst-das-Parkhaus#p' + str(i)
with urllib.request.urlopen(url) as page:
soup = bs(page)
galleryimage = soup.findAll('img', 'class': 'pda-fullgallery-large photo')
for imgtag in galleryimage:
try:
imgurl = frontstring + imgtag['src']
imgname = 'img/fullgallery-large' + str(i) + '.jpg'
urllib.request.urlretrieve(imgurl, imgname)
print('saving image from ' + imgurl + ' to ' + imgname)
except Exception as e:
raise
else:
pass
However the image is always the same. I don't know where it went wrong. If I open the url in the browser it's the correct page and image but the soup seems to be always the same code. Probably something really stupid and simple but I'm not seeing it after trying really long to find the mistake.
python python-3.x http beautifulsoup urllib
python python-3.x http beautifulsoup urllib
edited Nov 13 '18 at 5:53
Cœur
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
edited Nov 13 '18 at 5:53
Cœur
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
edited Nov 13 '18 at 5:53
Cœur
19k9112154
edited Nov 13 '18 at 5:53
Cœur
19k9112154
edited Nov 13 '18 at 5:53
Cœur
19k9112154
19k9112154
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
asked Jan 5 '17 at 19:07
sezi80sezi80
112111
112111
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20
add a comment |
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20
add a comment |
1 Answer
1
active
oldest
votes
1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/2
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/3
Those are the real urls, the url you seen is generated by JavaScript. You should disable JavaScrip before you scraping any site
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f41492785%2flooping-through-pages-always-gets-the-same-result%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/2
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/3
Those are the real urls, the url you seen is generated by JavaScript. You should disable JavaScrip before you scraping any site
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
add a comment |
1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/2
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/3
Those are the real urls, the url you seen is generated by JavaScript. You should disable JavaScrip before you scraping any site
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
add a comment |
1
1
1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/2
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/3
Those are the real urls, the url you seen is generated by JavaScript. You should disable JavaScrip before you scraping any site
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/1
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/2
http://www.haz.de/Hannover/Aus-der-Region/Lehrte/Fotostrecken/Digitales-Daumenkino-So-waechst-das-Parkhaus/(offset)/3
Those are the real urls, the url you seen is generated by JavaScript. You should disable JavaScrip before you scraping any site
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
answered Jan 6 '17 at 1:07
宏杰李宏杰李
8,80921123
8,80921123
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f41492785%2flooping-through-pages-always-gets-the-same-result%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Hi, you can't download images using python code because image links generated by javascript code. HTML code contains only one link to the same image. Try selenium for example to emulate browser
– sr3z
Jan 5 '17 at 19:59
I don't think I fully unuderstand what you mean. #p1 is another image than #p1 or 3, 4, 5 ... 50. I get 50 different pages and then the image. At least that's the idea.
– sezi80
Jan 5 '17 at 20:16
right but if you open html code for every page you'll find the same image url only, not different as you expect. javascript updates url for each page. but BeautifulSoup can not execute javascript code that is why you save the same image on each page
– sr3z
Jan 5 '17 at 20:20