Download failed while using quantmod function getSymbols in For loop
Download failed while using quantmod function getSymbols in For loop
I am trying to get data for the 1632 stocks that comprise the NSE index in India using the quantmod package. I am able to download stocks individually; however, when I loop over all the stocks, I am getting a timeout. How do I loop the getSymbols function to download the desire data?
quantmod
getSymbols
Following error is reported:
Error: '20MICRONS.NS' download failed after two attempts. Error message:
HTTP error 404.
5.
stop(Symbols.name, " download failed after two attempts. Error",
" message:n", attr(dl, "condition")$message, call. = FALSE)
4.
getSymbols.yahoo(Symbols = "'20MICRONS.NS'", env = ,
verbose = FALSE, warnings = TRUE, auto.assign = TRUE)
3.
do.call(paste("getSymbols.", symbol.source, sep = ""), list(Symbols = current.symbols,
env = env, verbose = verbose, warnings = warnings, auto.assign = auto.assign,
...))
2.
getSymbols(as.character(x), src = "yahoo")
1.
f(Symbol[i])
MyData <- read.csv(file="C:/Documents/EQUITY_L.csv", header=TRUE)
Symbol <- MyData$SYMBOL
f <- function(x) getSymbols(as.character(x), src='yahoo')
for (i in 1:1632) f(Symbol[i])
20MICRONS.NS
quantmod::getSymbols("20MICRONS.NS")
Sys.sleep(2)
1 Answer
1
Ok, now I see...
First download Symbols from:
https://www.nseindia.com/corporates/content/securities_info.htm
It's the first file listed on the page.
It appears each symbol from the NSE file needs to add an ".NS" suffix. That's why you can download the equities individually, but it fails when you pass the Symbol column of your file to getSymbols.
getSymbols
I'd also create a new environment to dump every stock into and keep your global environment manageable.
Finally, pass NSE_Symbols to quantmods getSymbols function for daily data.
I Like to use sapply coupled with try so that if you hit a bad symbol,
the HTTP error 404 won't halt the remaining symbols from downloading.
NSE_Symbols
getSymbols
sapply
try
HTTP error 404
EQUITY_L <- read.csv("~/R/stack-overflow/data/EQUITY_L.csv", stringsAsFactors = FALSE)
NSE_Symbols <- paste0(EQUITY_L$SYMBOL,".NS")
NSE_stocks <- new.env()
library(quantmod)
sapply(NSE_Symbols, function(x)try(getSymbols(x, env=NSE_stocks), silent=TRUE))
Next, test and find out which symbols didn't download. I was able to get all but 17.
length(NSE_Symbols[!(NSE_Symbols %in% names(NSE_stocks))])
[1] 17
NSE_Symbols[!(NSE_Symbols %in% names(NSE_stocks))]
[1] "3PLAND.NS" "BHAGYANGR.NS" "CHEMFAB.NS" "ELECTROSL.NS" "GANGESSECU.NS"
[6] "GMMPFAUDLR.NS" "GUJRAFFIA.NS" "HBSL.NS" "KALYANI.NS" "MAGADSUGAR.NS"
[11] "MANAKCOAT.NS" "MCDOWELL-N.NS" "NIRAJISPAT.NS" "PALASHSECU.NS" "SIGIND.NS"
[16]"SPTL.NS" "SUBCAPCITY.NS"
The symbols which downloaded succesfully will be neatly contained in the NSE_stocks environment.
NSE_stocks
Good luck,
rquantmod
Bonus: If you just need the closing price, you can bring the securities back into the global environment as single xts object with this:
NSE_xts <- do.call(merge, eapply(NSE_stocks, Ad))– Justin
Sep 12 '18 at 0:54
NSE_xts <- do.call(merge, eapply(NSE_stocks, Ad))
@JustinThank you but the code is not serving the purpose. I get the same missing Symbols as you do,however, this is only a string and not the actual xts object. I edited the paste function to NSE_Symbols <- paste0("'",MyData$SYMBOL,".NS'") and still it is not working.
– Catool
Sep 12 '18 at 15:27
If you get the same missing symbols then that means you downloaded all equities except 17, which is what the string reports. The rest of the equities are in the
NSE_stocks environment. See my comment above your on how to bring them into your global environment.– Justin
Sep 12 '18 at 15:37
NSE_stocks
Thank you very much. Yes! the variable NSE_xts holds the data for all the closing prices. I am new to coding and the environment and do.call definitely add value.
– Catool
Sep 12 '18 at 20:37
Yes do.call and environments are a good combination.
do.call especially is very useful, I'd study the documentation for that in depth, as well as that of sapply. In addition, search stack overflow for other examples which use those functions. Good luck!– Justin
Sep 12 '18 at 21:06
do.call
sapply
Thanks for contributing an answer to Stack Overflow!
But avoid …
To learn more, see our tips on writing great answers.
Required, but never shown
Required, but never shown
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
The triple quote around
20MICRONS.NSin the error message looks suspicious. I had no problem runningquantmod::getSymbols("20MICRONS.NS")on my machine just now. If you think you a pause between downloads would help, you can add a two second pause in each iteration of your loop withSys.sleep(2). To dig in further, I would need to see what's in the "Equity_L.csv" file.– Dan Y
Sep 11 '18 at 3:11