Yahoo Historical Data - Did they change the URL recently?

Discussion in 'Educational Resources' started by Rationalize, May 16, 2017.

  1. Ryan81

    Ryan81

    +1 to everything freedinner said.

    I used to keep multiple Yahoo Finance browser tabs open throughout the day, ever day. It was my go-to source for information about a company. No longer. I get info from Google Finance now.

    Every time they've changed things, they sacrificed 1) usability, 2) accuracy, 3) page loading speed

    But hey, at least it now adheres to "material design" principles. LOL.
     
    #21     May 17, 2017
    d08 likes this.
  2. d08

    d08

    Things started really going downhill with Mayer so this was expected.
    Few things to consider with the new csv download source:

    1. It requires cookies to be sent.
    2. OHL are adjusted for splits but not for dividends. None of the prices are adjusted for dividends, even the "adjusted close" field. This is completely different from the previous format when the adjusted close field was adjusted for everything and other fields were unadjusted. This means you have to do dividend adjustments yourself. Very confusing.
    3. When downloading, you get sudden injections of false data - I got "hi" multiple times in the middle of data randomly.
    4. Quite a few symbols have huge holes with data completely missing or certain dates having no information (data that is present on Google Finance).

    To me it seems Yahoo has hired new people and they don't exactly know what they're doing OR this is a deliberate act but in my opinion, short-sighted.
     
    #22     May 17, 2017
  3. d08

    d08

    No download required. Python and Pandas Market Calendars. But yes, there is a learning curve.

    Exactly, I have relatively poor internet speeds during peak hours and the page loading times are ridiculously slow. It's better than a few months ago when you could have lunch while the page loaded (Google Finance took a few seconds) but this is still bad.
     
    #23     May 17, 2017
  4. bellman

    bellman

    So, what is the best source for getting splits and dividends now???

    I don't mind paying for it, although free is nice obviously.

    Google, unfortunately, only provides the adjusted prices (and adjusted volume?). An ideal provider would have the dividends, splits, and the daily un-adjusted OHLC.

    I am comfortable scraping data from any format. Although pulling a delimited file is much nicer.
     
    #24     May 17, 2017
  5. +1 The simplest and most flexible format is surprisingly hard to find
     
    #25     May 17, 2017
  6. d08

    d08

    Yahoo still offers this, they just retired their old servers it seems. They have splits and dividends as well. Google isn't adjusted for dividends and they don't even provide data for it.
     
    #26     May 17, 2017
    gkishot likes this.
  7. Using R, even if you copy the exact link with crumb, e.g.

    https://query1.finance.yahoo.com/v7/finance/download/MSFT?period1=1463461200&period2=1494910800&interval=1d&events=history&crumb=XBOjLoYE5PS

    or, at R cmd line,
    SPY <- read.csv("https://query1.finance.yahoo.com/v7...&interval=1d&events=history&crumb=XBOjLoYE5PS")

    *Full command is truncated here, just mouse over and read bottom of browser.



    Yahoo server, still returns,

    https:// URLs are not supported by the default method: using "wininet"
    Error in file(file, "rt") : cannot open the connection
    In addition: Warning message:
    In file(file, "rt") : cannot open: HTTP status was '401 Unauthorized'

    even though the popup downloader is available.
     
    Last edited: May 17, 2017
    #27     May 17, 2017
  8. I said it earlier and this is just my opinion. If you want to still use yahoo for OHLC data (I personally will not) then the solution is to data scrape the html. The OHLC is there,
    no need to mess around with cookies. After you jump over that hurdle you then face the issue of possible floating point errors due to how they save the OHLC now. As d08 said, you also face data accuracy issues for ex. dividend adjustments..ect..ect...

    The data was once in descending date order. Now I think you would need to do a date sort to get the data right. I don't think the JSON has the dates ascending or descending. It might actually be jumbled. I would need to take a closer look but the investment in time is not worth it with yahoo. They might change their minds next week and come up with new URI structures....waste of time IMHO

    {"date":1494855000,"open":68.13999938964844,"high":68.4800033569336,"low":67.56999969482422,"close":68.43000030517578,"volume":31530300,"unadjclose":68.43000030517578},{"date":1494595800,"open":68.61000061035156,"high":68.61000061035156,"low":68.04000091552734,"close":68.37999725341797,"volume":18714100,"unadjclose":68.37999725341797},{"date":1494509400,"open":68.36000061035156,"high":68.7300033569336,"low":68.12000274658203,"close":68.45999908447266,"volume":28789400,"unadjclose":68.45999908447266},{"date":1494423000,"open":68.98999786376953,"high":69.55999755859375,"low":68.91999816894531,"close":69.30999755859375,"volume":17977800,"unadjclose":69.30999755859375},{"date":1494336600,"open":68.86000061035156,"high":69.27999877929688,"low":68.68000030517578,"close":69.04000091552734,"volume":22858400,"unadjclose":69.04000091552734},{"date":1494250200,"open":68.97000122070312,"high":69.05000305175781,"low":68.41999816894531,"close":68.94000244140625,"volume":18566100,"unadjclose":68.94000244140625},{"date":1493991000,"open":68.9000015258789,"high":69.02999877929688,"low":68.48999786376953,"close":69,"volume":19128800,"unadjclose":69},{"date":1493904600,"open":69.02999877929688,"high":69.08000183105469,"low":68.63999938964844,"close":68.80999755859375,"volume":21749400,"unadjclose":68.80999755859375},{"date":1493818200,"open":69.37999725341797,"high":69.37999725341797,"low":68.70999908447266,"close":69.08000183105469,"volume":28928000,"unadjclose":69.08000183105469},{"date":1493731800,"open":69.70999908447266,"high":69.70999908447266,"low":69.12999725341797,"close":69.30000305175781,"volume":23906100,"unadjclose":69.30000305175781},{"date":1493645400,"open":68.68000030517578,"high":69.55000305175781,"low":68.5,"close":69.41000366210938,"volume":31954400,"unadjclose":69.41000366210938},{"date":1493386200,"open":68.91000366210938,"high":69.13999938964844,"low":67.69000244140625,"close":68.45999908447266,"volume":39548800,"unadjclose":68.45999908447266},{"date":1493299800,"open":68.1500015258789,"high":68.37999725341797,"low":67.58000183105469,"close":68.2699966430664,"volume":34971000,"unadjclose":68.2699966430664},{"date":1493213400,"open":68.08000183105469,"high":68.30999755859375,"low":67.62000274658203,"close":67.83000183105469,"volume":26190800,"unadjclose":67.83000183105469},{"date":1493127000,"open":67.9000015258789,"high":68.04000091552734,"low":67.5999984741211,"close":67.91999816894531,"volume":30242700,"unadjclose":67.91999816894531},{"date":1493040600,"open":67.4800033569336,"high":67.66000366210938,"low":67.0999984741211,"close":67.52999877929688,"volume":29770000,"unadjclose":67.52999877929688},{"date":1492781400,"open":65.66999816894531,"high":66.69999694824219,"low":65.44999694824219,"close":66.4000015258789,"volume":32522600,"unadjclose":66.4000015258789},{"date":1492695000,"open":65.45999908447266,"high":65.75,"low":65.13999938964844,"close":65.5,"volume":22299500,"unadjclose":65.5},{"date":1492608600,"open":65.6500015258789,"high":65.75,"low":64.88999938964844,"close":65.04000091552734,"volume":26992800,"unadjclose":65.04000091552734},{"date":1492522200,"open":65.33000183105469,"high":65.70999908447266,"low":65.16000366210938,"close":65.38999938964844,"volume":15155600,"unadjclose":65.38999938964844},{"date":1492435800,"open":65.04000091552734,"high":65.48999786376953,"low":65.01000213623047,"close":65.4800033569336,"volume":16689300,"unadjclose":65.4800033569336},{"date":1492090200,"open":65.29000091552734,"high":65.86000061035156,"low":64.94999694824219,"close":64.94999694824219,"volume":17896500,"unadjclose":64.94999694824219},
     
    Last edited: May 17, 2017
    #28     May 17, 2017
  9. truetype

    truetype

    I'm sure they'll miss your business.
     
    #29     May 17, 2017
    jharmon likes this.
  10. LOL speaking of people being missed, where the hell is your buddy Surf?
     
    #30     May 17, 2017