I retrieved and formatted the data with a combination of - wget, bash (including other shell commands), and perl to get S&P 500 symbols from wikipedia.org - curl, bash (including other shell commands), and perl to get symbols plus exchanges from nasdaq.com - chrome (headless), bash (including other shell commands), and perl to get intermediate data from morningstar.com - bash (including other shell commands) and perl to format the final data It's fragile because Morningstar and NASDAQ don't always agree on the exchange for some symbols, and Morningstar could change the format of its rendered HTML. Also, the code isn't packaged to stand alone, so it wouldn't be easy to have someone else run it. But if you post desired symbols or post a link to a page with easily-parsable symbols, I can get and post similar data periodically.
Had not heard of headless chrome or bash - had to wiki them. Since you offered , I attached a list of 422 other stock symbols I'd like in both text and Excel. If some other format would be more convenient, let me know. This should hold me for awhile. Is it possible to go back to 2004 (vs. 2009) to get 15 years or are we limited by that Morningstar page? Thanks again!
I attached mstr_total_return.csv with almost the same format as before for the symbols in symbols_422.txt plus S&P 500 symbols. The date column has the "as of" date (I'm guessing that's for the close that day) or N/A instead of the retrieval date in the previous mstr_total_return.csv. KYN (Kayne Anderson MLP/Midstream) doesn't have an entry because Morningstar has no "Price vs Fair Value" page for it. This closed-end fund does have https://www.morningstar.com/cefs/xnys/kyn/total-returns that has the annual returns but in a different format. The Morningstar "Price vs Fair Value" pages the data comes from goes back for 10 years before the current year. To get annual returns for before 2009 would require getting adjusted prices from another source. I can use Yahoo Finance for daily data, so it wouldn't be too hard to calculate annual returns from that. As you and MKTrader noted before, Yahoo Finance data isn't perfect and would not match Morningstar's data. Let me know if you want me to try creating annual return data with Yahoo Finance's data.
Thanks again, so much! I tried the Yahoo Finance route for the annual returns. There's a few tricks. One must use the Adj Close to get the benefit of dividends (for stocks which issue dividends) which are part of the Morningstar "total returns". Even then, they don't seem to come out quite right. Example: IBM on Morningstar, 2018: -21.86, 2017: -4.02. Using Yahoo Finance 2018: -22.6 (109.75/141.72), 2017: -3.98 (141.72/147.60). May not seem like much of a difference but other cases are worse.
Differences for total returns might be because of different ways of calculating adjusted prices and not necessarily bad data. Yahoo says it uses a standard way of adjusting close prices: https://help.yahoo.com/kb/SLN28256.html What is the adjusted close? Adjusted close is the closing price after adjustments for all applicable splits and dividend distributions. Data is adjusted using appropriate split and dividend multipliers, adhering to Center for Research in Security Prices (CRSP) standards. So does Morningstar: http://www.morningstar.com/InvGlossary/total-return.aspx Total Return Expressed in percentage terms, Morningstar's calculation of total return is determined by taking the change in price, reinvesting, if applicable, all income and capital gains distributions during the period, and dividing by the starting price. Adjusted data from yahoo.com (raw open, high, and low prices adjusted by multiplying by adjusted_close / actual_close): <TICKER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOLUME>,<UNADJCLOSE>,<UNADJVOLUME> IBM,20161230,150000,148.003590,148.234786,147.167711,147.603439,2625720.956387,165.990005,2952800 IBM,20171229,150000,142.410076,142.918125,141.717285,141.717285,3073312.378244,153.419998,3327100 IBM,20181231,150000,109.420444,110.405253,108.541832,109.748711,4810907.775074,113.669998,4982800 2018 return == 109.748711 / 141.717285 * 100 - 100 == -22.5579921320113 percent 2017 return == 141.717285 / 147.603439 * 100 - 100 == -3.98781630013377 percent Adjusted data from quotemedia.com: <TICKER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOLUME>,<UNADJCLOSE>,<UNADJVOLUME> IBM,20161230,150000,148.003855,148.235055,147.167976,147.603700,2625747.909226,165.99,2952825 IBM,20171229,150000,142.410191,142.918238,141.712781,141.717400,3073302.823711,153.42,3327087 IBM,20181231,150000,109.420529,110.405342,108.542017,109.748800,4810868.144137,113.67,4982755 2018 return == 109.748800 / 141.717400 * 100 - 100 == -22.5579921731559 percent 2017 return == 141.717400 / 147.603700 * 100 - 100 == -3.98790816219376 percent https://www.macrotrends.net/stocks/charts/IBM/ibm/stock-price-history IBM Historical Annual Stock Price Data Year Average Stock Price Year Open Year High Year Low Year Close Annual % Change 2018 138.9086 145.8187 159.8759 106.3168 112.3457 -22.54% 2017 146.1554 152.1429 166.8903 130.7590 145.0341 -3.98% So these three web sites more-or-less agree with each other and disagree with morningstar.com.
Well, its important to agree with Morningstar. I have a purchased Morningstar database with return data for ETFs and mutual funds. I need the stock returns calculated in the same way so I'm comparing apples to apples.
I agree that the consistency of calculations on the data is important. If you are interested in comparing financial metrics for different stocks, Morningstar's key ratios ( e.g., https://financials.morningstar.com/ratios/r.html?t=0P000002RH&culture=en&platform=sal ) could be of some use. I used to use these for generating machine-learned rules for potential trades but eventually stopped to focus on data that changed more frequently and regularly.
The K-Ratio seems to be evolving with time. You might want to also consider the Ulcer Performance Index http://www.tangotools.com/ui/ui.htm
You're right. K-Ratio has been modified over time. I do some work in Amibroker and they have an ulcer index. Ulcer Index - Square root of sum of squared drawdowns divided by number of bars Ulcer Performance Index - (Annual profit - Tresury notes profit)/Ulcer Index'>Ulcer Performance Index. Currently tresury notes profit is hardcoded at 5.4. In future version there will be user-setting for this.