I'm looking for historical data providers offering 5+ years of 5sec bars of US stocks. I have been using the free IBKR API to fetch the data from their servers, but it's quite slow and takes ~15 seconds for a single day per ticker at 5sec bar resolution. It would take over 7 months of non-stop download to fetch 5 years of data for 1000 stocks and IBKR provides only ~3 years of data at 5sec anyway, so not quite ideal. Looks like polygon.io provides 5 years of high resolution historical data in their cheapest "Starter" $29/mo offering and it seems to be possible to aggregate the data to specified bar size prior to download to optimize the download, which is nice. Can anyone recommend other data providers with high resolution data for comparison? Looks like most providers have only 1min bars at max.
See here for various ressources what you need: https://quantpedia.com/links-tools/?category=historical-data
Does it really take 15 secs per day per ticker even it is 5-min interval? It sounds very slow, I am reading its java api for downloading 1-min interval data
5min is probably a lot faster (sub-second per day) since it's 60x less bar data. I'm fetching 5sec bars though not 5min.
Unlikely since I'm using C++ It's a server side pacing thing and IBKR specifically mentions that they are not really a historical market data provider: https://interactivebrokers.github.io/tws-api/historical_limitations.html
We have 6 years of Nasdaq/BX/PSX TotalView, BZX/BYX/EDGA/EDGX Multicast PITCH Depth, and NYSE/Arca/American Integrated. This is full order book ("L3" i.e. "MBO"), with nanosecond-resolution timestamps from the venues plus PTP. These are provided in raw format here and normalized format for 5 venues at the moment (Nasdaq, IEX, MIAX, NYSE National, NYSE Chicago), as described here. The next closest providers that I know for L3 prop feed coverage are Refinitiv and ICE, but last I checked, they don't have some of the smaller venues like MIAX, MEMX, Chicago, National. Mostly everyone else after that is just reselling the SIPs ("L1" i.e. top of the book only) or SIPs plus 1-2 low-cost feeds like Nasdaq Basic.
for a 1000 names, i think the supermajority aren’t traded at 5secs. Are you downloading bid/ask or traded? I am struggling to decide what data to study at 1min interval (mid, traded, vwap), leaning towards mid
Depends on the time of the day and how volatile the market is if 5sec bars have any value for any particular ticker. It's easy enough to build lower resolution bars though once you have the 5sec bar data, for example to improve parameter optimization performance. I'm using trades, but have also thought of fetching bid/ask for the spread to better estimate the slippage.