I've been thinking about buying intraday historical data for stocks for backtesting strategies. However, high quality tick data is expensive, and what I'd like to do is first pay for collected data, but then start building my own database by automatically downloading data from a website (or a broker, if possible). Does this make sense, or are updates/feeds relatively inexpensive compared to historical data? I'm only interested in high quality data (right now looking at Algoseek.com). It doesn't have to be tick data, 1-minute or 5-minute bars would be sufficient. Also, just need trade and volume data, so no quotes needed. Second, I'm going to need to download real-time price data anyway for live trading decisions. It would kind of make sense to combine this effort with forming my own database for backtesting, no? What website would be the best to collect real-time price data for live decisions? Is it available directly from a broker such as IB at an affordable price? Or should I try exchange websites (such as Nasdaq) directly? What is the industry standard practice (for retail quant traders)?
Is IQFeed very high quality? Especially for the live feed, there cannot be regular missing values or even occasional errors. Does the feed allow downloading real-time trade data updating multiple times for hundreds of different stocks within a 5-minute window, for example?
I’d consider is as high quality as you’d get elsewhere for similar price. They all have problems, but some of those problems come directly from the tape/exchanges, and everyone may cleanup their data differently. Though data issues are mainly pre-market and after-hours. During market hours you may want to collect tick data and summarize it yourself, otherwise you may get multiple updates to a minute bar during a minute, and even after the minute has already passed. You can stream up to 1000 stocks (or 1500?) at the same time, with more updates/ticks than you may be able to handle, like bid/ask/trades updated thousands of times per second. Though I haven’t used IQFeed recently so may not remember all details. Unless someone else responds, you can simply test it, as you may not be able to find many alternatives anyway. Polygon.io is another one and I think offers free trial. They’re newer and I’ve seen more reports of data issues, but they’re continually updating their system. And again, some data issues cannot be blamed on any particular data provider. You’ll need to be a problem solver anyway.
IQFeed is base 500 symbols. Theirs packages to get to 1500. Then theirs options for historical that aren’t as limited. QCollector for instance doesn’t stream RT and just requests historical data so they don’t flat the symbol limits(double check with their support). Simply pulls and updates .csv from what I see. Their tick data is micro second stamped with bid/ask available for the tick. For stocks their EOD data is split adjusted while their intraday is raw. Think if you used the API you could pull adjustment ratios but any software I’ve seen with a plug in doesn’t pull that data so you need to compare intraday gaps vS EOD files.
Thanks, will consider IQFeed. If I were to collect my own (real-time) price data (1-minute intervals), which source/website would be best to use? Would exchange websites such as nasdaq.com be the most reliable? Data seems to be free there, if you can build a program that scrapes the site? Although this might run into restrictions for multiple requests in a short period of time? Has anyone built something like this? Also, what about broker data (mainly interested in IB) - can you get real-time quality trade data from there?
Are there any (easy/out-of-the-box) alternatives beyond QCollector for collecting data from IQFeed? For my purposes, IQFeed + QCollector works fine for now. I set it to update in real time (every minute is sufficient for me; fastest update is 5 seconds) storing 1-minute data to a *.csv file and my other application (custom) updates my DB automatically whenever the *.csv is updated. I don't do any high latency stuff, so this works okay for me at this point in time, but would prefer to not have to rely on QCollector. I have the feeling that the software is not really updated/heavily supported these days. But for now it works fine.
I’ve actually never used as I used AMibroker with the feed at the time. Simply exported what I needed from Amibroker in a .csv if needed elsewhere. I did think it was nice tool though to bypass IQFeed API. I know their API has a yearly fee? But not sure if that’s for access to the API or for support and documentation. I ditched it when life got busy and never used the intraday. I use yahoo now for current updates(not backtesting, don’t trust them that much). Thinking about picking up Norgate though. They seem to do a fantastic job of maintaining survivorship bias free data and all the other adjustments that is a nightmare for stock data.