Whole-market options data snapshots intraday every 30 minutes

Discussion in 'Data Sets and Feeds' started by thecoder, Oct 29, 2021.

  1. thecoder

    thecoder

    So, do you mean the data is expensive b/c it's data, and not because of its size? I would say size matters very well... :)
     
    #11     Oct 29, 2021
  2. traider

    traider

    You can't get it for less than $100. The option exchanges all want to charge for realtime. You should just open a brokerage account and use that data. That is the cheapest but they will throttle you.
     
    #12     Oct 29, 2021
    thecoder likes this.
  3. rb7

    rb7

    .
    They DO mention snapshot, but maybe not the way you would like it to work. Usually, real-time subscriptions are sending snapshots when you initiate your subscription for real-time quotes. So, you would need to subscribe to real-time quote each 30 minutes, and then unsubscribe, then re-subscribe in 30 minutes, and so on.
    If you want to receive data on all listed option strikes (which comes out to approx. 1,000,000 strikes), you would need, each 30 minutes, to send 1,000,000 real-time subscriptions, but without having more than 500 real-time subscriptions at a time.
    It would go like (each 30 minutes):
    Send 500 rt subscriptions,
    Receive 500 snapshots (usually cumulative from the beginning of the trading day)
    Send 500 rt un-subscriptions,
    And repeat 2000 times...

    TANSTAAFL.
     
    #13     Oct 29, 2021
    thecoder likes this.
  4. Handle123

    Handle123

    Yes, the options data is unbelievable expensive, I bought historical data some years ago from cboe, cost more than my house.

    Every consider Tradestation, you can program what you want on it.
    Market Data | TradeStation
     
    #14     Oct 29, 2021
  5. traider

    traider

    Is 1 option chain counted as a whole or do each individual option count? I'm assuming you are talking about IB?
     
    #15     Oct 29, 2021
  6. thecoder

    thecoder

    Really insane prices for such IMO public data.

    As I understand it, the data is kept remote in their datacenter, but I need to have the data locally, and also store it in a local database for later use by backtester etc.
     
    #16     Oct 29, 2021
  7. thecoder

    thecoder

    I'm going to try the YahooFinance API for $9.99/month for 20,000 requests/month as offered here:
    https://www.yahoofinanceapi.com/pricing
    It looks like that a single request brings ALL options data of an expiration date for the underlying instrument, as one can try the API out here (the 2nd GET API function):
    https://www.yahoofinanceapi.com/

    If the above said is true, then it should be possible to get the whole data in about 4000 requests (but this way just the data for the current expiration date), but then the 20,000 requests/month is of course by far not enough; one rather needs the unlimited package ("Mega") for $259/month... :-(

    Here's what it sent for ticker PROG w/o specifiying an expiration time (this has to be unix time format or so as its a long number; haven't studied the API doc yet...):
    Code:
    {
      "optionChain": {
        "result": [
          {
            "underlyingSymbol": "PROG",
            "expirationDates": [
              1637280000,
              1639699200,
              1642723200,
              1645142400,
              1649894400,
              1657843200,
              1663286400,
              1674172800,
              1705622400
            ],
            "strikes": [
              0.5,
              1,
              1.5,
              2,
              2.5,
              3,
              3.5,
              4,
              4.5,
              5,
              5.5
            ],
            "hasMiniOptions": false,
            "quote": {
              "language": "en-US",
              "region": "US",
              "quoteType": "EQUITY",
              "quoteSourceName": "Nasdaq Real Time Price",
              "triggerable": true,
              "tradeable": false,
              "currency": "USD",
              "marketState": "POST",
              "shortName": "Progenity, Inc.",
              "exchange": "NGM",
              "longName": "Progenity, Inc.",
              "messageBoardId": "finmb_256359845",
              "exchangeTimezoneName": "America/New_York",
              "exchangeTimezoneShortName": "EDT",
              "gmtOffSetMilliseconds": -14400000,
              "market": "us_market",
              "esgPopulated": false,
              "firstTradeDateMilliseconds": 1592573400000,
              "priceHint": 4,
              "postMarketChangePercent": 1.1111166,
              "postMarketTime": 1635546350,
              "postMarketPrice": 3.64,
              "postMarketChange": 0.0400002,
              "regularMarketChange": -0.26999998,
              "regularMarketChangePercent": -6.9767437,
              "regularMarketTime": 1635537603,
              "regularMarketPrice": 3.6,
              "regularMarketDayHigh": 3.92,
              "regularMarketDayRange": "3.465 - 3.92",
              "regularMarketDayLow": 3.465,
              "regularMarketVolume": 33324635,
              "regularMarketPreviousClose": 3.87,
              "bid": 3.56,
              "ask": 3.63,
              "bidSize": 11,
              "askSize": 29,
              "fullExchangeName": "NasdaqGM",
              "financialCurrency": "USD",
              "regularMarketOpen": 3.77,
              "averageDailyVolume3Month": 56888827,
              "averageDailyVolume10Day": 146744462,
              "fiftyTwoWeekLowChange": 2.9429998,
              "fiftyTwoWeekLowChangePercent": 4.4794517,
              "fiftyTwoWeekRange": "0.657 - 7.86",
              "fiftyTwoWeekHighChange": -4.26,
              "fiftyTwoWeekHighChangePercent": -0.54198474,
              "fiftyTwoWeekLow": 0.657,
              "fiftyTwoWeekHigh": 7.86,
              "earningsTimestamp": 1628798400,
              "earningsTimestampStart": 1636405200,
              "earningsTimestampEnd": 1636750800,
              "epsTrailingTwelveMonths": -4.284,
              "epsForward": -1.01,
              "epsCurrentYear": -2.36,
              "priceEpsCurrentYear": -1.5254238,
              "sharesOutstanding": 132740000,
              "bookValue": -1.782,
              "fiftyDayAverage": 1.7938889,
              "fiftyDayAverageChange": 1.806111,
              "fiftyDayAverageChangePercent": 1.0068132,
              "twoHundredDayAverage": 2.300702,
              "twoHundredDayAverageChange": 1.2992978,
              "twoHundredDayAverageChangePercent": 0.5647397,
              "marketCap": 477864000,
              "forwardPE": -3.5643563,
              "priceToBook": -2.020202,
              "sourceInterval": 15,
              "exchangeDataDelayedBy": 0,
              "ipoExpectedDate": "2020-06-19",
              "averageAnalystRating": "2.7 - Hold",
              "displayName": "Progenity",
              "symbol": "PROG"
            },
            "options": [
              {
                "expirationDate": 1637280000,
                "hasMiniOptions": false,
                "calls": [
                  {
                    "contractSymbol": "PROG211119C00000500",
                    "strike": 0.5,
                    "currency": "USD",
                    "lastPrice": 3.1,
                    "change": -0.20000005,
                    "percentChange": -6.060608,
                    "volume": 274,
                    "openInterest": 397,
                    "bid": 3,
                    "ask": 3.2,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635528336,
                    "impliedVolatility": 0.500005,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00001000",
                    "strike": 1,
                    "currency": "USD",
                    "lastPrice": 2.6,
                    "change": -0.25,
                    "percentChange": -8.77193,
                    "volume": 74,
                    "openInterest": 745,
                    "bid": 2.55,
                    "ask": 2.65,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635530886,
                    "impliedVolatility": 0.500005,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00001500",
                    "strike": 1.5,
                    "currency": "USD",
                    "lastPrice": 2.05,
                    "change": -0.29999995,
                    "percentChange": -12.765956,
                    "volume": 552,
                    "openInterest": 4180,
                    "bid": 2.05,
                    "ask": 2.15,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537384,
                    "impliedVolatility": 0.500005,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00002000",
                    "strike": 2,
                    "currency": "USD",
                    "lastPrice": 1.65,
                    "change": -0.25,
                    "percentChange": -13.157895,
                    "volume": 690,
                    "openInterest": 13203,
                    "bid": 1.55,
                    "ask": 1.7,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537584,
                    "impliedVolatility": 1.5312523437499999,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00002500",
                    "strike": 2.5,
                    "currency": "USD",
                    "lastPrice": 1.33,
                    "change": -0.27999997,
                    "percentChange": -17.391302,
                    "volume": 1020,
                    "openInterest": 17236,
                    "bid": 1.25,
                    "ask": 1.35,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537541,
                    "impliedVolatility": 2.031254921875,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00003000",
                    "strike": 3,
                    "currency": "USD",
                    "lastPrice": 1.08,
                    "change": -0.25,
                    "percentChange": -18.796993,
                    "volume": 5356,
                    "openInterest": 39511,
                    "bid": 1.05,
                    "ask": 1.1,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537554,
                    "impliedVolatility": 2.3125042187499996,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00003500",
                    "strike": 3.5,
                    "currency": "USD",
                    "lastPrice": 0.9,
                    "change": -0.20000005,
                    "percentChange": -18.181822,
                    "volume": 3392,
                    "openInterest": 25643,
                    "bid": 0.85,
                    "ask": 0.9,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537558,
                    "impliedVolatility": 2.4062539843749997,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119C00004000",
                    "strike": 4,
                    "currency": "USD",
                    "lastPrice": 0.8,
                    "change": -0.19999999,
                    "percentChange": -19.999998,
                    "volume": 10495,
                    "openInterest": 27628,
                    "bid": 0.75,
                    "ask": 0.8,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537587,
                    "impliedVolatility": 2.6328159179687503,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119C00004500",
                    "strike": 4.5,
                    "currency": "USD",
                    "lastPrice": 0.65,
                    "change": -0.20000005,
                    "percentChange": -23.529417,
                    "volume": 3095,
                    "openInterest": 17618,
                    "bid": 0.65,
                    "ask": 0.7,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537593,
                    "impliedVolatility": 2.750003125,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119C00005000",
                    "strike": 5,
                    "currency": "USD",
                    "lastPrice": 0.6,
                    "change": -0.19999999,
                    "percentChange": -24.999998,
                    "volume": 7981,
                    "openInterest": 35026,
                    "bid": 0.55,
                    "ask": 0.6,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537575,
                    "impliedVolatility": 2.7890655273437504,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119C00005500",
                    "strike": 5.5,
                    "currency": "USD",
                    "lastPrice": 0.55,
                    "change": -0.14999998,
                    "percentChange": -21.428568,
                    "volume": 4092,
                    "openInterest": 18844,
                    "bid": 0.5,
                    "ask": 0.55,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537585,
                    "impliedVolatility": 2.9140652148437503,
                    "inTheMoney": false
                  }
                ],
                "puts": [
                  {
                    "contractSymbol": "PROG211119P00000500",
                    "strike": 0.5,
                    "currency": "USD",
                    "lastPrice": 0.03,
                    "change": -0.020000001,
                    "percentChange": -40.000004,
                    "volume": 4,
                    "openInterest": 1765,
                    "bid": 0,
                    "ask": 0.05,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635532954,
                    "impliedVolatility": 4.7500040624999995,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00001000",
                    "strike": 1,
                    "currency": "USD",
                    "lastPrice": 0.02,
                    "change": -0.01,
                    "percentChange": -33.333336,
                    "volume": 19,
                    "openInterest": 3308,
                    "bid": 0,
                    "ask": 0.05,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635520228,
                    "impliedVolatility": 3.06250234375,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00001500",
                    "strike": 1.5,
                    "currency": "USD",
                    "lastPrice": 0.05,
                    "change": 0,
                    "percentChange": 0,
                    "volume": 152,
                    "openInterest": 3813,
                    "bid": 0.05,
                    "ask": 0.1,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635536826,
                    "impliedVolatility": 2.81250296875,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00002000",
                    "strike": 2,
                    "currency": "USD",
                    "lastPrice": 0.14,
                    "change": -0.010000005,
                    "percentChange": -6.66667,
                    "volume": 314,
                    "openInterest": 9444,
                    "bid": 0.1,
                    "ask": 0.15,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537359,
                    "impliedVolatility": 2.3750040624999995,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00002500",
                    "strike": 2.5,
                    "currency": "USD",
                    "lastPrice": 0.3,
                    "change": -0.049999982,
                    "percentChange": -14.285709,
                    "volume": 667,
                    "openInterest": 7388,
                    "bid": 0.3,
                    "ask": 0.35,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537558,
                    "impliedVolatility": 2.5781285546875,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00003000",
                    "strike": 3,
                    "currency": "USD",
                    "lastPrice": 0.58,
                    "change": -0.030000031,
                    "percentChange": -4.918038,
                    "volume": 1307,
                    "openInterest": 7639,
                    "bid": 0.55,
                    "ask": 0.6,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537494,
                    "impliedVolatility": 2.6562533593750004,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00003500",
                    "strike": 3.5,
                    "currency": "USD",
                    "lastPrice": 0.9,
                    "change": 0,
                    "percentChange": 0,
                    "volume": 2640,
                    "openInterest": 2037,
                    "bid": 0.9,
                    "ask": 0.95,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537559,
                    "impliedVolatility": 2.8593778515625,
                    "inTheMoney": false
                  },
                  {
                    "contractSymbol": "PROG211119P00004000",
                    "strike": 4,
                    "currency": "USD",
                    "lastPrice": 1.3,
                    "change": 0,
                    "percentChange": 0,
                    "volume": 298,
                    "openInterest": 1589,
                    "bid": 1.25,
                    "ask": 1.35,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537445,
                    "impliedVolatility": 2.99219001953125,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119P00004500",
                    "strike": 4.5,
                    "currency": "USD",
                    "lastPrice": 1.75,
                    "change": 0.07000005,
                    "percentChange": 4.16667,
                    "volume": 12,
                    "openInterest": 690,
                    "bid": 1.65,
                    "ask": 1.7,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635521525,
                    "impliedVolatility": 3.031252421875,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119P00005000",
                    "strike": 5,
                    "currency": "USD",
                    "lastPrice": 2.1,
                    "change": 0,
                    "percentChange": 0,
                    "volume": 12,
                    "openInterest": 1696,
                    "bid": 2.05,
                    "ask": 2.2,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635537062,
                    "impliedVolatility": 3.2148457128906243,
                    "inTheMoney": true
                  },
                  {
                    "contractSymbol": "PROG211119P00005500",
                    "strike": 5.5,
                    "currency": "USD",
                    "lastPrice": 2.55,
                    "change": -0.049999952,
                    "percentChange": -1.9230751,
                    "volume": 7,
                    "openInterest": 978,
                    "bid": 2.5,
                    "ask": 2.65,
                    "contractSize": "REGULAR",
                    "expiration": 1637280000,
                    "lastTradeDate": 1635535595,
                    "impliedVolatility": 3.3437516406249994,
                    "inTheMoney": true
                  }
                ]
              }
            ]
          }
        ],
        "error": null
      }
    }
    
    It's practically the same data as on this web page as the web page surely uses the same API and database...:
    https://finance.yahoo.com/quote/PROG/options?date=1637280000&p=PROG

    Btw, in the JSON data above, the "impliedVolatility" value has to be multiplied by 100.
     
    Last edited: Oct 29, 2021
    #17     Oct 29, 2021
  8. sandy_s

    sandy_s

    You get for what you pay...

    Clean data is one of the most difficult things to maintain and get - when you can generate billions of tick simultaneously.

    There is a reason for the cost that is charged.

    Try getting cheap data and the issues you face with cleansing it.

    Another alternative is find out a way yourself to get real time tick data for the entire market and start building your own archive - you will soon see the challenges and why there is such cost involved in it..
     
    #18     Oct 29, 2021
  9. thecoder

    thecoder

    I haven't tried out yet, but it would be interesting to know whether one can request also past (ie. historic) options data via the yfapi.
     
    #19     Oct 29, 2021
  10. rb7

    rb7

    It's usually one subscription per strike.

    No, I'm not referring to IB specifically.
     
    #20     Oct 30, 2021