Whole-market options snapshot data from ORATS

Discussion in 'Data Sets and Feeds' started by thecoder, Nov 17, 2021.

  1. thecoder


    Experience Report

    I today started to test one of the special offers by ORATS for Elitetrader users,
    ie. the offers on this page: https://info.orats.com/elitetrader
    (On that page are also links for downloading sample data)

    Mine is 15-minutes delayed snapshot data of the whole US options market.
    15-min-delayed is enough for my uses cases.
    ORATS has also offers for non-delayed data, check their plans.

    I used this webpage for ordering a 14-day trial for $29:

    Today I already downloaded snapshots of 14:30 and 15:50 ET using normal https URL with an individual/secret "token" one gets from ORATS.

    Beware: I was told that with this budget/economic plan the download interval should not be shorter than 30 minutes, otherwise there can be a "surcharge" fee.

    Alternatively one can also specify an Amazon AWS S3 bucket where the data will be copied to by ORATS. I could not test this yet b/c my AWS space is not set up yet. AWS is new for me and also seems to be a little bit complicated to understand as it has dozens of options...

    The snapshot files are compressed gzip files, of size about 130 MB compressed.
    The download takes about 5 seconds on my Gigabit home Internet link
    (I also have some faster links via VPS etc., but 5 seconds is fast enough for me).
    The unpacked CSV file is about 380 MB (using the command "zcat fname >fname.csv").

    These are the columns in the CSV:
    I hadn't time yet to analyze the data itself, but this of course will come soon.

    See/search also postings of @Matt_ORATS here, for example this one:

    See also their Snapshot API:
    Last edited: Nov 17, 2021
  2. thecoder


    Some basic quick stats:

    The above mentioned snapshot file belonging to today's 15:50 time, ie. 202111171550.csv,
    has 722765 lines with strike data (ie. all the data of a strike is on the same CSV line)
    belonging to 5496 underlying tickers.
    The data is sorted in ascending order, I think on these first 5 columns:
    Last edited: Nov 17, 2021
  3. thecoder


    Also this video can be helpful for a quick introduction, incl. the AWS stuff:
  4. thecoder


    I wonder why some IVs are zero, whereas Bid and Ask for calculating the IV are present.
    For example in snapshot 202111171550 the three IVs for Put for this line are missing: TAL,2021-11-17,2021-11-19,3,17.5,...
    Or is something else is required as well, but which is missing?
    $ grep "TAL,2021-11-17,2021-11-19,3,17.5" 202111171550.csv
  5. thecoder


    1) The data as such (ie. timely snapshots of whole-market data) is good, but I miss the following fields (columns) as they are not part of the 1-min-data set (cf. the fields/columns table in the original posting):
    - last trade price
    - last trade size
    - last trade time
    @Matt_ORATS told me that these fields are part of the 2-minute subscription plan.
    Unfortunately, for my use case, the 2-min-data is a much different data set, and it also gets expensive to subscribe both.
    I suggested to append these 3 numbers to the 1-min-data as new columns, but unfortunately it seems not doable at the moment :-(

    2) I was told that the fields/columns of the 1-min data set are documented in this blog entry:
    [but it either is a little bit outdated or it covers also other data sets as many of the column names are different]

    3) Another important observation: it seems that one has to get the data on the same day as there is no possibility to download the data of previous days!
    Ie. "backfill" seems not possible.
    It seems this limitation applies only when using standard download method. When instead using AWS cloud space then the data will be copied to there, so will be available always in own AWS cloud data space. Btw, I don't have AWS yet, so I couldn't test this feature yet.
    [This is just my own experience/observation and preliminary interpretation as I haven't got a confirmation yet due to weekend]

    Last edited: Nov 21, 2021
  6. thecoder


    Regarding my above mentioned problems with signing-up for Amazon AWS cloud space (S3 etc.):

    This is the answer I got from Amazon support as reply to the support ticket I had opened
    b/c of the bank card problem when I tried to sign up for Amazon AWS cloud space (S3 etc.).
    Therein I also had asked whether payment via PayPal is possible instead of bank card.
    (mail is of course sanitized by me with XXX to protect privacy)

    Btw. they write below
    Since I tried it twice, they deducted $2 from my very bank card they say has a problem :)
    They say by this $1 method they check the validity of the data of the customer... Blah..blah...
    Btw, I used this card also when subscribing to ORATS data; it worked flawlessly.
    And: I haven't got the $2 back yet... :) Very funny, I wonder if that trick (or fraud? :)) is the real reason how Jeff Bezos has become the richest (or 2nd richest) person on the planet... :)

    There are 3 transactions from Amazon Web Services (or Aws Emea) logged in the activity log of my bank card acct: the first one is about checking the card, and the latter two have deducted $1 each. The 1st one has also this comment: "This was Amazon Web Services checking your card is active. Any charges it made will be returned on or before 24/11."

    Last edited: Nov 21, 2021
  7. thecoder


    The topic of this thread is, as said, my testing of the 15-minute delayed whole US options market data offered by ORATS for especially for Elitetrader users (ie. a special offer for ET users as advertised on that page https://info.orats.com/elitetrader ). On that page are multiple offers, I just am interested in the 15-min delayed data.

    Unfortunately some drawbacks:

    When not using AWS cloud space, ie. when using normal download method, then the 15-min delayed plan has only the possibility to download the data exactly at the time they are produced and published, ie. specifying "tradeDate=YYYYMMDDHHMM" in the URL just gets simply ignored and instead one gets the latest available 15-min-delayed-data file.
    So, you have to be very exact at what times you issue the download requests.
    This is a job for scheduled automatic downloads, ie. trying to do it manually is IMO problematic.
    As said this limitation is the case when not using AWS. With AWS one just self-defines the download times and submits ORATS a short file containing these times so that their system can automatically copy the desired data to client's own AWS cloud space.

    @Matt_ORATS, told me that for an extra $100 one can get the feature to overcome the above limitation with API usage. Ie. then the total price becomes $99 + $100 = $199/month. Of course the limitation of not making requests in shorter intervals than 15 minutes still persist, otherwise another fee of $50 to $299 per month occurs depending on the time interval (minutes) the subsequent requests are made. Citing Matt on this: "There is a Data API surcharge if you download every: 1min/$299, 5/$100, 15/$50, 1hr/free. If you use the API every 30 minutes, you do not have a surcharge."
    Ie. I interpret this so: if one downloads in intervals > 15 then there is no such surcharge to pay.
    Hmm. is that really > 15 minutes or is it rather >= 15 minutes, for the delayed 15-minute data?

    [Matt, please correct me if I got you wrong somewhere as this all seems so complicated to understand for my little brain :)
    Almost a week now and I still don't know the full specifics of the subscription I'm testing, as I get the specifics like some bread crumbs when I specifically ask for it. There seems to be no product document available that explains the product full and in detail. ORATS, you clearly have to improve on that :)]
    Last edited: Nov 23, 2021
  8. thecoder


    Btw, on the above said page with special offers for ET users, ie. at https://info.orats.com/elitetrader
    they have also advertised these two subscription services:
    Data API Special $49 Month
      Unlimited symbols
      1,000 API Calls Per Month
      1 user
      For individual use
    Data API Basic $99 Month
      20,000 API Calls Per Month
    But, it is unfortunately unclear, at least to me, what these packages include in detail.
    Is it just the permission to use the API and make that much API calls per month,
    but that besides that, one needs also a data package subscription to be able to access via these very API packages?
    Matt, please explain if you read this. Thx.

    Ie. can I use this API subscription to get 15-min-delayed snapshot files? :)
    Would be nice, b/c would be cheaper: $49 vs. $99 for the other subscription package (cf. original posting here).
    But, I'm afraid I surely must somehow have misinterpreted these offers :-(
    Last edited: Nov 23, 2021
  9. thecoder


    Regarding my above said problem with Amazon AWS S3 cloud space registration: today got this mail from their support, and tried again as suggested, and surprise surprise: NADA! Same shitty error happens, and has again deducted $1 from my bank card, and gave the same error as in the previous 2 attempts... Are they Idis @ Work? :)
    Last edited: Nov 23, 2021
  10. thecoder


    The above said Amazon problem seems to have its root cause in the fact that my bank card has a security code against fraud (called CVV etc., there are many many names for it; see the wiki page below), but Amazon seems not to like such security codes, and tells me to talk to my bank to remove that security code! Unbelievable!!! See mail below!

    Amazon called me today by phone and then also sent me the following email:

    Last edited: Nov 23, 2021
    #10     Nov 23, 2021