Histdata.com - how to get forex (not only) tick data for free - automated way

Discussion in 'Data Sets and Feeds' started by WiktorK, Nov 23, 2021.

  1. WiktorK

    WiktorK

    This website https://www.histdata.com/download-free-forex-data/ allows access to tick/1sec/1min decent quality data for many forex pairs, probably known to some of you. It is possible to download the data month by month and unzip it, yet this is far from effective and takes hours if you need 10 years of data for a few instruments

    So I have made a little bash script that creates a one time token (needed to have access to data) and uses it to download data and extract csv files, this allows downloading files without manually clicking. THought it might be useful :)

    Code:
    #!/bin/bash
    
    for i in gbpchf chfjpy nzdusd nzdjpy usdhkd bcousd jpxusd udxusd xaugbp;
    do for year in 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020;
    do for month in 1 2 3 4 5 6 7 8 9 10 11 12;
    do TK=$(curl https://www.histdata.com/download-free-forex-historical-data/\?/ninjatrader/tick-last-quotes/$i/$year/$month | grep tk | awk '{print $5}' | awk -F '"' '{print $2}' | head -1) && sleep 1;
    PAIR=$(echo $i | tr a-z A-Z) \
    && sleep 1 \
    &&  if [ ${#month} -lt 2 ]; then month_long=0$month ; else month_long=$month; fi\
    && curl 'https://www.histdata.com/get.php' \
      -H 'Connection: keep-alive' \
      -H 'Cache-Control: max-age=0' \
      -H 'Upgrade-Insecure-Requests: 1' \
      -H 'Origin: https://www.histdata.com' \
      -H 'Content-Type: application/x-www-form-urlencoded' \
      -H 'User-Agent: ### COPY FROM YOUR BROWSER ###' \
      -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
      -H 'Sec-GPC: 1' \
      -H 'Sec-Fetch-Site: same-origin' \
      -H 'Sec-Fetch-Mode: navigate' \
      -H 'Sec-Fetch-User: ?1' \
      -H 'Sec-Fetch-Dest: iframe' \
      -H 'Referer: https://www.histdata.com/download-free-forex-historical-data/?/ninjatrader/tick-last-quotes/'$i'/'$year'/'$month'' \
      -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
      -H 'Cookie: cookielawinfo-checkbox-non-necessary=yes; cookielawinfo-checkbox-necessary=yes' \
      --data-raw 'tk='$TK'&date='$year'&datemonth='$year''$month_long'&platform=NT&timeframe=T_LAST&fxpair='$PAIR'' \
      --compressed --output ${i}_${year}_${month_long}.zip \
      && unzip ${i}_${year}_${month_long}.zip;
    done; done; done;
    
    rm *.zip && rm *.txt
    
    To get User-Agent download any file manually and inspect the website (see screenshot)

    This is an example for [gbpchf chfjpy nzdusd nzdjpy usdhkd bcousd jpxusd udxusd xaugbp] pairs, last 10 years

    This URL in Referer is for 1-second quotes, you can change it for any other - just copy-paste from the browser leaving the '$i'/'$year'/'$month'' part


    Code:
    I'm not affiliated with hostdata.com in any way, also I suggest paying for access to this data (despite the fact it is unsecured and publically available)
    
    This data is also "dirty", but can be usefull
     
  2. I have no need for this data, but I feel the fact you've done this as a bash script deserves some recognition.

    GAT
     
    WiktorK likes this.
  3. WiktorK

    WiktorK

    I've forgot to add screenshot, here you go :)