NASDAQ ITCH 50 Book Constructor

Given the NASDAQ Total View ITCH 50 data feed, reconstruct the full depth order book and related messages.

This is an efficient c++ implementation of reconstructing a Limit Order Book from data feed messages issued by NASDAQ according to the ITCH 50 data protocol specified at https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf. Some samples data are publicly available through NASDAQ public ftp at ftp://emi.nasdaq.com/ITCH/. The program outputs two csv files containing the messages and the related limit order book for the relative stock.

Folder Structure

ITCH
│
└───bin     (c++ executable)
│
└───build   (dependency and object files)
│
└───doc
│   │
│   └───latex       (LaTex-pdf documentation created with Doxygen)
│
└───data
│   │
│   └───binary      (ITCH binary gzip compressed files from NASDAQ)
│   │
│   └───book        (output of book csv)
│   │
│   └───messages    (output of messages csv)
│   │
│   └───pk          (pickle object files)
│
└───gtests    (.cpp for tests)
│
└───images    (images for README.md Markdown)
│
└───include   (.hpp for main program)
│
└───examples  (python implementation of some application example)
│
└───src       (.cpp for main program)

Brief Description

This program aims to facilitate research in High Frequency Trading (HFT) providing the maximum amount of data released by NASDAQ, ready for use.
NASDAQ receives orders by traders and market participants, then its matching engine constructs the order book according to specific rules that may depend on the venue, and eventually sells to clients the data feed.
We were able to reconstruct the book with total depth at about 1Mio-2Mio messages per second, this figure also includes the parsing from the binary data.
To understand the difficulties of such an exercise is important to understand that the messages do not coincide with the orders received by NASDAQ, i.e. we do not need a matching engine to match supply and demand, but we are already given the result of the matching engine. Apart from interpreting the binary data according to the specification of the protocol we also have to retrieve information of past orders that new messages are referring to.
We can see this in the following example of submission and deletion of an order, according to the specifications the Add and Delete order would be:

As we can see once we observe the deletion order no information about the direction, size, stock and price are reported. Hence at each time we have to keep track of all the active orders in the book, in order to know what to do once we encounter the deletion order.

The output of the program are two .csv file with the following structure:

messages of 08/30/2018 PSX AAPL

book of 08/30/2018 PSX AAPL

Installation

OS X & Linux:

git clone https://github.com/martinobdl/ITCH
cd ITCH
make

In order to get some data needed to run the program (800MB)

wget ftp://anonymous:@emi.nasdaq.com/ITCH/Nasdaq\ PSX\ ITCH/20190327.PSX_ITCH_50.gz -P ./data/binary

# this file exists at date 05/02/2019
# otherwise download manually from the ftp and put the
# .gz file in ITCH/data/binary

Usage example

To reconstruct the book we can use the bash wrapper BookConstructor.sh which has the following usage:

usage:    ./BookConstructor.sh [-lfh] [-n #] data_folder mm/dd/yyyy venue ticker
          ./BookConstructor.sh [-l] data_folder
          ./BookConstructor.sh [-h]

 -h, --help     Display usage instructions
 -l, --list     Display all the date venues available at data_folder/binary
 -f, --force    To force program execution if output files already exists
 -n,            Number of levels to store for the book, default is 5

for example ./BookConstructor.sh data_folder mm/dd/yyyy venue ticker -n N will produce two output files named:

data_folder/book/mmddyyyy.venue_ITCH50_ticker_book_N.csv
data_folder/messages/mmddyyyy.venue_ITCH50_ticker_message.csv

example

Given that you have in /../../ITCH/data/binary the file PSX_ITCH/20190327.PSX_ITCH_50.gz

cd /../../ITCH
./BookConstructor.sh ./data 03/27/2019 PSX SPY

this will create two output .csv files, namely:

/../../ITCH/data/book/03272019.PSX_ITCH50_AAPL_book_5.csv
/../../ITCH/data/messages/03272019.PSX_ITCH50_AAPL_message.csv

Detailed output description

Here we report detailed description of message file output of the program:

At discrete time T1,T2,T3,... an output will be released by NASDAQ, the field names may have different meaning depending on the type of message. Hence here we will give a detailed description of the message csv.

Development setup

In the gtest folders there are unit tests in order to test the principal components of the software, they are the Google c++ unittesting libraries. To compile and run the tests you would need the gtests and gmock libraries. They can be installed in the following way. Refer to https://github.com/google/googletest/blob/master/googletest/README.md for more details about the installation.

To install gtest-gmock:

# gtests
sudo apt-get install libgtest-dev
sudo apt-get install cmake
cd /usr/src/gtest
sudo cmake CMakeLists.txt
sudo make
sudo cp *.a /usr/lib

#gmock
sudo apt-get install libgmock-dev
cd /usr/src/gmock
sudo mkdir build
sudo cmake ..
sudo make
sudo cp *.a /usr/lib

To run the tests:

cd /../../ITCH
make test
./bin/executeTests

Application Example

In the python folder there are some examples on how the program might be used to perform research on HFT.
environment.py is the environment definition and Algo.py is the agent abstract class from which we would implement the strategy.
The environment definition provides also some visualization tools.

requirements:

python3
numpy
pandas
matplotlib
tkinter
tqdm

To run the environment visualization tool:

python3 /../../ITCH/python/environment.py mmddyyyy venue stock

To run a simple low regret fixed spread strategy:

python3 /../../ITCH/python/Algo_fixedSpread.py

The implemented strategy can be found at

[# 2013] Abernethy, J., Kale, S., Adaptive Market Making via Online Learning.

Screen shots

Implementation Class Diagram

For more details about the API see the documentation in the doc/ folder

Order Book visualization Tool (from environment.py)

Citations

If you use this software in your research, please cite the paper using the following BibTeX:

@misc{https://doi.org/10.5281/zenodo.5209267,
  doi = {10.5281/ZENODO.5209267},
  url = {https://zenodo.org/record/5209267},
  author = {Bernasconi-De-Luca,  Martino and Fusco,  Luigi and Dragić,  Ozrenka},
  title = {martinobdl/ITCH: ITCH50Converter},
  publisher = {Zenodo},
  year = {2021},
  copyright = {Open Access}
}

Contributors

• Luigi Fusco https://github.com/luigif9
• Ozrenka Dragić https://github.com/oz-dr
• Martino Bernasconi-de-Luca https://github.com/martinobdl