Historical Options & Futures Data using TWS API

The ability to extract and analyze historical data efficiently is critical for the success of any quantitative strategy. Thankfully, TWS API affords its users the ability to extract historical data at scale for a number of products including derivatives such as options and futures. While there are many good resources available online explaining how to extract historical data for stocks, I was unable to find any good resource for derivatives. Since many of my students regularly reach out to me seeking help with the same, I have decided to write this two-part article delving into this topic. 

While the approach to extract historical data for derivatives and stocks is similar, there are some derivative specific peculiarities which could make things tricky. In addition, getting historical data for expired derivatives contracts is a challenge because IB does not provide historical data for expired options contracts and provides historical data for expired futures contracts only going back a few years. However, there is a way to extract historical data for expired contracts as well. Don’t believe me? Refer to the below screenshot wherein I have extracted the data for an option contract which expired a few months back. 

But before you get too excited about raiding the historical data for expired options, let me warn you that the approach has a learning curve and therefore it is important that we go step by step. By the end of this two-part article you should become reasonably proficient in extracting historical data for both live and expired derivatives contracts. In this first part, let us take care of the basics required to extract derivatives data using TWS API. 

This article assumes that you have basic understanding about TWS API. If not, please refer to my earlier article about implementing market scanner using TWS API where I have explained the basics of TWS API and the building blocks of a TWS API based trading application. 

Building the Application 

Before we create a trading application that requests historical data for the given derivative contract, let us build our boilerplate trading application. 

As explained in my previous article, when executed, this trading application will simply create a websocket connection with your locally running TWS software. However, since we have not requested any information or sent any commands to the TWS in the above code, this application will not do anything much. 

To request historical data from the TWS, we need to execute the reqHistoricalData() Eclient function. We need to pass some arguments to the reqHistoricalData() function so that TWS understands the scope of our request. We will get into the details of this function in a bit but as of now it is important to note that one of the most critical arguments to be passed to this function is the contract for which you want to extract data. 

Quick Introduction to IBAPI Contract Class 

IBAPI’s contract class lets us define the contract to be used in our trading application. Once we define the contract, we can perform operations such as requesting historical data, submitting orders etc. for that contract. To create an object of the contract class we need to define the relevant parameters that helps TWS identify the exact contract. For example, the critical parameters required to identify a stock are the stock symbol, the security type, the exchange where it is listed and the currency in which it trades. However, the above parameters are not sufficient for derivatives since each derivative contract has an expiry date and multiple derivative contracts for a given underlying may be trading simultaneously. Therefore, it is important that we create our contract by defining the required parameters. TWS API documentation has a very useful section wherein sample contract definition for a number of products is provided https://interactivebrokers.github.io/tws-api/basic_contracts.html and it may be a good idea to refer to this resource. 

Now how do we get information on all the parameters required to define a contract? There are a couple of ways to do that. We can add the contract in the TWS monitor panel and then select “description” by right clicking on the contract. This will provide all the details pertaining to the contract. Refer to the below screenshots. 

We can also access contract details by searching for the contract in the IB Contract & Symbol Database (https://www.interactivebrokers.com/en/index.php?f=463) 

It is important to note that we need to purchase the relevant market data pack from IB in order to access historical data. Thankfully, the subscription fee tends to be quite reasonable and you can subscribe to whichever market data you are interested in from your IB account settings>>User Settings>>Market Data Subscription. While I am not subscribed to the futures market data from the Globex exchange, I do have subscription to the Futures data from India’s National Stock Exchange and that is what I will be using for this demo. Let us now improve our trading application by creating the contract for which we want to extract historical data. 

As you can see in the above code snippet, I first imported the Contract class from IBAPI which allows me create objects of the Contract class. I then created an object of the Contract class and proceeded to define the parameters that would help TWS identify the exact futures contract that I am interested in. Now that we have taken care of the most important argument of the reqHistoricalData() let us get into the details of this function. 

Quick Introduction to the Historical Data API 

TWS API’s reqHistoricalData function provides historical data in the form of successive “bars”. Each bar (or candle) contains the “Open”, “High”, “Low”, “Close” and “Volume” information for a specific time duration in the past. Refer to the the documentation on historical data https://interactivebrokers.github.io/tws-api/historical_bars.html which should provide a good overview of the historical data API.  

Let us now expand our trading application code further and add the reqHistoricalData() function. 

The parameters for this function are: 

reqID – This is the request number which helps TWS identify one request from another 

contract – This is the object of Contract class for which historical data is requested 

endDateTime – This is the time (in YYYYMMDD format) until which historical data is required. Empty              string passed as an argument is interpreted as request to provide data till the current moment. 

durationStr – The duration or length or time for which data is required (Refer to historical data documentation for valid values)  

barSizeSetting – The granularity of data requested (Refer to historical data documentation for valid values) 

whatToShow – The method of creating bar data from bid and trade data e.g. bars created using bid and ask, mid-point, adjusted last price etc. (Refer to historical data documentation for valid values) 

UseRTH – Whether you want data within regular trading hours (1) or want pre and post market data as well (0) 

fotmatDate – The date format to be returned (use 1) 

keepUpToDate – Whether you want unfinished real-time bars (1) or not (0). 

chartOptions – Documentation does not provide any detail on this parameter so safe to pass blank list. 

Below is our updated code: 

As you can see In the above code, we have applied the reqHistoricalData() function to our trading application and passed the relevant arguments to request 30 minute bars for the last 300 days of the future contract of Infosys expiring on 27^th Jan 2022.  

Now you may ask, where have we defined this reqHistoricalData function? Well, this function is already defined in the EClient class and when we created our trading application (TradeApp), we inherited from the EClient and the Ewrapper classes and therefore have access to all the functions already defined in these classes. The relevant EClient and EWrapper functions for various applications (such as requesting historical data, streaming market data, placing orders etc.) can be obtained from the TWS API documentation (http://interactivebrokers.github.io/tws-api/index.html) 

In the above code, we have used the historicalData() Ewrapper function in our trading application. This function handles the response from TWS once we execute the app.reqHistoricalData() function. The TWS returns the reqID and bar information which can be manipulated in the historicalData() function. In the above code, I am simply printing out the received data but you can store the data in your desired data structure (topic for later). Below is the partial output that we receive once we run the updated code. 

Quick Introduction to the Historical Data API 

Let us now change our contract object and try to extract historical data for an expired future contract. Let me change the lastTradeDateOrContractMonth attribute of the contract object to a date in the past. 

Below is the output when we run our updated code. 

As expected, TWS does not identify the expired Future contract. Thankfully, all we need to do here is add another attribute, includeExpired, to our contract object which will instruct TWS not to be facetious and provide the data for the expired contract. Please note that IB keeps data of expired contracts going back only a few years (I think it is two years but you may want to double check). 

Below is the partial output with the updated code. 

Summary & Next Steps 

In this article we gained an understanding of the TWS API tools required to extract historical data. We also gained proficiency in creating and tweaking the contract object and learnt how to extract historical data for both live and expired futures contracts. In the follow up article, we will discuss historical data for options which has some additional challenges. 

The author can be reached at contact@rasuquant.com 

If you found this article interesting, you may consider enrolling in my Interactive Brokers API centric courses on https://learning.rasuquant.com. The courses range from basics of TWS API to advanced concepts that will help you backtest and build fully automated trading applications on Interactive Broker’s platform.