See Part I, Part II and Part III to get started.
Q Table and Q Learning
Q table and Q learning might sound fancy, but it is a very simple concept.
At each time step, the RL agent needs to decide which action to take. What if the RL agent had a table which would tell her which action will give the maximum reward. Then simply select that action. This table is Q-table.
In the Q-table, the rows are the states (in this case, the days) and the actions are the columns (in this case, hold and sell). The values in this table are called the Q-values.
| Date | Sell | Hold |
| 23-07-2020 | 0.954 | 0.966 |
| 24-07-2020 | 0.954 | 0.985 |
| 27-07-2020 | 0.954 | 1.005 |
| 28-07-2020 | 0.954 | 1.026 |
| 29-07-2020 | 0.954 | 1.047 |
| 30-07-2020 | 0.954 | 1.068 |
| 31-07-2020 | 0.954 | 1.090 |
From the above Q-table, on 23 July, which action would RL agent take? Yes, that’s right. A “hold” action would be taken as it has a q-value of 0.966 which is greater than q-value of 0.954 for Sell action.
But how to create the Q-table?
Let’s create a Q-table with the help of an example. For simplicity sake, let us take the same example of price data from July 22 to July 31 2020. We have added the percentage returns and cumulative returns as shown below.
| Date | Closing Price | Percentage returns | Cumulative Returns |
| 22-07-2020 | 97.2 | ||
| 23-07-2020 | 92.8 | -4.53% | 0.95 |
| 24-07-2020 | 92.6 | -0.22% | 0.95 |
| 27-07-2020 | 94.8 | 2.38% | 0.98 |
| 28-07-2020 | 93.3 | -1.58% | 0.96 |
| 29-07-2020 | 95 | 1.82% | 0.98 |
| 30-07-2020 | 96.2 | 1.26% | 0.99 |
| 31-07-2020 | 106.3 | 10.50% | 1.09 |
You have bought one stock of Apple a few days back and you have no more capital left. The only two choices for you are “hold” or “sell”. As a first step, you need to create a simple reward table.
If we decide to hold, then we will get no reward till 31 July and at the end, we get a reward of 1.09. And if we decide to sell on any day then the reward will be cumulative returns up to that day. The reward table (R-table) looks like below. If we let the RL model choose from the reward table, the RL model will sell the stock and gets a reward of 0.95.
| State/Action | Sell | Hold |
| 22-07-2020 | 0 | 0 |
| 23-07-2020 | 0.95 | 0 |
| 24-07-2020 | 0.95 | 0 |
| 27-07-2020 | 0.98 | 0 |
| 28-07-2020 | 0.96 | 0 |
| 29-07-2020 | 0.98 | 0 |
| 30-07-2020 | 0.99 | 0 |
| 31-07-2020 | 1.09 | 1.09 |
But the price is expected to increase to $106 on July 31 resulting in a gain of 9%. Therefore, you should hold on to the stock till then. We have to represent this information. So that the RL agent can make better decisions to Hold rather than Sell.
How to go about it? To help us with this, we need to create a Q table. You can start by copying the reward table into the Q table and then calculate the implied reward using the Bellman equation on each day for Hold action.
Stay tuned for the next installment in which Ishan will demonstrate the Bellman equation.
Visit QuantInsti to download practical code: https://blog.quantinsti.com/reinforcement-learning-trading/.
Disclosure: Interactive Brokers
Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.
This material is from QuantInsti and is being posted with its permission. The views expressed in this material are solely those of the author and/or QuantInsti and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.