Language

Multilingual content from IBKR

Close Navigation
Learn more about IBKR accounts
Understanding the LSTM Model

Understanding the LSTM Model

Posted June 7, 2024 at 11:02 am
Sang-Heon Lee
SHLee AI Financial Model

This post gives a brief introduction to the basic structure of LSTM (Long-Short Term Memory) model with an intuitive figures and explanations. Let’s interpret “gate” as “rate” or “weight” for grasping the concept of LSTM faster since we are not engineers but finance-related practitioners.

Understanding the LSTM model

I will explain the LSTM model and then talk about the RNN model by eliminating the LSTM gates one by one. Now let’s understand the structure of LSTM model.

Activation function : Sigmoid versus Tanh

LSTM uses two types of activation functions: sigmoid (S) and hyperbolic tangent or tanh (T). The sigmoid determines the ratio or rate of information and tanh normalizes information into the range between -1 and 1. These two activation function have the following responses respectively.

Input, Output (State) and Memory

LSTM is a function so that it has input (xt−1, ht−1) and output (ht). It is a variant of RNN model and has two temporal dependence mechanism: short-term memory (𝑡) and long-term memory (Ct).

These variables are summarized as follows.

  • Ct : memory cell
  • ht−1 : previous output or state
  • xt : input data or input sequence at time t (i.e. yt−1)
  • ht : hidden state or output (fitted y value)
  • yt : target data

The inner structure of LSTM

Given xt (current input) and ht−1 (previous state or output), the inner structure of LSTM is defined by a set of equations as follows.

The main component of LSTM is the memory cell (Ct) which penetrates through the whole layers. This is adjusted and added and finally reduced into output combined with the current and previous information.

At time t𝑡,

  • Ct-1 enters into the LSTM layer and some portion of it is deleted by 𝑓𝑡, which is the forget rate.
  • The current memory cell is constructed by the current and previous short-term information (xt, ht−1) and also some portion of it is selected by it, which is the input rate.
  • The updated memory cell (Ct) is determined by the sum of 1) and 2). This is essentially a weighted average like an updating of the Kalman filter and is transferred into the next LSTM layer.
  • As Ct contains information of the previous state, current input, and the memory cell, it determines the current output (xt) which is some portion of the updated memory cell (Ct) by adjusting the output rate (𝑜𝑡).

I use the word “rate” rather than “gate” since I think that “rate” has a bound between 0 and 1 (ratio or weight) and reflects the functionalities of three gates more intuitively.

These set of equations for the LSTM model can be illustrated by the following figure.

It is worth to note that the current information (xt) and the previous state (ht−1) are fed into the LSTM layer and convey short-term information and the memory cell governs the relatively long-term information.

RNN (Recurrent Neutral Network)

If the memory cell is absent, Ct-1Ct and 𝑓𝑡 can be deleted. This also leads to it being one (full weight) since  is not calculated. This reasoning leads to the following simple RNN equation.

Concluding Remarks

This post gives short introduction to the basic structure of LSTM model with an intuitive figures and explanations. The remaining job is to know how to implement LSTM or RNN model using Keras or Tensorflow. 

Originally posted on SHLee AI Financial Model blog.

Join The Conversation

If you have a general question, it may already be covered in our FAQs. If you have an account-specific question or concern, please reach out to Client Services.

Leave a Reply

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from SHLee AI Financial Model and is being posted with its permission. The views expressed in this material are solely those of the author and/or SHLee AI Financial Model and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

IBKR Campus Newsletters

This website uses cookies to collect usage information in order to offer a better browsing experience. By browsing this site or by clicking on the "ACCEPT COOKIES" button you accept our Cookie Policy.