Excerpt
As analysts, we spend a lot of time processing, transforming and inferring data, we handle large amounts of data and devote a great deal of time to its analysis and manipulation.
It is convenient to have a mechanism that allows us to save the processed data for future retrieval without going through the same costly process again. Pickle is a utility that allows us to save Python objects in a binary file. In other words:
Pickle allows us to save time.
The need for Pickle Python
When we process large amounts of data in our analysis and backtesting, the machine needs a few hours, if not days, to process all the information.
The backtesting of a large portfolio of financial assets with historical data running into decades or the training of our ML algorithms are heavy processes from the point of view of the machine time needed to digest the data.
Repeating this procedure over and over again, most of the time, is pointless and a waste of time and resources. So it is convenient to have a mechanism that allows us to save the processed data for future retrieval without having to repeat the same costly process.
In Python there are multiple mechanisms and formats such as plain text files, binary files and structured and unstructured databases.
Among the most popular plain text files are csv (comma separated values), json (JavaScript Object Notation) or xml (eXtended Markup Language). The main feature of plain text files is that they are human-readable and can be exchanged between machines.
Structured and unstructured databases are able to store large amounts of information, relate data to each other and provide fast and accurate answers to queries.
Finally, we can use binary files to store the information. These files are not human-readable since they store bytes of information that can only be understood by machines.
Their main characteristic is the speed of storage/retrieval and the small size compared to the previous ones. Pickle is a utility that allows us to save Python objects in a binary file.
What is Pickle Python?
From the official documentation, the technical explanation about Pickle Python is as follows:
The pickle module implements binary protocols for serializing and de-serializing a Python object structure.
“Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy.
Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” 1 or “flattening”; however, to avoid confusion, the terms used here are “pickling” and “unpickling”.
For simplicity, we can say that Pickle stores and retrieves Python objects to/from the machine’s RAM.
It is important to remember here that in Python even variables are objects and that regardless of where the data we are handling comes from, the information resides in the machine’s volatile memory, also called RAM (Random Access Memory).
Unless we save this information in a storage system such as a hard disk, in any file format kind or database, the information is lost at the end of the Python session.
Example of Pickle Python
Let’s suppose the following scenarios.
Pickle Python Scenario 1
Pickle Python example – High level abstraction
The figure shows a very high level abstraction typically seen in a ML project.
The ETL (Extraction, Transformation and Load) is the tool for:
- Extract or fetch data from the data source,
- Transform the data by cleaning, sanitizing, checking, resumes, inferences, relations, etc. and finally
- Load in a database, save csv/hdf5 files or load into the model directly.
The Model Training is the most cumbersome process from the point of view of CPU time, it is also a very cumbersome process from the analyst’s point of view as the model requires adjustments until it is trained.
Once the model has been trained and adjusted, it is necessary to test the model‘s performance and verify if it fits the training provided.
Visit QuantInsti to read the full article: https://blog.quantinsti.com/pickle-python/
Disclosure: Interactive Brokers
Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.
This material is from QuantInsti and is being posted with its permission. The views expressed in this material are solely those of the author and/or QuantInsti and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.