Pagination in client
We have mentioned earlier that the max_results parameter while searching tweets can have a maximum value of 100. So what do we do if we need more than 100 tweets? The answer is pagination.
Pagination in Client is similar to how we used Cursors for the API. Let us see an example of how we can fetch 1000 tweets. We have taken the same query that we used in the previous section.
Want another example? Here you go.
We have covered some of the important methods to fetch data from Twitter. There are many more to explore. If you would like to read about all the methods that are available, please read the official Client documentation ⁽⁷⁾.
The methods for the Client return a Response object with the results. In the subsequent sections, we will look at how we can save the search results to different data formats for analysis.
Using expansions to get user and media information
The Twitter API provides expansions ⁽⁸⁾ as a means to expand the payload we get from a search for a user or tweet lookup. Expansions help include additional data in the same response without the need for separate queries.
We can expand on the attachments.media_keys to view the media object, author_id to view the user object, and referenced_tweets.id to view the Tweet object the originally requested Tweet was referencing. The expanded objects are nested in the includes object.
Let us look at an example to get the user and media information using expansion:
Writing the search results to a text file
Let us now save the search results of a query to a text file.
Putting the search results into a DataFrame
Pandas DataFrames are great for working with a large amount of data. So let us save our search results as a Pandas DataFrame. If you would like to save this data in a csv file, you can use the Pandas to_csv() method.
Twitter API v2 GitHub
We have attempted to cover most of the common use cases for pulling Twitter data. If you want to explore the Twitter API v2 further, this Twitter Developer Platform Resources ⁽⁹⁾ repository is a great place to start. This repository has many sample codes ⁽¹⁰⁾ for the versatile functionality offered by the Twitter API v2.
In this two-part blog series, we explored how we can pull data from Twitter using the API and Client interfaces of the Tweepy library. We also looked at Twitter APIs and their functionalities.
Now that we have nicely organised data in Pandas DataFrames, we can perform natural language processing on it. Want to learn more? Don’t forget about our great course on Twitter Sentiment Analysis, check it out and go live trading!
Till then, happy coding!
Visit QuantInsti for additional insights on this topic: https://blog.quantinsti.com/twitter-api-v2/.
Disclosure: Interactive Brokers
Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.
This material is from QuantInsti and is being posted with its permission. The views expressed in this material are solely those of the author and/or QuantInsti and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.
Disclosure: API Examples Discussed
Throughout the lesson, please keep in mind that the examples discussed are purely for technical demonstration purposes, and do not constitute trading advice. Also, it is important to remember that placing trades in a paper account is recommended before any live trading.