How to get Tweets using Python and Twitter API v2 – Part III

See Part I for an overview of the Tweepy client for Twitter API v2, and Part II to learn how to get Tweet(s) with Tweet Id(s) using client.

Pagination in client

We have mentioned earlier that the max_results parameter while searching tweets can have a maximum value of 100. So what do we do if we need more than 100 tweets? The answer is pagination.

Pagination in Client is similar to how we used Cursors for the API. Let us see an example of how we can fetch 1000 tweets. We have taken the same query that we used in the previous section.

Want another example? Here you go.

We have covered some of the important methods to fetch data from Twitter. There are many more to explore. If you would like to read about all the methods that are available, please read the official Client documentation ⁽⁷⁾.

The methods for the Client return a Response object with the results. In the subsequent sections, we will look at how we can save the search results to different data formats for analysis.

---

Using expansions to get user and media information

The Twitter API provides expansions ⁽⁸⁾ as a means to expand the payload we get from a search for a user or tweet lookup. Expansions help include additional data in the same response without the need for separate queries.

We can expand on the attachments.media_keys to view the media object, author_id to view the user object, and referenced_tweets.id to view the Tweet object the originally requested Tweet was referencing. The expanded objects are nested in the includes object.

Let us look at an example to get the user and media information using expansion:

---

Writing the search results to a text file

Let us now save the search results of a query to a text file.

---

Putting the search results into a DataFrame

Pandas DataFrames are great for working with a large amount of data. So let us save our search results as a Pandas DataFrame. If you would like to save this data in a csv file, you can use the Pandas to_csv() method.

---

Twitter API v2 GitHub

We have attempted to cover most of the common use cases for pulling Twitter data. If you want to explore the Twitter API v2 further, this Twitter Developer Platform Resources ⁽⁹⁾ repository is a great place to start. This repository has many sample codes ⁽¹⁰⁾ for the versatile functionality offered by the Twitter API v2.

---

Conclusion

In this two-part blog series, we explored how we can pull data from Twitter using the API and Client interfaces of the Tweepy library. We also looked at Twitter APIs and their functionalities.

Now that we have nicely organised data in Pandas DataFrames, we can perform natural language processing on it. Want to learn more? Don’t forget about our great course on Twitter Sentiment Analysis, check it out and go live trading!

Till then, happy coding!

Visit QuantInsti for additional insights on this topic: https://blog.quantinsti.com/twitter-api-v2/.