Timing Python Processes

Articles From: TheAutomatic.net
Website: TheAutomatic.net


Blogger, TheAutomatic.net, and Senior Data Scientist

Timing Python processes is made possible with several different packages. One of the most common ways is using the standard library package, time, which we’ll demonstrate with an example. However, another package that is very useful for timing a process — and particularly telling you how far along a process has come — is tqdm. As we’ll show a little further down the posttqdm will actually print out a progress bar as a process is running.

Basic timing example

Suppose we want to scrape the HTML from some collection of links. In this case, we’re going to get a collection of URLs from Bloomberg’s homepage. To do this, we’ll use BeautifulSoup to get a list of full-path URLs. From the code below, this gives us a list of around 100 URLs. This first section of code should run pretty quickly; where timing a process comes in is if we wanted to cycle through some (or all) of these links and scrape the HTML from the respective pages.

# load packages
import time
from bs4 import BeautifulSoup
import requests
# get HTML of Bloomberg's website
html = requests.get("https://www.bloomberg.com/").content
# convert HTML to BeautifulSoup object
soup = BeautifulSoup(html, "lxml")
# find all links on Bloomberg's homepage
links = soup.find_all("a")
# get the URL of each link object
urls = [x.get("href") for x in links]
# filter out URLS that are None-values
urls = [x for x in urls if x is not None]
# get only the URLS with full path
with_http = [x for x in urls if "http" in x]
# check how many URLS we have

Let’s suppose we want to get the HTML of the first 50 webpages associated with our URLs. This could take a little bit of time, so let’s time it using the time package.

# get start epoch time
start = time.time()
# create an empty dictionary to store HTML
# from each page
html_store = {}
# get HTML from first 50 webpages
for url in with_http[:50]:
    html = requests.get(url).content
    html_store[url] = html
# get end epoch time
end = time.time()
# print number of seconds the above process took
print(end - start)

Above, we’re calling time.time(), which returns the current epoch time i.e. the number of seconds since January 1, 1970 (see here for reference). Since we call this immediately before running the for loop, and immediately after, we can just subtract these two values to get the number of seconds it took for our process to run. Hitting the 50 links took just under 28 seconds, though that time will vary depending on your internet connection.

How much longer will this loop last?

Although the above is useful in telling us how long our process takes, it doesn’t do anything to let us know how long the process is taking while it is actually running. We had to wait until the process is over to find out how long it took. There’s a solution to this conundrum using an amazing package, called tqdm. As mentioned above, the tqdm package provides a progress status as a process is running to show how far Python has come in the loop. Not only that, tqdm gives an estimate of how much longer the process has to run. This estimate gets updated as the program snippet runs. Let’s use it in our code from above.

# get tqdm method from tqdm package
from tqdm import tqdm
# create an empty dictionary to store HTML
# from each page
html_store = {}
# get HTML from first 50 webpages
for url in tqdm(with_http[:50]):
    html = requests.get(url).content
    html_store[url] = html

In the above code, we simply wrap the list of URLs we’re looping through in the tqdm method, which gets imported from the tqdm package.

The output from tqdm looks like this:

Above, the 18 represents that 18 seconds have passed so far in the loop, while the 13 represents how many seconds tqdm estimates is left in the process based upon on how it long the code has been running so far. The 29 / 50 shows that Python has hit 29 of the 50 links in the loop. At the end of the loop, the output looks like below:

We can also use tqdm with list or dictionary comprehensions. Let’s change our code from above to work as a dictionary comprehension.

html_store = {url : requests.get(url).content for url in tqdm(with_http[:50])}

Please check out my other posts by clicking one of the links on the right side of the page, or by perusing through the archives here: http://theautomatic.net/blog/.

Originally posted on TheAutomatic.net.

Leave a Reply

Your email address will not be published. Required fields are marked *

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from TheAutomatic.net and is being posted with its permission. The views expressed in this material are solely those of the author and/or TheAutomatic.net and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.