Python Requests Library

Python Requests Library

The prior tutorial covered some ways to access web services in Python using the urllib module which is part of the Python standard library. In this tutorial, we’ll take a look at the Python Requests library, which is a more powerful and user-friendly alternative to urllib. Python Requests simplifies and improves on the built-in urllib modules. Python Requests is a full-featured library with more features than we can cover here, but we’ll look at the most important ones to know about. Unlike urllib, Requests does not come packaged with a standard Python distribution. Since we are familiar with PIP and python virtual environments, it will be pretty easy to get Requests installed so we can test out its features. The API for Requests is user-friendly and provides the ability to work with all of the available HTTP operations. Each HTTP operation actually maps to the same method name for the Requests API. With Requests, can handle sending data parameters, headers, and will try to automatically decode web responses for you. This works especially well if the returned data is in JSON format.


Install Requests

To get the Requests library installed in our Python virtual environment we can type pip install requests.

install python requests in virtual env
When you type pip install requests, you’ll see that the pip package manager goes ahead and downloads Requests and any supporting dependencies that might be needed. By typing pip freeze after the downloads complete, we can see that in addition to requests, the certifi, chardet, idna, and urllib3 packages are installed.


Making A Simple Request

A GET request is the most simple type of request you can make with the library. To do so you use the get method of the library and you pass the URL that you want to fetch data from. The syntax below represents a simple GET request, and the table contains some of the common parameters you can use with each request.

response = requests.get(url)

params Key-value pairs that will be sent in the query string
headers Dictionary of header values to send along with the request
auth Authentication tuple to enable different forms of authentication
timeout Value in seconds to wait for the server to respond

Testing Requests

In our Pycharm sandbox we can add a new file to test out some code from the Requests library.

pycharm python requests

Making A GET Request

requests_tutorial.py

Response Code: +----- 200 -----+


Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 18:03:20 GMT', 'Content-Type': 'application/xml', 'Content-Length': '522', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+
<?xml version='1.0' encoding='us-ascii'?>

<!--  A SAMPLE set of slides  -->

<slideshow 
    title="Sample Slide Show"
    date="Date of publication"
    author="Yours Truly"
    >

    <!-- TITLE SLIDE -->
    <slide type="all">
      <title>Wake up to WonderWidgets!</title>
    </slide>

    <!-- OVERVIEW -->
    <slide type="all">
        <title>Overview</title>
        <item>Why <em>WonderWidgets</em> are great</item>
        <item/>
        <item>Who <em>buys</em> WonderWidgets</item>
    </slide>

</slideshow>

Process finished with exit code 0

In our testing code we use a python function of printResponse() that wraps up the logic to print the Response Code, Headers, and Returned Data. These can be accessed on the response object using the status_code, headers, and text properties.

Including Parameters

In this snippet of code, we can send some parameters along with the request. First, we’ll change the URL we are using from http://httpbin.org/xml to http://httpbin.org/get. The GET endpoint at httpbin echoes back the content of the get request in JSON format. To add the parameters, we define a dictionary that holds simple key-value pairs and gives it the name of payload. We then issue the request using requests.get(url, params=payload). This does not need to be encoded as it did with the urllib library.

Response Code: +----- 200 -----+


Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 18:13:37 GMT', 'Content-Type': 'application/json', 'Content-Length': '410', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+
{
  "args": {
    "Cream": "True", 
    "Size": "Large", 
    "Sugar": "False"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.23.0", 
    "X-Amzn-Trace-Id": "Root=1-5e692a51-71b500ab1d13d674526bc5d0"
  }, 
  "origin": "192.168.10.1", 
  "url": "http://httpbin.org/get?Size=Large&Cream=True&Sugar=False"
}


Process finished with exit code 0

Making A POST Request

Now we want to test out making a POST request using the requests library. The same payload will be used to send the request and we’ll see the difference that makes when using POST vs GET. To make a POST request, we can use the .post() method of the requests class. Note that the second parameter to post() data instead of params like it was for the GET request. The requests library has method names that map directly to the standard Http verbs.

Response Code: +----- 200 -----+


Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 20:23:51 GMT', 'Content-Type': 'application/json', 'Content-Length': '526', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "Cream": "True", 
    "Size": "Large", 
    "Sugar": "False"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "33", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.23.0", 
    "X-Amzn-Trace-Id": "Root=1-5e6948d7-4b5b42c85acf7660e4e2c1a8"
  }, 
  "json": null, 
  "origin": "10.10.10.10", 
  "url": "http://httpbin.org/post"
}


Process finished with exit code 0

Sending Custom Headers

To test out sending a custom header, let’s change back to the http://httpbin.org/get endpoint and remove the data payload. We set up a customHeader variable and assign a dictionary that holds key-value pairs. Inside that dictionary, we can specify the User-Agent, which is often used to identify custom web applications.

Response Code: +----- 200 -----+


Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 20:46:31 GMT', 'Content-Type': 'application/json', 'Content-Length': '312', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Gardens-Delight-App / 1.0.1", 
    "X-Amzn-Trace-Id": "Root=1-5e694e27-6ade43401b07635c60af1748"
  }, 
  "origin": "1.2.3.4", 
  "url": "http://httpbin.org/get"
}


Process finished with exit code 0

Handling Errors With HTTPError

Things sometimes go wrong when you’re communicating with remote servers on the internet. A server could be down or a user mistypes a URL, or maybe a connection just simply times out. The Python code needs to be able to respond to these types of situations. To test out error states, httpbin provides a few different endpoints to simulate problems. First up, we’ll test out the 404 endpoint.

Http Error: 404 Client Error: NOT FOUND for url: http://httpbin.org/status/404

Process finished with exit code 0

Handling A Timeout

Request timed out: HTTPConnectionPool(host='httpbin.org', port=80): Read timed out. (read timeout=3)

Process finished with exit code 0

Authentication With Requests

Using authentication in Http requests is a common task. The Python Requests library has support for web-based authentication in the form of Basic Auth, Digest credentials, and Oauth. We can test some authentication requests using the httpbin website.

httpbin auth testing

In order to test basic authentication, we first need to specify the credentials to authorize against on the httpbin website.

httpbin basic auth setup

At httpbin we now have an endpoint setup that we can use for testing. The endpoint is https://httpbin.org/basic-auth/vegibit/secret based on the credentials we set above. If you are following along, you can use whatever username and password you like. So now we can test this out in our Python code using the HTTPBasicAuth class of the Requests library.

Response Code: +----- 200 -----+


Headers: +----------------------+
{'Date': 'Thu, 12 Mar 2020 14:36:41 GMT', 'Content-Type': 'application/json', 'Content-Length': '50', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+
{
  "authenticated": true, 
  "user": "vegibit"
}


Process finished with exit code 0

In the above test, we ran this in Pycharm and we can see the results. The JSON feedback shows that we have an authenticated status of true, and the user is vegibit. If we provide the incorrect password and send the request again, we now get some different feedback. A 401 unauthorized message is returned indicating that based on the credentials provided, we do not have access. In addition in the returned data area, we can see that there is none.

Response Code: +----- 401 -----+


Headers: +----------------------+
{'Date': 'Thu, 12 Mar 2020 14:41:42 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'WWW-Authenticate': 'Basic realm="Fake Realm"', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}


Returned data: +----------------------+


Process finished with exit code 0

This gives us a good idea of how basic authentication works using the Python Requests library.

Learn More About Requests

Python Requests Library Summary

In this tutorial, we learned about the Python Requests library. Requests builds on the features provided with the default urllib package in the Python Standard Library. Some of the manual tasks that are needed with urllib like encoding data parameters are done for you automatically in Requests. In addition, Requests will try to automatically decode the returned response based on its type. It has a simple API where each method name matches the available Http verbs. Developers can incorporate parameters, headers, and cookies as needed and all of the common types of authentication are supported.

  • Simple API – each HTTP verb is a method name
  • Makes working with parameters, headers, and cookies easier
  • Automatically decodes returned content
  • Automatically parses JSON content when detected
  • Handles redirects, timeouts, and errors
  • Supports authentication and sessions

Example Requests