The prior tutorial covered some ways to access web services in Python using the urllib module which is part of the Python standard library. In this tutorial, we’ll take a look at the Python Requests library, which is a more powerful and user-friendly alternative to urllib. Python Requests simplifies and improves on the built-in urllib modules. Python Requests is a full-featured library with more features than we can cover here, but we’ll look at the most important ones to know about. Unlike urllib, Requests does not come packaged with a standard Python distribution. Since we are familiar with PIP and python virtual environments, it will be pretty easy to get Requests installed so we can test out its features. The API for Requests is user-friendly and provides the ability to work with all of the available HTTP operations. Each HTTP operation actually maps to the same method name for the Requests API. With Requests, can handle sending data parameters, headers, and will try to automatically decode web responses for you. This works especially well if the returned data is in JSON format.
Install Requests
To get the Requests library installed in our Python virtual environment we can type pip install requests.
When you type pip install requests, you’ll see that the pip package manager goes ahead and downloads Requests and any supporting dependencies that might be needed. By typing pip freeze after the downloads complete, we can see that in addition to requests, the certifi, chardet, idna, and urllib3 packages are installed.
Making A Simple Request
A GET request is the most simple type of request you can make with the library. To do so you use the get method of the library and you pass the URL that you want to fetch data from. The syntax below represents a simple GET request, and the table contains some of the common parameters you can use with each request.
response = requests.get(url)
| params | Key-value pairs that will be sent in the query string |
| headers | Dictionary of header values to send along with the request |
| auth | Authentication tuple to enable different forms of authentication |
| timeout | Value in seconds to wait for the server to respond |
Testing Requests
In our Pycharm sandbox we can add a new file to test out some code from the Requests library.
Making A GET Request
requests_tutorial.py
import requests
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
# Use requests to issue an HTTP GET request
url = 'http://httpbin.org/xml'
resp = requests.get(url)
printResponse(resp)
Response Code: +----- 200 -----+
Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 18:03:20 GMT', 'Content-Type': 'application/xml', 'Content-Length': '522', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
<?xml version='1.0' encoding='us-ascii'?>
<!-- A SAMPLE set of slides -->
<slideshow
title="Sample Slide Show"
date="Date of publication"
author="Yours Truly"
>
<!-- TITLE SLIDE -->
<slide type="all">
<title>Wake up to WonderWidgets!</title>
</slide>
<!-- OVERVIEW -->
<slide type="all">
<title>Overview</title>
<item>Why <em>WonderWidgets</em> are great</item>
<item/>
<item>Who <em>buys</em> WonderWidgets</item>
</slide>
</slideshow>
Process finished with exit code 0
In our testing code we use a python function of printResponse() that wraps up the logic to print the Response Code, Headers, and Returned Data. These can be accessed on the response object using the status_code, headers, and text properties.
Including Parameters
In this snippet of code, we can send some parameters along with the request. First, we’ll change the URL we are using from http://httpbin.org/xml to http://httpbin.org/get. The GET endpoint at httpbin echoes back the content of the get request in JSON format. To add the parameters, we define a dictionary that holds simple key-value pairs and gives it the name of payload. We then issue the request using requests.get(url, params=payload). This does not need to be encoded as it did with the urllib library.
import requests
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
# Send some parameters to the URL via a GET request
# Requests handles this for you, no manual encoding
payload = {'Size': 'Large', 'Cream': True, 'Sugar': False}
url = 'http://httpbin.org/get'
resp = requests.get(url, params=payload)
printResponse(resp)Response Code: +----- 200 -----+
Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 18:13:37 GMT', 'Content-Type': 'application/json', 'Content-Length': '410', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
{
"args": {
"Cream": "True",
"Size": "Large",
"Sugar": "False"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.23.0",
"X-Amzn-Trace-Id": "Root=1-5e692a51-71b500ab1d13d674526bc5d0"
},
"origin": "192.168.10.1",
"url": "http://httpbin.org/get?Size=Large&Cream=True&Sugar=False"
}
Process finished with exit code 0
Making A POST Request
Now we want to test out making a POST request using the requests library. The same payload will be used to send the request and we’ll see the difference that makes when using POST vs GET. To make a POST request, we can use the .post() method of the requests class. Note that the second parameter to post() data instead of params like it was for the GET request. The requests library has method names that map directly to the standard Http verbs.
import requests
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
# Send some parameters to the URL via a GET request
# Requests handles this for you, no manual encoding
payload = {'Size': 'Large', 'Cream': True, 'Sugar': False}
url = 'http://httpbin.org/post'
resp = requests.post(url, data=payload)
printResponse(resp)Response Code: +----- 200 -----+
Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 20:23:51 GMT', 'Content-Type': 'application/json', 'Content-Length': '526', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
{
"args": {},
"data": "",
"files": {},
"form": {
"Cream": "True",
"Size": "Large",
"Sugar": "False"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "33",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.23.0",
"X-Amzn-Trace-Id": "Root=1-5e6948d7-4b5b42c85acf7660e4e2c1a8"
},
"json": null,
"origin": "10.10.10.10",
"url": "http://httpbin.org/post"
}
Process finished with exit code 0
Sending Custom Headers
To test out sending a custom header, let’s change back to the http://httpbin.org/get endpoint and remove the data payload. We set up a customHeader variable and assign a dictionary that holds key-value pairs. Inside that dictionary, we can specify the User-Agent, which is often used to identify custom web applications.
import requests
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
# Pass a custom header to the server
url = "http://httpbin.org/get"
customHeader = {'User-Agent': 'Gardens-Delight-App / 1.0.1'}
resp = requests.get(url, headers=customHeader)
printResponse(resp)
Response Code: +----- 200 -----+
Headers: +----------------------+
{'Date': 'Wed, 11 Mar 2020 20:46:31 GMT', 'Content-Type': 'application/json', 'Content-Length': '312', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Gardens-Delight-App / 1.0.1",
"X-Amzn-Trace-Id": "Root=1-5e694e27-6ade43401b07635c60af1748"
},
"origin": "1.2.3.4",
"url": "http://httpbin.org/get"
}
Process finished with exit code 0
Handling Errors With HTTPError
Things sometimes go wrong when you’re communicating with remote servers on the internet. A server could be down or a user mistypes a URL, or maybe a connection just simply times out. The Python code needs to be able to respond to these types of situations. To test out error states, httpbin provides a few different endpoints to simulate problems. First up, we’ll test out the 404 endpoint.
import requests
from requests.exceptions import HTTPError, Timeout
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
try:
url = 'http://httpbin.org/status/404'
resp = requests.get(url)
resp.raise_for_status()
printResponse(resp)
except HTTPError as error:
print(f'Http Error: {error}')
except Timeout as error:
print(f'Request timed out: {error}')Http Error: 404 Client Error: NOT FOUND for url: http://httpbin.org/status/404 Process finished with exit code 0
Handling A Timeout
import requests
from requests.exceptions import HTTPError, Timeout
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
try:
url = 'http://httpbin.org/delay/5'
resp = requests.get(url, timeout=3)
resp.raise_for_status()
printResponse(resp)
except HTTPError as error:
print(f'Http Error: {error}')
except Timeout as error:
print(f'Request timed out: {error}')Request timed out: HTTPConnectionPool(host='httpbin.org', port=80): Read timed out. (read timeout=3) Process finished with exit code 0
Authentication With Requests
Using authentication in Http requests is a common task. The Python Requests library has support for web-based authentication in the form of Basic Auth, Digest credentials, and Oauth. We can test some authentication requests using the httpbin website.
In order to test basic authentication, we first need to specify the credentials to authorize against on the httpbin website.
At httpbin we now have an endpoint setup that we can use for testing. The endpoint is https://httpbin.org/basic-auth/vegibit/secret based on the credentials we set above. If you are following along, you can use whatever username and password you like. So now we can test this out in our Python code using the HTTPBasicAuth class of the Requests library.
import requests
from requests.auth import HTTPBasicAuth
def printResponse(resp):
print(f'Response Code: +----- {resp.status_code} -----+')
print('n')
print('Headers: +----------------------+')
print(resp.headers)
print('n')
print('Returned data: +----------------------+')
print(resp.text)
# Access a URL that requires authentication - the format of this
# URL is that you provide the username/password to auth against
url = 'https://httpbin.org/basic-auth/vegibit/secret'
# Create a credentials object using HTTPBasicAuth
credentials = HTTPBasicAuth('vegibit', 'secret')
# Issue the request with the authentication credentials
resp = requests.get(url, auth=credentials)
printResponse(resp)
Response Code: +----- 200 -----+
Headers: +----------------------+
{'Date': 'Thu, 12 Mar 2020 14:36:41 GMT', 'Content-Type': 'application/json', 'Content-Length': '50', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
{
"authenticated": true,
"user": "vegibit"
}
Process finished with exit code 0
In the above test, we ran this in Pycharm and we can see the results. The JSON feedback shows that we have an authenticated status of true, and the user is vegibit. If we provide the incorrect password and send the request again, we now get some different feedback. A 401 unauthorized message is returned indicating that based on the credentials provided, we do not have access. In addition in the returned data area, we can see that there is none.
Response Code: +----- 401 -----+
Headers: +----------------------+
{'Date': 'Thu, 12 Mar 2020 14:41:42 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'WWW-Authenticate': 'Basic realm="Fake Realm"', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
Returned data: +----------------------+
Process finished with exit code 0
This gives us a good idea of how basic authentication works using the Python Requests library.
Learn More About Requests
- How To Get Started With The Requests Library In Python (digitalocean.com)
- Python Requests Tutorial (codementor.io)
- Python Requests (zetcode.com)
- Python Requests Tutorial (edureka.co)
- Making Http Requests In Python (datacamp.com)
- Python Requests Tutorial (nitratine.net)
- How To Perform Http Requests With Python Request Library (linuxconfig.org)
- Python Requests (realpython.com)
- Using Requests In Python (pythonforbeginners.com)
- Install Requests Library In Python (agiratech.com)
- Python Requests Tutorial With Example Requests Module (appdividend.com)
- Zetcode Python Requests (zetcode.com)
Python Requests Library Summary
In this tutorial, we learned about the Python Requests library. Requests builds on the features provided with the default urllib package in the Python Standard Library. Some of the manual tasks that are needed with urllib like encoding data parameters are done for you automatically in Requests. In addition, Requests will try to automatically decode the returned response based on its type. It has a simple API where each method name matches the available Http verbs. Developers can incorporate parameters, headers, and cookies as needed and all of the common types of authentication are supported.
- Simple API – each HTTP verb is a method name
- Makes working with parameters, headers, and cookies easier
- Automatically decodes returned content
- Automatically parses JSON content when detected
- Handles redirects, timeouts, and errors
- Supports authentication and sessions
Example Requests
result = requests.get('http://example.com')
result = requests.put('http://example.com/put', data = {'key':'value'})
result = requests.delete('http://example.com/delete')
result = requests.head('http://example.com/head')
result = requests.options('http://example.com/options')
Create and Manage Threads with the Python threading Module
Digging Deep Into Requests
Interacting with the Web Using Requests
Authentication In Python Requests
How To Create A UUID In Python
Unit Testing Your Python Code with the unittest Module
How To Perform Web Scraping With Python and Beautiful Soup
What Is Botocore