Testing HTTP calls in Python

Many applications make calls to external services, or other services that are part of the application. Testing those HTTP calls can be challenging, but there are some different options available in Python.

Mocking

One option for testing your HTTP calls is to mock out your function that makes the HTTP call. This way, your function doesn’t make the HTTP call, since it’s replaced by a mock function that just returns whatever you want it to.

Here’s an example of mocking out your HTTP call:

import requests

class SomeClass:

  def __init__(self):
    self.data = self._fetch_data()

  def _fetch_data(self):
    r = requests.get('https://repository.library.brown.edu/api/collections/')
    return r.json()

  def get_collection_ids(self):
    return [c['id'] for c in self.data['collections']]

from unittest.mock import patch
MOCK_DATA = {'collections': [{'id': 1}, {'id': 2}]}

with patch.object(SomeClass, '_fetch_data', return_value=MOCK_DATA) as mock_method:
  thing = SomeClass()
  assert thing.get_collection_ids() == [1, 2]

Another mocking option is the responses package. Responses mocks out the requests library specifically, so if you’re using requests, you can tell the responses package what you want each requests call to return.

Here’s an example using the responses package (SomeClass is defined the same way as in the first example):

import responses
import json
MOCK_JSON_DATA = json.dumps({'collections': [{'id': 1}, {'id': 2}]})

@responses.activate
def test_some_class():
  responses.add(responses.GET,  'https://repository.library.brown.edu/api/collections/',
 body=MOCK_JSON_DATA,
 status=200,
 content_type='application/json'
 )
  thing = SomeClass()
  assert thing.get_collection_ids() == [1, 2]

test_some_class()

Record & Replay Data

A different type of solution is to use a package to record the responses from your HTTP calls, and then replay those responses automatically for you.

• VCR.py – VCR.py is a Python version of the Ruby VCR library, and it supports various HTTP clients, including requests.

Here’s a VCR.py example, again using SomeClass from the first example:

import vcr 
IDS = [674, 278, 280, 282, 719, 300, 715, 659, 468, 720, 716, 687, 286, 288, 290, 296, 298, 671, 733, 672, 334, 328, 622, 318, 330, 332, 625, 740, 626, 336, 340, 338, 725, 724, 342, 549, 284, 457, 344, 346, 370, 350, 656, 352, 354, 356, 358, 406, 663, 710, 624, 362, 721, 700, 661, 364, 660, 718, 744, 702, 688, 366, 667]

with vcr.use_cassette('vcr_cassettes/cassette.yaml'):
  thing = SomeClass()
  fetched_ids = thing.get_collection_ids()
  assert sorted(fetched_ids) == sorted(IDS)

• betamax – From the documentation: “Betamax is a VCR imitation for requests.” Note that it is more limited than VCR.py, since it only works for the requests package.

Here’s a betamax example (note: I modified the code in order to test it – maybe there’s a way to test the code with betamax without modifying it?):

import requests

class SomeClass:
    def __init__(self, session=None):
        self.data = self._fetch_data(session)

    def _fetch_data(self, session=None):
         if session:
             r = session.get('https://repository.library.brown.edu/api/collections/')
         else:
             r = requests.get('https://repository.library.brown.edu/api/collections/')
         return r.json()

    def get_collection_ids(self):
        return [c['id'] for c in self.data['collections']]


import betamax
CASSETTE_LIBRARY_DIR = 'betamax_cassettes'
IDS = [674, 278, 280, 282, 719, 300, 715, 659, 468, 720, 716, 687, 286, 288, 290, 296, 298, 671, 733, 672, 334, 328, 622, 318, 330, 332, 625, 740, 626, 336, 340, 338, 725, 724, 342, 549, 284, 457, 344, 346, 370, 350, 656, 352, 354, 356, 358, 406, 663, 710, 624, 362, 721, 700, 661, 364, 660, 718, 744, 702, 688, 366, 667]

session = requests.Session()
recorder = betamax.Betamax(
 session, cassette_library_dir=CASSETTE_LIBRARY_DIR
 )

with recorder.use_cassette('our-first-recorded-session', record='none'):
    thing = SomeClass(session)
    fetched_ids = thing.get_collection_ids()
    assert sorted(fetched_ids) == sorted(IDS)

Integration Test

Note that with all the solutions I listed above, it’s probably safest to cover the HTTP calls with an integration test that interacts with the real service, in addition to whatever you do in your unit tests.

Another possible solution is to test as much as possible with unit tests without testing the HTTP call, and then just rely on the integration test(s) to test the HTTP call. If you’ve constructed your application so that the HTTP call is only a small, isolated part of the code, this may be a reasonable option.

Here’s an example where the class fetches the data if needed, but the data can easily be put into the class for testing the rest of the functionality (without any mocking or external packages):

import requests

class SomeClass:

    def __init__(self):
        self._data = None

    @property
    def data(self):
        if not self._data:
            r = requests.get('https://repository.library.brown.edu/api/collections/')
            self._data = r.json()
        return self._data

    def get_collection_ids(self):
        return [c['id'] for c in self.data['collections']]


import json
MOCK_DATA = {'collections': [{'id': 1}, {'id': 2}]}

def test_some_class():
    thing = SomeClass()
    thing._data = MOCK_DATA
    assert thing.get_collection_ids() == [1, 2]

test_some_class()