How use Python retry decorator function with API
An ever-increasing proportion of a typical company’s infrastructure is moving to the cloud. More companies are shifting towards a micro-service approach. These paradigm shifts away from local to cloud-based means that you probably also have faced a situation where you had to pull data from somewhere or write data somewhere that is not your local computer.
On a small scale, there rarely are problems around that. If some extraction or writeback fails, you would typically notice that and would be able to remedy the mistake. But, as you move towards larger-scale operations and potentially hundreds of thousands of transactions, you don’t want to get screwed over by a temporary drop of internet-connection, too many concurrent writes, a temporarily unresponsive source system, or god knows what else.
I found a very simple retry-decorator to be a saving grace in countless situations like that. Most of my projects, at one point or another, end up having the retry decorator in some util module.
Retry decorator function
from functools import wraps
import time
import logging
import random
logger = logging.getLogger(__name__)
def retry(exceptions, total_tries=4, initial_wait=0.5, backoff_factor=2, logger=None):
"""
calling the decorated function applying an exponential backoff.
Args:
exceptions: Exeption(s) that trigger a retry, can be a tuple
total_tries: Total tries
initial_wait: Time to first retry
backoff_factor: Backoff multiplier (e.g. value of 2 will double the delay each retry).
logger: logger to be used, if none specified print
"""
def retry_decorator(f):
@wraps(f)
def func_with_retries(*args, **kwargs):
_tries, _delay = total_tries + 1, initial_wait
while _tries > 1:
try:
log(f'{total_tries + 2 - _tries}. try:', logger)
return f(*args, **kwargs)
except exceptions as e:
_tries -= 1
print_args = args if args else 'no args'
if _tries == 1:
msg = str(f'Function: {f.__name__}\n'
f'Failed despite best efforts after {total_tries} tries.\n'
f'args: {print_args}, kwargs: {kwargs}')
log(msg, logger)
raise
msg = str(f'Function: {f.__name__}\n'
f'Exception: {e}\n'
f'Retrying in {_delay} seconds!, args: {print_args}, kwargs: {kwargs}\n')
log(msg, logger)
time.sleep(_delay)
_delay *= backoff_factor
return func_with_retries
return retry_decorator
def log(msg, logger=None):
if logger:
logger.warning(msg)
else:
print(msg)
@retry(TimeoutError, total_tries=2, logger=logger)
def test_func(*args, **kwargs):
rnd=random.random()
print(f'random number is: {rnd}')
if rnd < .2:
raise ConnectionAbortedError('Connection was aborted :-(')
elif rnd < .4:
raise ConnectionRefusedError('Connection was refused :-/')
elif rnd < .6:
raise ConnectionResetError('Guess the connection was reset')
elif rnd < .8:
raise TimeoutError('This took too long')
else:
return 'Yaba daba doo'
if __name__ == '__main__':
# wrapper = retry((ConnectionAbortedError), tries=3, delay=.2, backoff=1, logger=logger)
# wrapped_test_func = wrapper(test_func)
# print(wrapped_test_func('hi', 'bye', hi='ciao'))
wrapper_all_exceptions = retry(Exception, total_tries=8, logger=logger)
wrapped_test_func = wrapper_all_exceptions(test_func)
print(wrapped_test_func('hi', 'bye', hi='ciao'))
Wrapping a wrapped function. That is some inception stuff right there. But bear with me, it is not that complicated!
Let’s walk through the code step by step:
-
Outmost function retry: This parameterizes our decorator, i.e. what are the exceptions we want to handle, how often do we want to try, how long do we wait between tries, and what is our exponential backoff-factor (i.e. with what number do we multiply the waiting time each time we fail).
-
retry_decorator: This is the parametrized decorator, which is being returned by our retry function. We are decorating the function within the retry_decorator with
@wraps. Strictly speaking, this is not necessary when it comes to functionality. This wrapper updates the__name__and__doc__of the wrapped function (if we didn't do that our function__name__would always be func_with_retries) -
func_with_retries applies the retry logic. This function wraps the function calls in try-except blocks and implements the exponential backoff wait and some logging.
Usage
@retry(Exception, total_tries=6, logger=logger)
def test_func(*args, **kwargs):
rnd=random.random()
print(f'random number is: {rnd}')
if rnd < .2:
raise ConnectionAbortedError('Connection was aborted :-(')
elif rnd < .4:
raise ConnectionRefusedError('Connection was refused :-/')
elif rnd < .6:
raise ConnectionResetError('Guess the connection was reset')
elif rnd < .8:
raise TimeoutError('This took too long')
else:
return 'Yaba daba doo'
Alternatively, a little bit more specific:
@retry(TimeoutError, total_tries=6, logger=logger)
def test_func(*args, **kwargs):
rnd=random.random()
print(f'random number is: {rnd}')
if rnd < .2:
raise ConnectionAbortedError('Connection was aborted :-(')
elif rnd < .4:
raise ConnectionRefusedError('Connection was refused :-/')
elif rnd < .6:
raise ConnectionResetError('Guess the connection was reset')
elif rnd < .8:
raise TimeoutError('This took too long')
else:
return 'Yaba daba doo'
Calling the decorated function and running into errors would then lead to something like this:
>>> print(test_func('Hi', 'Bye', first_param='first'))
1. try:
random number is: 0.1083287469475479
Function: test_func
Exception: Connection was aborted :-(
Retrying in 0.5 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
2. try:
random number is: 0.6873538483235806
Function: test_func
Exception: This took too long
Retrying in 1.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
3. try:
random number is: 0.4933965994051601
Function: test_func
Exception: Guess the connection was reset
Retrying in 2.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
4. try:
random number is: 0.951438953536428
Yaba daba doo
Or with error after 6th tries:
>>> print(test_func('Hi', 'Bye', first_param='first'))
1. try:
random number is: 0.18813199271094727
Function: test_func
Exception: Connection was aborted :-(
Retrying in 0.5 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
2. try:
random number is: 0.752460211423362
Function: test_func
Exception: This took too long
Retrying in 1.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
3. try:
random number is: 0.373256392319842
Function: test_func
Exception: Connection was refused :-/
Retrying in 2.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
4. try:
random number is: 0.4914540817108365
Function: test_func
Exception: Guess the connection was reset
Retrying in 4.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
5. try:
random number is: 0.09416797403173982
Function: test_func
Exception: Connection was aborted :-(
Retrying in 8.0 seconds!, args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
6. try:
random number is: 0.7535716152483288
Function: test_func
Failed despite best efforts after 6 tries.
args: ('Hi', 'Bye'), kwargs: {'first_param': 'first'}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 18, in func_with_retries
File "<stdin>", line 12, in test_func
TimeoutError: This took too long
