Python provides a lot of ways to ask questions about your code. Whether it's basic things like help() function, builtin functions like dir() or more advanced methods in inspect module - the tools are there to help you find the answers to your questions.
Let's find out what kinds of questions about our own code can Python answer for us and how it can help us during debugging sessions, dealing with type annotations, validating inputs and much more.
The Builtins
As already mentioned - there are a couple categories of introspection tools/functions in Python - let's start with the most basic one, which is the builtin functions.
Python includes a basic set of builtin function, most of which we already know - such as len(), range() or print() - there are however a couple obscure ones that can help us answer some questions about our code:
if "some_var" in locals():
... # some_var exists
if "some_var" in globals():
... # some_var exists
if hasattr(instance, "some_class_attr"):
... # instance.some_class_attr exists
The locals(), globals() and hasattr(), can help us find out whether local/global variable or class instance attribute exists.
Furthermore, we can use builtin functions to also check whether a variable is a function:
print(callable(some_func))
# True
# Prior to Python 3.2
print(hasattr(some_func, "__call__"))
# True
# Careful!
print(inspect.isfunction(sum))
# False
There are a couple ways to do that - the best option is to use callable(), if you're however running Python 3.1, then you'd have to check for presence of __call__ attribute using hasattr function instead. Last option would be to use isfunction() from inspect module, be careful with that one though, as it will return False for builtin functions such as sum, len or range because these are implemented in C, so they're not Python functions.
Another thing you might want to check is whether a variable is list (sequence) or a scalar:
import collections.abc
# Works with builtin types, but includes strings. Also doesn't work with NumPy arrays:
print(isinstance([1, 2, 3], collections.abc.Sequence)) # True
print(isinstance(np.array([1, 2, 3]), collections.abc.Sequence)) # False - wrong
print(isinstance("1234", collections.abc.Sequence)) # True - wrong
# Works also for NumPy arrays, but will return True for dictionaries
print(hasattr([1, 2, 3], "__len__")) # True
print(hasattr(np.array([1, 2, 3]), "__len__")) # True
print(hasattr({"a": 1}, "__len__")) # True - wrong (?)
print(isinstance([1, 2, 3], (collections.abc.Sequence, np.ndarray))) # True
print(isinstance(np.array([1, 2, 3]), (collections.abc.Sequence, np.ndarray))) # True
print(isinstance("1234", (collections.abc.Sequence, np.ndarray))) # True - wrong
The "obvious" solution is to use isinstance to check whether variable is an instance of abstract class Sequence. That however won't work with non-builtin types such as NumPy arrays. It also considers strings a sequence which is correct, but might not be desirable. Alternative is to use hasattr to check whether the variable has __len__ attribute, this will work for NumPy arrays, but will also return True for dictionaries. Last option is to pass a tuple of types to isinstance to customize the behavior to your needs.
We've already seen how we can use globals() to check if variable exists, we can however use it also to call function by a string:
def some_func():
print("Call me!")
func_name = "some_func"
globals()[func_name]()
# Call me!
Similar can be used for object (class) attributes with getattr(instance, "func_name")().
Last example with builtin functions uses locals() - let's say you have a function that takes a lot of arguments, all of which need to be passed to another function, and you don't want to write them all out. Well you can simply use locals() with the ** (destructuring) operator, like so:
def func(a, b, c, d, e):
other_func(**locals())
def other_func(a, b, c, d, e):
print(a, b, c, d, e)
func(1, 2, 3, 4, 5)
# Prints: 1 2 3 4 5
The above should give you a decent idea of what you can do with the builtin functions, it's however not an exhaustive list. There are couple more functions, such as dir() or vars(), so be sure to check out the docs to get a full picture.
Object Attributes
If the above builtins aren't giving you all the answers, then we can dig a little deeper. Every object in Python has a quite extensive set of attributes that can tell us more about a particular object.
These are essentially the values used to construct the output of help(object), so anything you find in output of help(object) can be pulled from object attribute. For example, you can find there object's doc string (.__doc__), function's source code (.__code__), or traceback/stack information (e.g. .tb_frame).
This can be useful - for example - if you need to know number of function argument (func.__code__.co_argcount), their names (func.__code__.co_varnames) or their default values (.__defaults__).
Messing with attributes is not ideal though, but there's a better way. Let's take a look at inspect module...
inspect Module
The inspect module leverages all the above attributes (and some more) to allow us to introspect our code more efficiently.
You might have already used the builtin dir() method to get all attributes of an object. inspect module has a similar one called getmembers():
import inspect
print(inspect.getmembers(SomeClass))
# [('__class__', <class 'type'>),
# ('__delattr__', <slot wrapper '__delattr__' of 'object' objects>),
# ('__dict__', mappingproxy({...})),
# ...,
# ('__init__', <slot wrapper '__init__' of 'object' objects>), ('__repr__', <slot wrapper '__repr__' of 'object' objects>),
# ('__weakref__', ..., ('some_func', <function SomeClass.some_func at 0x7f3fdf62e5e0>)]
methods = inspect.getmembers(SomeClass, lambda attr: not(inspect.ismethod(attr))) # No attributes
methods_filtered = [m for m in methods if not(m[0].startswith("__") and m[0].endswith("__"))]
# [('some_func', <function SomeClass.some_func at 0x7fc823f77430>), ...]
attributes = inspect.getmembers(SomeClass, lambda attr: not(inspect.isroutine(attr))) # No functions
attrs_filtered = [a for a in attributes if not(a[0].startswith("__") and a[0].endswith("__"))]
# [('some_var', 'value'), ...]
inspect.getmembers() has the advantage of providing second argument which can be used for filtering the attributes. Here we use it to filter out the variable attributes and functions respectively.
Another great use-case for inspect module is debugging. You can - for example - use it to debug state of a generator:
import inspect
def some_gen():
yield 1
print(inspect.isgeneratorfunction(some_gen))
# True
gen = some_gen()
print(inspect.getgeneratorstate(gen))
# GEN_CREATED
next(gen)
print(inspect.getgeneratorstate(gen))
# GEN_SUSPENDED
try:
next(gen)
except StopIteration:
pass
print(inspect.getgeneratorstate(gen))
# GEN_CLOSED
Here we define dummy generator by putting yield in a body of a function. We can then test whether it's really a generator using isgeneratorfunction(). We can also check its state using getgeneratorstate() which will return one of GEN_CREATED (not yet executed), GEN_RUNNING, GEN_SUSPENDED (waiting at yield) or GEN_CLOSED (consumed).
With help of inspect.signature you can also debug things related to function signature, such as mutable default arguments:
def some_func(var_with_default=[]):
var_with_default.append("value")
print(inspect.signature(some_func))
# (var_with_default=[])
some_func()
some_func()
some_func()
print(inspect.signature(some_func))
# (var_with_default=['value', 'value', 'value'])
It's common knowledge that you should not use mutable types for argument defaults, such as list, because it will get modified (mutated) during each function execution. Inspecting the function signature with inspect.signature() makes it very clear here.
While on the topic of argument defaults, we can also use the signature() function to read them:
def some_func(some_arg=42):
...
signature = inspect.signature(some_func)
print({
k: v.default
for k, v in signature.parameters.items()
if v.default is not inspect.Parameter.empty
})
# {'some_arg': 42}
# If you know the argument name:
print(inspect.signature(some_func).parameters["some_arg'].default)
# 42
This can be helpful if you want to pass the defaults of one function to another one. The above example shows that you can either iterate over all the arguments and pick out the ones that have non-empty default value, or if you know the argument name then you can query it directly.
Some more advanced uses of signature() function include injecting extra arguments to function using a decorator:
from functools import wraps
import inspect
def optional_debug(fn):
if "debug" in inspect.signature(fn).parameters:
raise TypeError('"debug" argument already defined')
@wraps(fn)
def wrapper(*args, debug=False, **kwargs):
if debug:
print("Calling", fn.__name__)
return fn(*args, **kwargs)
sig = inspect.signature(fn)
params = list(sig.parameters.values())
params.append(inspect.Parameter("debug",
inspect.Parameter.KEYWORD_ONLY,
default=False))
wrapper.__signature__ = sig.replace(parameters=params)
return wrapper
@optional_debug
def func(x):
return x
print(inspect.signature(func))
# (x, *, debug=False)
func(42, debug=True)
# Prints: Calling func
The above snippet defines a decorator called optional_debug which injects debug argument when applied to a function. It does this by first inspecting function signature and checking for presence of debug argument. If not present, it then replaces the original function signature with a new one with keyword-only argument appended.
Now, let's say you have a function that only declares *args and **kwargs for arguments. This makes the function very "general-purpose", but makes parameter checking quite messy. We can solve this with Signature and Parameter classes from inspect module:
from inspect import Signature, Parameter
params = [Parameter("first", Parameter.POSITIONAL_ONLY),
Parameter("second", Parameter.POSITIONAL_OR_KEYWORD),
Parameter("third", Parameter.KEYWORD_ONLY, default="default_value")]
# Options:
# POSITIONAL_ONLY (ones before "/")
# POSITIONAL_OR_KEYWORD
# VAR_POSITIONAL ("*args")
# KEYWORD_ONLY (ones after a "*" or "*args")
# VAR_KEYWORD ("**kwargs")
sig = Signature(params)
print(sig)
# (first, /, second, *, third=None)
def func(*args, **kwargs):
bound_values = sig.bind(*args, **kwargs)
for name, value in bound_values.arguments.items():
print(name, value)
func(10, "value")
# first 10
# second value
# func(second="value", third="another")
# TypeError: missing a required argument: 'first'
First, we define the expected parameters (params) using the Parameter class, specifying their types and defaults, after which we create signature from them. Inside the "general-purpose" function, we use bind method of signature to bind the provided *args and **kwargs to the prepared signature. If the parameters don't satisfy the signature, we receive an exception. If all is good, we can proceed to access the bound parameter using the return value of bind method.
The above worked for validating parameters inside a basic function, but what if we wanted to validate that a generator gets all the necessary parameters from a decorated function?
import functools
import inspect
def login_required(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
func_args = inspect.Signature.from_callable(fn).parameters
if "username" not in func_args:
raise Exception("Missing username argument")
# ... Perform authentication logic
return fn(*args, **kwargs)
return wrapper
@login_required
def handler(username):
...
handler() # Argument not provided...
# TypeError: handler() missing 1 required positional argument: 'username'
The above snippet shows an implementation of an authentication decorator that expects username parameter to be passed to the decorated function. To validate that the parameter is present, we use from_callable() method of Signature class which pulls the supplied parameters from the function. We then check whether username is in the returned tuple before performing any actual logic.
Moving on from the inspect.signature, you can also use the inspect module to introspect your source code files:
from pathlib import Path
import datetime
print(Path(inspect.getfile(some_func)).resolve())
# /home/martin/.../examples.py
print(Path(inspect.getfile(datetime.date)).resolve())
# /usr/lib/python3.8/datetime.py
Here we simply look up location (file) of some_func function using getfile. Same also works for builtin functions/objects, demonstrated above with datetime.date.
The inspect module also includes helpers for working with tracebacks. You can use it - for example - to find source file and line that's being currently executed:
from inspect import currentframe, getframeinfo, stack
frame_info = getframeinfo(currentframe())
frame_info = getframeinfo(stack()[0][0]) # Same as above
print(f"{frame_info.filename}: {frame_info.lineno}")
# examples.py: 235
def some_func():
# Equivalent to "getframeinfo(stack()[1][0]).lineno"
frame = currentframe()
print(f"Line: {frame.f_back.f_lineno}") # <- Line 243
some_func() # <- Line 245
# Prints: "Line: 245"
There are couple ways of doing that, the most straightforward is using getframeinfo(currentframe()), you can however also access the stack frames through stack() function - in that case you will have to index into the stack to find the correct frame.
Another option is to use currentframe() directly, in that case you will have to access the f_back and f_lineno to find the correct frame and line respectively.
Next thing you can do with traceback helpers, is to access the caller object:
class SomeClass:
def some_func(self):
some_other_class = SomeOtherClass()
some_other_class.some_other_func()
class SomeOtherClass:
def some_other_func(self):
import inspect
print(inspect.currentframe().f_back.f_locals["self"])
some_class = SomeClass()
some_class.some_func()
# <__main__.SomeClass object at 0x7f341fe173d0>
To do so, we use inspect.currentframe().f_back.f_locals['self'] which gives us access to the "parent" object. This should only ever be used for debugging - in case you need to access the caller object, you should pass it into the function as an argument, rather than crawling through the stack.
Last but not least, if you're using type hints in your code, then you might be familiar with typing.get_type_hints which helps you inspect type hints. This function however will often throw NameError (especially when called on a class), so if you're on Python 3.10 you should probably switch to using inspect.get_annotations which handles many edge cases for you:
import pytest
help(pytest.exit)
# Help on function exit in module _pytest.outcomes:
# exit(reason: str = '', returncode: Optional[int] = None, *, msg: Optional[str] = None) -> 'NoReturn'
# Exit testing process.
# ...
inspect.get_annotations(pytest.exit)
# {'reason': <class 'str'>, 'returncode': typing.Optional[int], 'msg': typing.Optional[str], 'return': 'NoReturn'}
For more details about accessing annotations in Python 3.10 and beyond, see docs.
Closing Thoughts
As we've seen, there are more than enough tools to introspect and interrogate your Python code. Some of these are quite obscure, and you might never need to use them, but it's good to be at aware that they exist, especially the ones that can be used for debugging.
If however the above isn't enough, or you just want to dive a bit deeper, then you might want to take a look at ast, which can be used to traverse and inspect the syntax tree or even the dis module used for disassembling Python bytecode.