Background

When annotations were introduced to Python in PEP 3107 and PEP 526, they were defined simply as expressions that relate to a variable and are stored in a new special dictionary __annotations__. That is, writing var_name: expression just causes the Python interpreter to evaluate expression and store the result in the enclosing function’s or class’s __annotations__ dictionary under the var_name key.

This approach not only makes the implementation very simple, it also gives users the freedom to use annotations to attach various kinds of metadata to variables. One possibility are type hints, which specify what sets of values a variable is expected to contain and how a function can safely be called. These have proven to be very popular and have first received official support in PEP 484 and many additional features since.

But this implementation also creates challenges for type hints. While simple types such as the set of all integers can be spelled by just writing var: int, there is no built-in syntax for more complicated typing concepts like generics. When 484 introduced these, it thus had to repurpose an existing form of expressions, namely indexing, and define the needed operators on classes that should be usable in generic type annotations. Note that while conceptually the expressions some_dict["key"] and list[int] denote very different operations, dictionary indexing and specialization of a generic type, Python treats them exactly the same, an invocation of the __getitem__ operator.

PEP 649 and PEP 749 changed this behavior in some ways. Instead of the __annotations__ dictionary being built as the annotations are encountered, the annotation’s execution is now delayed until __annotations__ is actually accessed. Internally, a new method __annotate__ is synthesized that creates the dictionary by evaluating the class’s or function’s annotations. However, what remains unchanged is that annotations are still treated as ordinary expressions. They are evaluated just like any other expression in a different context, their execution just is delayed until they are needed.

Problems with the Current Approach

This behavior can cause problems where users want to use an expression in annotations in a way that clashes with the expression’s usual behavior. For example, consider literal types, i.e. types that consist of some particular set of literal values. Currently, these are written as e.g. Literal[1]. Many users of type annotations would prefer to drop the redundant Literal[], the context of it occurring in a type annotation already makes it clear that the 1 represents the literal type containing only 1. This also is reflected in other languages, such as TypeScript, implementing literal types that way.

The problem arises when these literal types are combined with other typing constructs. For example, the union type 1 | 2 | 3, representing values that are either 1, 2 or 3, cannot be properly evaluated at runtime. We would want it to evaluate to a UnionType containing references to 1, 2, 3. But since the __or__ special method is already implemented on int as the bitwise or operation, it will just be evaluated to 3. There is no way of recovering the 1 and 2 present in the annotation from the __annotation__ dictionary.

Similar issues prevent us from using display syntax to denote the built-in container types, forcing us to write set[int], list[int] and dict[int, str] instead of just {int}, [int] and {int: str}. For example, in the first case the display syntax causes a set object to be constructed rather than a special object representing a type. And set already implements __or__ to create a new set containing all elements of both arguments, rather than a Union object.

While all of these examples do have existing workarounds, the additional work required to spell these types and their readability issues do present real problems. Many Python users, particularly those new to typing, intuitively reach towards the easier and shorter syntax like (int, str) to denote a tuple type. The more verbose syntax also is cumbersome to understand when it occurs as a type argument for a generic type. A very common example are matrix libraries like numpy that use integer tuples to define a matrix’s shape. When creating such a matrix, you simply write ndarray(..., shape=(16, 1000)). But that matrix’s type is spelled as ndarray[tuple[Literal[16], Literal[1000]], ...]. A user unfamiliar with the internals of the Python type system is hard-pressed to see such an annotation and understand what information it is trying to tell them and why they have to use these seemingly redundant Literal tags.

This example also shows that the annotation specification’s intention of them being treated as any other expression is not being honored very well by this implementation. While an annotation can contain arbitrary expressions, this is only true from the interpreter’s point of view. The vast majority of users, which use type annotations have to switch from the familiar (16, 1000) syntax to the annotation-specific tuple[Literal[16], Literal[1000]].

There also are new typing features that are being held back by this requirement to create syntactical workarounds. For example, inline typed dictionaries could intuitively be defined inline as {"some_key": int, "other_key": str}. But this again does not work because the semantics of dict objects to not work properly in typing contexts. There is previous discussion of this, with the main hurdle to implementation being the runtime behavior. Other examples are conditional types, which let users write ternary statements in type definitions, like type SomeAlias[T] = int if issubclass(T, str) else str. Or extending integer literal types to support basic arithmetic operations. This would enable array libraries to properly track shapes across operations like concatenation. The existing work on this unfortunately concluded that while this is a very useful feature for many users, implementing it with current semantics is too verbose and cumbersome to gain much traction.

The AST Format

In this PEP, we address this issue by proposing to add a new format, AST. It instructs __annotate__ methods to return the abstract syntax tree of the annotation expressions, rather than their value. This will let consumers of annotations evaluate the AST using whatever semantics are appropriate to their domain rather than the built-in Python expression semantics.

Going back to our example, func will then have an __annotate__ method that conceptually might look something like this:

def __annotate__(self, format):
    if format <= 2:
        return {
            "arg": 1 | 2,
        }
    elif format == 5:
        return {
            "arg": BinOp(
                left=Constant(value=1),
                op=BitOr(),
                right=Constant(value=2),
            ),
        }
    else:
        raise NotImplementedError

The newly supported format 5 will be the value of the AST format. When it is passed, the function returns a similar dictionary that contains the annotation’s AST, as represented by the existing classes from the ast module. Users will then be able to analyze the full annotation by inspecting this AST object.

!!!! TUPLES !!!!

Analyzing the AST

The AST format intentionally leaves open many possibilities for users to treat the returned AST objects in domain-specific ways. That being said, there are some special considerations afforded to the most popular usage of annotations, type hints. In particular, the typing module will contain helper functions that internally use the AST format to retreive annotations and then parse the syntax trees into familiar typing objects.

In our example, we are using type annotations and are presuming that the typing spec will have been changed to let bare integer literals refer to the literal type containing them.

We do not propose adding direct support for any of the specific type annotation changes or additions that are discussed in this PEP and restrict this proposal to laying the necessary groundwork that enables them. Such changes are mentioned only as motivating examples to show why the AST format is needed.

In order to support such future additions to the typing spec, the typing.get_type_hints helper function will support a mode that uses the AST format to evaluate type annotations with the typing spec’s semantics.

There are some places other than annotations where users need to spell a type, for example, the first argument to typing.cast. These use cases will be covered by the existing practice of wrapping types in string literals. If such a type needs to be introspected at runtime, a new helper function typing.eval_type can be used, which internally uses the new format to evaluate a string under the typing spec’s semantics.

The AST Format

A new value called AST is added to the annotationlib.Format enum with value 5. Annotation functions do not have to support this format. If an annotation function is called with this format, it must return instances of ast.expr that do not contain ast.NamedExpr, ast.Yield, ast.YieldFrom or ast.Await nodes (the Python grammar already prohibits the use of these features in annotations).

Compiler-generated annotation functions will always support this format and return the abstract syntax tree of the class’s or function’s annotations. They will store the necessary AST data stored as tuple constants and then construct a new ast.expr object each time they are called. These objects will be functionally identical to the objects created by parsing the annotation code directly.

Annotate functions that are generated by library code will need to manually support the new format or choose to raise NotImplementedError() when it is requested. Most of these use cases synthesize new annotation functions by introspecting existing function or class objects and combining and/or modifying their return values. The new format makes this easier to do since the AST objects can always be created regardless of the existence of forward references or similar issues, they can be modified to denote the synthesized annotations using existing tools, and they can be evaluated to create the objects requested by other formats.

Helper Functions

The existing typing.get_type_hints function will be modified to accept a new, optional Boolean keyword argument use_type_syntax that defaults to False. When set to true, get_type_hints does not call the underlying annotation function with the requested format but always with Format.AST. The returned AST is then evaluated to an object, using the semantics of Python expressions, modified by rules that are defined in the typing spec. At the time this PEP is written, no such rules exist, and thus the AST is always evaluated normally.

When the keyword argument is set, any annotation expressions that are not valid type annotation expressions, as defined by the typing spec <https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions>__, cause get_type_hints to raise a SyntaxError().

When the typing spec is changed to define new type expressions, get_type_hints will be modified accordingly and return a particular object when it previously would have raised a SyntaxError on the same inputs. This is an exemption to the usual deprecation policy and will be documented as such. Other behavior of get_type_hints will not be affected; users can still rely on the stability of returned values and other raised exceptions.

A new helper method eval_type(tp: str, globals: Mapping[str, object], locals: Mapping[str, object], format: Format = Format.VALUE) -> TypeForm will be added to the typing module. It evaluates a string using the same rules that get_type_hints uses with the type-specific semantics. This function can be used to evaluate stringified types that are used in places other than annotations.

In order to ease the creation of synthesized annotation functions, a new helper function create_annotate_from_asts(annotations: dict[str, tuple[ast.expr, Mapping[str, object]]]) -> Callable[[Format], dict[str, object]] will be added to annotationlib. It creates an __annotate__ method that supports the VALUE, FORWARDREF, SOURCE and AST formats. The returned values are the objects of the correct type that would be created if each of the passed syntax trees were evaluated in the corresponding namespace.

Pseudocode

A function defined as def f(first: int, second: tuple[int, str]) will have a compiler-generated __annotate__ method that might look something like this were it written in Python:

def __annotate__(self, format):
    if format <= 2:
        return {
            "first": int,
            "second": tuple[int, str],
        }
    if format == 5:
        from ast import _ast_from_tuple
        return {
            "first": _ast_from_tuple((26, "int", 1, 1, 1, 14)),
            "second": _ast_from_tuple((...)),
        }
    raise NotImplementedError

A New Argument for Annotation Functions

An initial idea was to add a new optional argument to annotation functions, which instructs them to return objects of the specified format’s type but using type annotation-specific semantics. This was rejected since it means that the type-specific semantics need to be defined in the interpreter itself, which limits the kinds of semantics that are possible. It also locks the usage of typing features to the Python version that is being used, rather than allowing the usual backporting via typing_extensions.