Skip to content

Conformance suite: Minor tweaks to several assertions for better compatibility with ty #2178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AlexWaygood wants to merge 6 commits into
base: main
Choose a base branch
from
+124 −93
Open

Conformance suite: Minor tweaks to several assertions for better compatibility with ty #2178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
11875d9
Conformance suite: Minor tweaks to several assertions for better comp…
AlexWaygood Feb 16, 2026
652dde5
update results for other type checkers
AlexWaygood Feb 16, 2026
7b6a5d2
revert some questionable changes to tests around type-parameter defaults
AlexWaygood Feb 16, 2026
88bf117
Apply suggestion from @rchen152
AlexWaygood Feb 16, 2026
9955ae1
Mark Zuban as partially conformant with `generics_scoping.toml`
AlexWaygood Feb 16, 2026
d59f132
rename tag
AlexWaygood Feb 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 26 additions & 22 deletions conformance/results/mypy/generics_scoping.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,30 @@
output = """
generics_scoping.py:29: error: Argument 1 to "meth_2" of "MyClass" has incompatible type "str"; expected "int" [arg-type]
generics_scoping.py:50: error: Type variable "generics_scoping.S" is unbound [valid-type]
generics_scoping.py:50: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:50: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:54: error: Type variable "generics_scoping.S" is unbound [valid-type]
generics_scoping.py:54: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:54: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:65: error: Free type variable expected in Generic[...] [misc]
generics_scoping.py:75: error: Type variable "T" is bound by an outer class [valid-type]
generics_scoping.py:78: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:78: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:78: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:87: error: Can't use bound type variable "T" to define generic alias [valid-type]
generics_scoping.py:94: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:94: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:94: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:95: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:95: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:95: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:96: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:96: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:96: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:16: error: Expression is of type "int", not "Literal[1]" [assert-type]
generics_scoping.py:20: error: Expression is of type "str", not "Literal['a']" [assert-type]
generics_scoping.py:34: error: Argument 1 to "meth_2" of "MyClass" has incompatible type "str"; expected "int" [arg-type]
generics_scoping.py:50: error: Expression is of type "str", not "Literal['abc']" [assert-type]
generics_scoping.py:54: error: Expression is of type "bytes", not "Literal[b'abc']" [assert-type]
generics_scoping.py:61: error: Type variable "generics_scoping.S" is unbound [valid-type]
generics_scoping.py:61: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:61: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:65: error: Type variable "generics_scoping.S" is unbound [valid-type]
generics_scoping.py:65: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:65: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:76: error: Free type variable expected in Generic[...] [misc]
generics_scoping.py:86: error: Type variable "T" is bound by an outer class [valid-type]
generics_scoping.py:89: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:89: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:89: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:98: error: Can't use bound type variable "T" to define generic alias [valid-type]
generics_scoping.py:105: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:105: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:105: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:106: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:106: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:106: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:107: error: Type variable "generics_scoping.T" is unbound [valid-type]
generics_scoping.py:107: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:107: note: (Hint: Use "T" in function signature to bind "T" inside a function)
"""
conformance_automated = "Pass"
errors_diff = """
Expand Down
18 changes: 9 additions & 9 deletions conformance/results/mypy/generics_syntax_scoping.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,18 @@ generics_syntax_scoping.py:95: error: "T" already defined as a type parameter [
generics_syntax_scoping.py:98: error: "T" already defined as a type parameter [misc]
generics_syntax_scoping.py:98: error: Variable "generics_syntax_scoping.ClassE.T" is not valid as a type [valid-type]
generics_syntax_scoping.py:98: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
generics_syntax_scoping.py:116: error: Expression is of type "Any", not "TypeVar" [assert-type]
generics_syntax_scoping.py:116: note: "assert_type" expects everything to be "Any" in unchecked functions
generics_syntax_scoping.py:121: error: Expression is of type "Any", not "complex" [assert-type]
generics_syntax_scoping.py:121: note: "assert_type" expects everything to be "Any" in unchecked functions
generics_syntax_scoping.py:124: error: Expression is of type "Any", not "complex" [assert-type]
generics_syntax_scoping.py:124: note: "assert_type" expects everything to be "Any" in unchecked functions
generics_syntax_scoping.py:117: error: Expression is of type "Any", not "TypeVar" [assert-type]
generics_syntax_scoping.py:117: note: "assert_type" expects everything to be "Any" in unchecked functions
generics_syntax_scoping.py:122: error: Expression is of type "Any", not "complex" [assert-type]
generics_syntax_scoping.py:122: note: "assert_type" expects everything to be "Any" in unchecked functions
generics_syntax_scoping.py:125: error: Expression is of type "Any", not "complex" [assert-type]
generics_syntax_scoping.py:125: note: "assert_type" expects everything to be "Any" in unchecked functions
"""
conformance_automated = "Fail"
errors_diff = """
Line 62: Unexpected errors ['generics_syntax_scoping.py:62: error: Expression is of type "Any", not "str" [assert-type]']
Line 67: Unexpected errors ['generics_syntax_scoping.py:67: error: Expression is of type "Any", not "int" [assert-type]']
Line 116: Unexpected errors ['generics_syntax_scoping.py:116: error: Expression is of type "Any", not "TypeVar" [assert-type]']
Line 121: Unexpected errors ['generics_syntax_scoping.py:121: error: Expression is of type "Any", not "complex" [assert-type]']
Line 124: Unexpected errors ['generics_syntax_scoping.py:124: error: Expression is of type "Any", not "complex" [assert-type]']
Line 117: Unexpected errors ['generics_syntax_scoping.py:117: error: Expression is of type "Any", not "TypeVar" [assert-type]']
Line 122: Unexpected errors ['generics_syntax_scoping.py:122: error: Expression is of type "Any", not "complex" [assert-type]']
Line 125: Unexpected errors ['generics_syntax_scoping.py:125: error: Expression is of type "Any", not "complex" [assert-type]']
"""
24 changes: 14 additions & 10 deletions conformance/results/pyrefly/generics_scoping.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,20 @@ Does not implement several scoping checks/restrictions for generics
"""
conformance_automated = "Fail"
errors_diff = """
Line 50: Expected 1 errors
Line 54: Expected 1 errors
Line 75: Expected 1 errors
Line 78: Expected 1 errors
Line 87: Expected 1 errors
Line 94: Expected 1 errors
Line 95: Expected 1 errors
Line 96: Expected 1 errors
Line 61: Expected 1 errors
Line 65: Expected 1 errors
Line 86: Expected 1 errors
Line 89: Expected 1 errors
Line 98: Expected 1 errors
Line 105: Expected 1 errors
Line 106: Expected 1 errors
Line 107: Expected 1 errors
"""
output = """
ERROR generics_scoping.py:29:10-13: Argument `Literal['a']` is not assignable to parameter `x` with type `int` in function `MyClass.meth_2` [bad-argument-type]
ERROR generics_scoping.py:65:11-20: Redundant type parameter declaration [invalid-type-var]
ERROR generics_scoping.py:16:12-34: assert_type(int, Literal[1]) failed [assert-type]
ERROR generics_scoping.py:20:12-38: assert_type(str, Literal['a']) failed [assert-type]
ERROR generics_scoping.py:34:10-13: Argument `Literal['a']` is not assignable to parameter `x` with type `int` in function `MyClass.meth_2` [bad-argument-type]
ERROR generics_scoping.py:50:12-48: assert_type(str, Literal['abc']) failed [assert-type]
ERROR generics_scoping.py:54:12-50: assert_type(bytes, Literal[b'abc']) failed [assert-type]
ERROR generics_scoping.py:76:11-20: Redundant type parameter declaration [invalid-type-var]
"""
24 changes: 14 additions & 10 deletions conformance/results/pyright/generics_scoping.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
conformant = "Pass"
output = """
generics_scoping.py:29:10 - error: Argument of type "Literal['a']" cannot be assigned to parameter "x" of type "int" in function "meth_2"
generics_scoping.py:16:13 - error: "assert_type" mismatch: expected "Literal[1]" but received "int" (reportAssertTypeFailure)
generics_scoping.py:20:13 - error: "assert_type" mismatch: expected "Literal['a']" but received "str" (reportAssertTypeFailure)
generics_scoping.py:34:10 - error: Argument of type "Literal['a']" cannot be assigned to parameter "x" of type "int" in function "meth_2"
  "Literal['a']" is not assignable to "int" (reportArgumentType)
generics_scoping.py:50:13 - error: Type variable "S" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:54:19 - error: Type variable "S" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:65:29 - error: TypeVar "T" is already in use by an outer scope (reportGeneralTypeIssues)
generics_scoping.py:75:24 - error: TypeVar "T" is already in use by an outer scope (reportGeneralTypeIssues)
generics_scoping.py:78:17 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:87:5 - error: Generic type alias within class cannot use bound type variables T (reportInvalidTypeForm)
generics_scoping.py:94:14 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:95:19 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:96:6 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:50:13 - error: "assert_type" mismatch: expected "Literal['abc']" but received "str" (reportAssertTypeFailure)
generics_scoping.py:54:13 - error: "assert_type" mismatch: expected "Literal[b"abc"]" but received "bytes" (reportAssertTypeFailure)
generics_scoping.py:61:13 - error: Type variable "S" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:65:19 - error: Type variable "S" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:76:29 - error: TypeVar "T" is already in use by an outer scope (reportGeneralTypeIssues)
generics_scoping.py:86:24 - error: TypeVar "T" is already in use by an outer scope (reportGeneralTypeIssues)
generics_scoping.py:89:17 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:98:5 - error: Generic type alias within class cannot use bound type variables T (reportInvalidTypeForm)
generics_scoping.py:105:14 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:106:19 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
generics_scoping.py:107:6 - error: Type variable "T" has no meaning in this context (reportGeneralTypeIssues)
"""
conformance_automated = "Pass"
errors_diff = """
Expand Down
2 changes: 1 addition & 1 deletion conformance/results/results.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ <h3>Python Type System Conformance Test Results</h3>
<tr><th class="column col1">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;generics_scoping</th>
<th class="column col2 conformant">Pass</th>
<th class="column col2 conformant">Pass</th>
<th class="column col2 conformant">Pass</th>
<th class="column col2 partially-conformant">Partial</th>
<th class="column col2 partially-conformant"><div class="hover-text">Partial<span class="tooltip-text" id="bottom"><p>Does not implement several scoping checks/restrictions for generics</p></span></div></th>
</tr>
<tr><th class="column col1">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;generics_self_advanced</th>
Expand Down
49 changes: 27 additions & 22 deletions conformance/results/zuban/generics_scoping.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,31 @@
conformance_automated = "Pass"
conformance_automated = "Fail"
conformant = "Partial"
errors_diff = """
Lines 15, 16: Expected error (tag 'fun1')
Lines 19, 20: Expected error (tag 'fun2')
Lines 49, 50: Expected error (tag 'method-str')
Lines 53, 54: Expected error (tag 'method-bytes')
"""
output = """
generics_scoping.py:29: error: Argument 1 to "meth_2" of "MyClass" has incompatible type "str"; expected "int" [arg-type]
generics_scoping.py:50: error: Type variable "generics_scoping.S" is unbound [misc]
generics_scoping.py:50: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:50: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:54: error: Type variable "generics_scoping.S" is unbound [misc]
generics_scoping.py:54: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:54: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:65: error: Free type variable expected in Generic[...] [misc]
generics_scoping.py:75: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:78: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:80: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:87: error: Can't use bound type variable "T" to define generic alias [misc]
generics_scoping.py:94: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:94: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:94: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:95: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:95: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:95: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:96: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:96: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:96: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:34: error: Argument 1 to "meth_2" of "MyClass" has incompatible type "str"; expected "int" [arg-type]
generics_scoping.py:61: error: Type variable "generics_scoping.S" is unbound [misc]
generics_scoping.py:61: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:61: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:65: error: Type variable "generics_scoping.S" is unbound [misc]
generics_scoping.py:65: note: (Hint: Use "Generic[S]" or "Protocol[S]" base class to bind "S" inside a class)
generics_scoping.py:65: note: (Hint: Use "S" in function signature to bind "S" inside a function)
generics_scoping.py:76: error: Free type variable expected in Generic[...] [misc]
generics_scoping.py:86: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:89: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:91: error: Type variable "T" is bound by an outer class [misc]
generics_scoping.py:98: error: Can't use bound type variable "T" to define generic alias [misc]
generics_scoping.py:105: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:105: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:105: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:106: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:106: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:106: note: (Hint: Use "T" in function signature to bind "T" inside a function)
generics_scoping.py:107: error: Type variable "generics_scoping.T" is unbound [misc]
generics_scoping.py:107: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
generics_scoping.py:107: note: (Hint: Use "T" in function signature to bind "T" inside a function)
"""
4 changes: 3 additions & 1 deletion conformance/tests/annotations_generators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,4 +190,6 @@ async def generator30() -> AsyncIterator[int]:
yield


assert_type(generator30, Callable[[], AsyncIterator[int]])
async def uses_generator30() -> None:
async for x in generator30():
assert_type(x, int)
Comment on lines -193 to +195
Copy link
Member Author

@AlexWaygood AlexWaygood Feb 16, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty infers a "function-literal" type for generator30, not a Callable type. A function-literal type is more precise than a Callable type: it retains, for example, the information that generator30 is an instance of types.FunctionType, which is not true for all arbitrary Callable types.

I don't believe the spec states anywhere that type checkers must infer a Callable type for function objects (only that functions must be considered to inhabit Callable types), so I think this change is appropriate.

I tried doing just assert_type(generator30(), AsyncIterator[int]), but then zuban (reasonably) complained that the result of generator30() was never awaited, which wasn't relevant to what the test was trying to assert.

21 changes: 16 additions & 5 deletions conformance/tests/generics_scoping.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Specification: https://typing.readthedocs.io/en/latest/spec/generics.html#scoping-rules-for-type-variables

from typing import TypeVar, Generic, Iterable, TypeAlias, assert_type
from typing import TypeVar, Generic, Iterable, TypeAlias, assert_type, Literal

# > A type variable used in a generic function could be inferred to represent
# > different types in the same code block.
Expand All @@ -11,8 +11,13 @@ def fun_1(x: T) -> T: # T here
def fun_2(x: T) -> T: # and here could be different
return x

assert_type(fun_1(1), int)
assert_type(fun_2('a'), str)
# One of these two should pass; either is acceptable:
assert_type(fun_1(1), int) # E[fun1]
assert_type(fun_1(1), Literal[1]) # E[fun1]

# One of these two should pass; either is acceptable:
assert_type(fun_2("a"), str) # E[fun2]
assert_type(fun_2("a"), Literal["a"]) # E[fun2]

# > A type variable used in a method of a generic class that coincides
# > with one of the variables that parameterize this class is always bound
Expand All @@ -39,8 +44,14 @@ def method(self, x: T, y: S) -> S:
return y

x: Foo[int] = Foo()
assert_type(x.method(0, "abc"), str)
assert_type(x.method(0, b"abc"), bytes)

# Either of these is acceptable; one of the two should pass:
assert_type(x.method(0, "abc"), str) # E[method-str]
assert_type(x.method(0, "abc"), Literal["abc"]) # E[method-str]

# Either of these is acceptable; one of the two should pass:
assert_type(x.method(0, b"abc"), bytes) # E[method-bytes]
assert_type(x.method(0, b"abc"), Literal[b"abc"]) # E[method-bytes]

# > Unbound type variables should not appear in the bodies of generic functions,
# > or in the class bodies apart from method definitions.
Expand Down
27 changes: 14 additions & 13 deletions conformance/tests/generics_syntax_scoping.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,23 +102,24 @@ def method3[T](self, x: T): # E
T = int(0)


class Outer2[T]:
Copy link
Member Author

@AlexWaygood AlexWaygood Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the assertions in this test ran into issues because ty inferred T as having type Literal[""] where T = str("").

ty also rejected the assert_type(T, complex) call lower down after the T = 3j assignment, because we view complex in a type expression as immediately expanding to the union int | float | complex, and the second argument to assert_type takes a type expression -- but complex (the type inferred for T) is not equivalent to int | float | complex.

T = int(1)

assert_type(T, int)
def f(a: int, b: str, c: complex):
class Outer2[T]:
T = a

class Inner1:
T = str("")
assert_type(T, int)

assert_type(T, str)
class Inner1:
T = b

def inner_method(self):
assert_type(T, TypeVar)
assert_type(T, str)

def outer_method(self):
T = 3j
def inner_method(self):
assert_type(T, TypeVar)

assert_type(T, complex)
def outer_method(self):
T = c

def inner_func():
assert_type(T, complex)

def inner_func():
assert_type(T, complex)