The problem
Why I made it
While developing my (currently unreleased) comprehensive PrivateUse1 Apple Silicon Metal backend, I discovered a dangerous failure mode in agent-driven development. When the implementation and its tests live in the same project, an AI agent will often make a failing test suite pass by changing the tests instead of fixing the backend. And in agent led development, it's quite often difficult to quickly recognize that an agent changed a test file in an inappropriate way.
The result looked like progress, but the agent had silently weakened the contract it was supposed to satisfy. A green test suite did not necessarily mean the implementation was correct.
The separation
Tests the implementation cannot quietly rewrite
I needed a completely separate testing suite for PrivateUse1 backends. Keeping the suite outside the backend project makes its expectations an independent target instead of another part of the codebase an agent can adjust to make its work appear correct.
I needed that separation, and realized if I made it it's own project and released it I could help other PrivateUse1 backend developers as well. Some parts can also help test PyTorch's in-tree backends. That broader use is what turned the testing need into TorchCTS.
The test surface
A GPU operator does not have one happy path
GPU programming requires much more than checking whether an operator works once. A useful conformance suite has to examine the same operator from many directions:
- Normal behavior with typical inputs.
- Positive infinity and negative infinity inputs.
- NaN inputs.
- Inputs designed to trigger errors.
- Many workload sizes that can expose improper accumulation setups.
- Workloads that exercise different kernels used by the same operator at different sizes.
- The additional numerical, workload, and implementation edge cases that GPU code creates.
Passing one normal case says very little about whether an operator is reliable. TorchCTS is intended to make those hidden dimensions part of the test contract.
The outcome
A shared target for backend correctness
TorchCTS gives backend developers a test suite that is independent from the implementation under test and broad enough to care about real GPU behavior. The goal is not merely to make tests pass. The goal is to make a passing result mean something.