Schemathesis for coding agents

After changing API code, a coding agent can run Schemathesis against the running service to check whether it still follows its API description. The tool reads an OpenAPI or GraphQL schema, then generates tests from it and checks the responses. The tests can find server errors and responses that do not match the schema.

Schemathesis uses property-based testing: instead of checking only a few handpicked examples, it generates many different inputs from the schema's rules. By default it sends both valid and invalid data, so it can also find invalid input that the API accepts. After changing code, an agent can run Schemathesis against a local service. Failures point to places where the service's behavior conflicts with the API schema. Some generated requests can change data, so check which operations the schema includes and where the service runs.

Install and run a local check

Install Schemathesis in your development Python environment:

python -m pip install schemathesis

The Schemathesis README also shows uv tool install schemathesis. The command-line program is st, and schemathesis is a longer name for the same command. The CLI reference documents its run command and options.

To avoid the usual create, update, and delete requests, limit the run to GET operations with the documented --include-method filter, and start the corresponding API locally. The following example assumes the schema is at ./openapi.yaml and the service listens on port 8000:

st run ./openapi.yaml --url http://127.0.0.1:8000 --include-method GET

The schema file is read from disk. --url sets the base address of the API under test, and Schemathesis sends generated HTTP requests to it. The method filter keeps the run to GET operations. Check the schema before running the command: a GET endpoint can still have side effects if the service is designed incorrectly.

Schemathesis reports test progress and failures in the terminal. A failure gives the agent a concrete request and response to investigate, such as a server error or a response that violates the schema. The quick start shows that each failure also comes with a curl command that repeats the request. The result depends on the schema and service code, so report what the command finds rather than assume it will pass or fail.

For a private service that requires authentication, keep the token in an environment variable and let the shell put it into a request header. The CLI documents --header (short form -H) for custom headers:

st run ./openapi.yaml \
  --url http://127.0.0.1:8000 \
  --include-method GET \
  --header "Authorization: Bearer $API_TOKEN"

Set API_TOKEN in the shell before running the command, and keep the token itself out of the agent prompt. The authentication guide also shows the same header in a schemathesis.toml config file, where Schemathesis reads ${API_TOKEN} from the environment. The guide says Schemathesis hides sensitive values in its output and reproduction commands by default.

Run tests against a local target or one you are explicitly authorized to test. Schemathesis sends real requests, so mutating operations in the schema can change data on a shared or production service. The project README tells you to point it at a dev or staging environment and to keep it away from production.

Give the agent a focused task

Ask the coding agent to change an endpoint, start the local service, and run Schemathesis against the matching schema. The agent should inspect failures, fix code when the schema describes the intended behavior, and report any mismatch between the schema and the implementation rather than silently changing the contract.

For example, an agent prompt could say:

Update the local API implementation to satisfy the existing OpenAPI contract. Start the service locally, then run st run ./openapi.yaml --url http://127.0.0.1:8000 --include-method GET. Do not call a remote service or change the schema to make a failing test pass. Summarize any failures you cannot resolve.

If the agent should leave a report file, the CLI reference documents --report with formats such as junit and ndjson:

st run ./openapi.yaml --url http://127.0.0.1:8000 --include-method GET --report junit

The report goes to the schemathesis-report folder unless --report-dir names another one. Schemathesis options can change between major versions. Version 4 removed older options such as --base-url and --junit-xml, which older examples still use. An agent should check st --version and follow the documentation for the installed version instead of copying an old command from a blog post.

Know what the results cover

Schemathesis can test only behavior represented in the API schema. If the schema leaves out an endpoint, generated tests cannot check it. The same applies when the schema omits a rule for valid inputs or the fields a response should contain. Keep the schema aligned with the intended API contract, and retain focused functional tests for important known inputs and business rules. A schema-conforming response does not, by itself, prove that the response contains the correct business result.

The Schemathesis documentation covers OpenAPI and GraphQL schemas. It also documents report formats and the schemathesis.toml config file. An agent can rerun the same command after code changes and use specific failures to investigate problems. The operator still needs to decide whether a failure indicates a code defect, an incorrect schema, or an unsafe test target.