Datadog datadog-ci for coding agents

When a coding agent changes a web service, it can use Datadog’s datadog-ci command-line tool to run configured Synthetic tests before the change proceeds toward deployment. Synthetic tests check how a service responds to specified browser actions or API requests. The CLI asks Datadog to run the selected tests and retrieves their results. A failing test can make the command return a failure to the continuous integration (CI) job or agent.

Datadog documents datadog-ci for continuous integration and delivery (CI/CD) pipelines. Its documentation does not describe a coding-agent integration. An agent with shell access can run the same documented commands, but it still needs a configured Datadog account, credentials, and Synthetic tests. The example below runs tests tagged for staging and saves JSON and JUnit reports.

Install and authenticate

Install the CLI as a repository development dependency:

npm install --save-dev @datadog/datadog-ci

The package provides the datadog-ci command. Datadog’s CI/CD configuration documentation covers installing and configuring the CLI for Synthetic test runs.

Set DATADOG_API_KEY and DATADOG_APP_KEY in the shell environment that runs the command. In CI, store these credentials in the CI provider’s secret store and expose them to the job. Do not paste them into an agent prompt, commit them to the repository, or put them in a file the agent can read unnecessarily. Set DATADOG_SITE as well when the Datadog account uses a site other than the default.

The CLI also needs a configuration file for the Synthetic test run. The file’s path depends on the repository, so ./datadog-ci.json below is a placeholder. Use the path to the configuration your repository already maintains. Datadog’s Continuous Testing CI/CD instructions describe the available test selection and pipeline setup.

Run tagged tests and save reports

From the repository root, run:

npx datadog-ci synthetics run-tests \
  --config ./datadog-ci.json \
  -s 'tag:staging' \
  --json ./synthetics-results.json \
  --junit ./synthetics-results.xml

The -s 'tag:staging' option selects Synthetic tests with the staging tag. Datadog documents tag-based selection with this command pattern in its CI/CD integration documentation. The --json and --junit options write results in JSON and JUnit formats, respectively. JUnit output is useful when a CI provider displays test results, while JSON gives scripts or an agent structured result data to inspect.

When you install the CLI, npm updates local dependency files and downloads the package. When the agent runs the test command, the CLI contacts Datadog, starts the matching tests, and retrieves their results. Those tests exercise the targets and actions defined in Datadog; they are not local unit tests. Check the configuration and selected tag before running the command, especially if a test interacts with a shared staging service.

When the selected tests pass, the command produces reports. When a test fails, the command returns a failure signal and report data to inspect. Have the agent check the command result before it reports success. Datadog’s Synthetic CI/CD overview describes using these results to block a deployment when key tests fail.

Give the agent a bounded task

A useful coding-agent task is to make a code change, run the staging-tagged Synthetic tests, and summarize failures from the generated reports. For example:

After making the requested change, inspect the repository’s Datadog Synthetic configuration and confirm that tag:staging selects the intended tests. Run the documented datadog-ci synthetics run-tests command with the repository’s configuration, JSON report, and JUnit report. Do not print or request Datadog credentials. Report the command’s exit status and summarize any failed tests from the reports. Do not change Datadog tests or deployment settings.

The agent should use credentials the execution environment already provides. If the repository has no Datadog configuration or the environment lacks credentials, it should report that prerequisite instead of inventing a config file or substituting credentials.

Know what the CLI does not cover

datadog-ci runs Datadog Synthetic tests and retrieves their results; an agent’s own coding or reasoning checks remain separate from those tests. The documented use is through CI/CD commands, and the available evidence does not establish a separate agent-specific integration or agent-driven test-definition feature. Keep the agent’s task focused on running existing tests and interpreting their output.