Use CI to Catch Slowdowns in AI-Written Game Code

Code written by Claude Code or Codex can slow a game down even when it passes unit tests. A refactor or feature may work correctly while frame time, GPU time, or memory usage gets worse. CI therefore needs to check not only whether the build works, but also whether it is slower than the previous build.

Passing tests does not rule out a slowdown

When writing code gets cheap, the bottleneck moves from writing to verifying. Whether a feature works is something tests answer on their own. Whether that code just pushed real-device frame time up by a few milliseconds is not something a green test suite will tell you.

As the number of changes grows, hand-profiling each one stops being realistic. An unnoticed allocation creeps onto the render thread; an update loop quietly gets more expensive; the GC starts running more often. Each of these passes its unit tests, slips past code review, and only shows up as a noticeable slowdown on real hardware.

Functional tests ask, “Does it work?” Build comparison asks, “Did it get slower?”

A change that makes performance worse than before is a performance regression. Rather than relying only on reviewer intuition to find one, you need to check every build against the same criteria.

Make CI catch the slowdown

The mechanism is straightforward. Collect performance telemetry for each build, compare it against the last known-good build, and fail CI when the slowdown exceeds a threshold. That surfaces a performance regression before merge without requiring someone to inspect every number by hand.

Framedash’s framedash perf-diff reports both P50 and P95 for two builds’ frame time, memory, and GPU time. Add --fail-on-regression and it exits 1 when the candidate build’s P50 regresses past the threshold, which fails CI. The decision uses the P50 comparison; P95 is reported alongside as context.

framedash perf-diff --baseline "$BASE_SHA" --candidate "$GITHUB_SHA" \
  --threshold 5 --fail-on-regression

Beyond frame time, memory, and GPU time, it also compares map load time (load_time_ms) and disk I/O (io.*), all treated as lower-is-better metrics. You can narrow the comparison to a single metric with --metric, or limit its scope with --map and --platform.

The comparison depends on build_id. Unless it is clear which build each measurement belongs to, a build-over-build comparison is not reliable.

The workflow end to end

Here’s how to check the performance of agent-written code before merge.

The command that runs this whole flow in CI is framedash run-profile-test. It writes out the FRAMEDASH_* automated-session variables, launches the profiling build you point it at, waits for that telemetry to ingest, then compares it with a baseline using perf-diff.

framedash run-profile-test \
  --command "./Build/Game.exe -nullrhi -ExecCmds='Automation RunTest Perf'" \
  --scenario nightly --api-key-file ci-read.key \
  --baseline "$BASE_SHA" --threshold 5 --fail-on-regression

On the game side, you call the automated-session API once in your automated-test entry point. When the SDK calls BeginAutomatedSessionFromEnvironment(), it reads the FRAMEDASH_BUILD_ID / FRAMEDASH_GIT_BRANCH / FRAMEDASH_GIT_COMMIT / FRAMEDASH_TEST_SCENARIO variables that run-profile-test wrote out, and from then on every event carries the CI build along with its branch, commit, and scenario. No per-event tagging code is required.

TelemetrySDK.Instance.BeginAutomatedSessionFromEnvironment();
// ... run the profiling scenarios ...
TelemetrySDK.Instance.EndAutomatedSession();

One operational detail: a short or headless profiling run can exit before the SDK’s periodic flush. Let the run stay alive until its telemetry is actually sent (each SDK’s CI and headless guide covers how); run-profile-test then waits for that telemetry to ingest before comparing the builds.

The build_id is recorded as a top-level field; branch, commit, and scenario ride the ci.branch / ci.commit / ci.scenario attributes. That makes the build_id from the agent’s commit directly the candidate in the build comparison.

One thing to keep straight is the keys. The CI comparison reads telemetry with an analytics:read key, while the launched game sends telemetry with a separate events:write ingest key. To avoid a name clash, pass the comparison key with --api-key-file rather than the FRAMEDASH_API_KEY environment variable, so FRAMEDASH_API_KEY stays free as the game’s ingest key.

When CI detects a slowdown, where to look

When perf-diff fails CI, what you learn is which metric worsened and by how much. Where it worsened isn’t in that number.

That’s what the performance heatmap is for. It overlays FPS, frame time, GPU time, and memory usage cell by cell on your game map. You can filter by device or build profile, so you can pin down which area of which map got heavier in the slower build, as a location on the map.

The spatial view is drawn from events that include position data. When your profiling scenarios report player position with a registered map ID, the SDK attaches camera orientation (yaw and pitch) to those events automatically. With that in place, you can chase something as specific as “P95 spikes in one corner of desert_ruins on build 1042, looking north.” The slowdown becomes a reproducible investigation target.

Use telemetry to guide the agent’s investigation

At this point, a person still has to open the heatmap, narrow down the cause, and write the fix.

You can hand that step to the agent too. The Framedash MCP server gives an LLM read-only tools and resources for querying your telemetry. The agent can request heatmap grid data, dashboard KPIs, and even raw SQL results using natural-language instructions.

claude mcp add framedash \
  -e FRAMEDASH_API_KEY=fd_xxx \
  -e FRAMEDASH_PROJECT_ID=your-project-uuid \
  -- npx -y @framedash/mcp-server

Now the agent that made the change can continue into investigation and repair. It reads the spatial overview with get_heatmap, isolates the candidate build’s hotspots with a query filtered by build_id, forms a hypothesis, and edits the code. The tools are read-only, so the agent can’t alter or delete your telemetry. The next build runs the same CI comparison to verify whether the fix removed the slowdown.

CI detects the slowdown from the numbers, and the agent investigates likely causes. A human decides where to set the threshold and reviews whether the fix is sound.

Getting started

All you need is to add the SDK to your game, pass a build_id from CI, and add a performance comparison to your pipeline. SDK integration is a few lines, and the supported engines are Unity, UE5, and Godot (C#).

• The concrete SDK and CI setup is laid out in CI-Integrated Profiling.
• The full command list is in the CLI Reference.
• To put it in an agent’s hands, the Claude Code plugin installs the MCP server and skills in one go.

claude plugin marketplace add crane-valley/framedash-claude-plugin
claude plugin install framedash@framedash