Skip to main content
The walltime instruments allow measuring the walltime of your benchmarks directly in the CI. It leverages bare-metal runners managed and provided by CodSpeed to measure the performance of your benchmarks with low noise and high precision.
Example of a walltime benchmark run

Example of a walltime benchmark run

Multiple benchmark processesWith the walltime instrument, you should try and avoid running multiple benchmark processes in parallel since this can lead to noisy measurements.Thus, using pytest-xdist or similar tools is not recommended.

​
Automated Profiling

When using the walltime instrument, CodSpeed automatically collects profiling data and generates flame graphs for each benchmark. This allows you to quickly identify performance changes and their root causes. Function list
Supported languages and integrationsAt the moment, the walltime instrument is supported for the following languages and integrations:If you want to use the walltime instrument with a different language or integration, please reach out on Discord or email our support.

​
Usage with GitHub Actions

Requirements:
  • CodSpeedHQ/action >= 3.1.0
The CI setup is exactly the same as the one with the instrumentation but instead of running on a GitHub-hosted runner, you’ll need to request a “codspeed-macro” runner. Macro Runners are bare-metal machines managed by CodSpeed and will provide you with a more stable and precise environment to run your benchmarks. Simply replace the runs-on: ubuntu-latest line with runs-on: codspeed-macro and use the mode: walltime option in the action: 
jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: codspeed-macro
    steps:
      - uses: actions/checkout@v4
      # ...
      # Setup your environment here:
      #  - Configure your Python/Rust/Node version
      #  - Install your dependencies
      #  - Build your benchmarks (if using a compiled language)
      # ...
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v4
        with:
          mode: walltime
          run: <Insert your benchmark command here>
          token: ${{ secrets.CODSPEED_TOKEN }} # optional for public repos
Your benchmarks will now run on a CodSpeed-managed runner, the action and the benchmark integration will automatically collect walltime data and you’ll be able to see the new measurements in the CodSpeed dashboard.
CachingIf you’re using caches in the GitHub action workflow, make sure your cache keys include runner.arch to avoid cache misses since CodSpeed Macro runners are running on the ARM64 architecture.For example:
- uses: actions/cache@v4
  with:
    path: /home/.cache/pip
    key: pip-${{ runner.arch }}-${{ hashFiles('pyproject.toml') }}

​
Usage on personal GitHub accounts

At the moment, the macro runners are only available for organizations and not for personal accounts. This is because registering GitHub self-hosted runners on repositories instead of organizations would require the Repository: Administration (Read/Write) permission, which is too broad. To use the macro runners on a repository owned by a personal GitHub account, the only solution is to create a new organization and transfer the repository to that organization.

​
Usage on public repositories

By default, the macro runners are only available for the private repositories of your organization. If you want to use them on a public repository, you’ll need to explicitly allow them from your GitHub organization settings (under Organization Settings > Actions > Runner groups > Default). Allowing macro runners on a public repository

​
Next Steps

Macro Runners

Learn more about CodSpeed managed bare-metal runners designed for Walltime benchmarks

Profiling

Learn more about automated profiling for Walltime benchmarks
⌘I