Ci/cd For Cuda Applications (testing On Gpu Runners): Thrive

Have you ever thought about using GPU (graphics processing unit) runners to test CUDA applications and speed up your build process? Our CI/CD pipelines help spot problems faster and keep your code running smoothly. In our guide, we show you how to set up GPU-enabled runners on GitLab.com and GitHub Actions. We cover everything, from installing the correct Docker image to configuring the NVIDIA CUDA driver with basic nvidia-smi checks. Follow our step-by-step instructions and learn how to adjust your workflow for quicker builds and cleaner tests.

Setting up CI/CD pipelines for CUDA applications with GPU runners

We make it simple to add GPU runners to your CI/CD pipeline using GitLab.com SaaS or a self-hosted GitHub Actions setup. On GitLab.com, your pipeline runs on a GCP n1-standard-4 instance. This instance has 4 vCPUs, 15 GB of RAM, and an NVIDIA T4 GPU (16 GB). For Premium or Ultimate subscribers, these GPU runners are already active. Just add the tag "saas-linux-medium-amd64-gpu-standard" in your .gitlab-ci.yml file to let the system know you want to use the specialized runners.

Your job should use a Docker image that comes with the NVIDIA CUDA driver. A preferred choice is the nvidia/cuda:11.8-base image. In your job setup, include a before_script that updates apt-get, installs nvidia-utils-525, and runs nvidia-smi to check the GPU status. For example, you can use this command:
apt-get update && apt-get install -y nvidia-utils-525 && nvidia-smi
This ensures your job sees the GPU with the proper drivers in place.

If you use GitHub Actions, set up a self-hosted GPU runner by configuring a Linux server with an NVIDIA GPU and at least 16 GB of RAM. Install Docker, NVIDIA drivers (such as the 525 series), and the NVIDIA Container Toolkit (nvidia-docker2). After installing these, restart the Docker daemon. Then, label your runner as "self-hosted," "gpu," and "linux." This way, your workflow YAML file can specify runs-on: [self-hosted, gpu]. You can also set up a matrix strategy to test multiple CUDA versions if needed.

These steps let you automate testing for CUDA apps with containerized environments and ensure that hardware acceleration is in play. The result is a smoother build and test process that cuts down on manual tasks and speeds up your development cycle.

Configuring GPU-enabled runners on GitLab for CI/CD

This guide helps you set up GPU-enabled runners for faster tests and smoother deployments on GitLab. If you have a Premium or Ultimate subscription, add the runner tag "saas-linux-medium-amd64-gpu-standard" in your .gitlab-ci.yml file so that your jobs use the GPU.

Choose a CUDA-enabled image, like nvidia/cuda:11.8-base, with the image: directive. Then add before_script commands that update the package lists, install nvidia-utils-525, and run nvidia-smi to check the GPU status. For example:

apt-get update && apt-get install -y nvidia-utils-525
nvidia-smi

These commands check that your GPU is detected and its drivers are working. If nvidia-smi shows an error, double-check your Docker configuration and make sure the nvidia-utils package installed correctly. Even small version mismatches can block the GPU from being recognized, so confirm driver versions if problems occur.

When you run multiple runners, be sure each job uses the correct tag to avoid mix-ups. Here is a quick reference table:

For more complex setups, look at the logs to spot driver differences or container access issues. This will help refine your GPU testing pipelines.

Setting up self-hosted GPU runners for GitHub Actions with CUDA tests

Once your Linux server is set up with NVIDIA drivers (525 series), Docker, and the NVIDIA Container Toolkit following our GitHub Actions guide, here are some extra tips for running CUDA tests and troubleshooting issues.

To test different CUDA versions, use a matrix strategy in your workflow YAML file. For instance, add a section like this:

matrix:
  cuda-version: [11.2, 11.4, 11.8]

This change lets you run tests in parallel on your GPU runners, which means you can check CUDA code across various versions at the same time. Here’s a neat fact: by testing multiple CUDA environments concurrently, you can significantly cut down your overall testing time.

If you run into test failures, it’s a good idea to check that each container has the right CUDA libraries. A quick way to verify this is to run a command inside your container. For example:

docker run --rm nvidia/cuda:11.8-base nvidia-smi

This command helps confirm that the container is aligned with the intended CUDA version and minimizes the chance of version mismatches.

Optimizing and troubleshooting CI/CD pipelines on GPU runners

Improving your CI/CD pipelines on GPU runners starts by cutting down setup time. Pre-pulling CUDA images and activating Docker layer caching means your builds kick off faster. You can also use autoscaling groups to adjust runner capacity based on job demand, keeping bottlenecks at bay during busy times.

It is key to match the NVIDIA driver version with the CUDA toolkit version (a collection of tools for GPU computing). A mismatch may cause container startups to fail. Running a simple diagnostic command like nvidia-smi confirms that all parts work together.

You can also add retry policies in your .gitlab-ci.yml or workflow YAML. If a job fails unexpectedly, setting retries to a maximum of 2 before triggering an automated rollback with Git tags helps cut downtime and stops errors from spreading.

Monitoring performance metrics and job health is crucial. Regular log checks and alerts catch issues early. Testing in isolated environments keeps one failure from taking down your whole build.

Common pipeline issues and corresponding solutions include:

• GPU driver mismatch – Make sure the driver and CUDA toolkit versions match.
• Resource contention – Set concurrent = 1 in the runner configuration or limit job concurrency.
• Container GPU invisibility – Add the –gpus all flag and verify with nvidia-smi.
• Flaky tests – Isolate GPU tests in separate jobs, with retries set to a maximum of 2.
• Image pull delays – Use a private registry mirror or caching mechanism.

Following these steps leads to optimized GPU runner usage and more reliable, efficient CI/CD pipelines for CUDA applications.

Sample CI/CD pipeline configuration for CUDA testing on GPU runners

This pipeline divides the work into two clear stages: build and test. In the build stage, we use an image based on nvidia/cuda:11.8-base to compile the CUDA (NVIDIA compute toolkit) files. The nvcc (NVIDIA CUDA compiler) command builds all files from the src/ folder into a binary saved as bin/app. Then, in the test stage, we run integration tests by executing the compiled binary and measuring its performance using nvprof. Logs created during testing are saved in a dedicated logs folder for one hour, ensuring you have all the details needed for troubleshooting and improvements.

CI YAML example

Below is an example GitLab CI configuration (.gitlab-ci.yml) designed for CUDA application testing. We first define the stages:

stages:
  - build
  - test

build:
  stage: build
  image: nvidia/cuda:11.8-base
  script:
    - nvcc src/*.cu -o bin/app
  artifacts:
    paths:
      - bin/app

test:
  stage: test
  image: nvidia/cuda:11.8-base
  script:
    - nvprof bin/app
    - mkdir -p logs && echo "Profiling complete" > logs/run.log
  artifacts:
    paths:
      - logs/
    expire_in: 1h

In the build section, we compile the CUDA files into a binary that is stored as an artifact for later use. The test section runs performance profiling with nvprof and creates logs that are collected for further examination. This setup provides smooth automation from compile to integration testing, while also giving you clear checkpoints and artifact logs to help troubleshoot and optimize your GPU-enabled CI/CD process.

Final Words

In the action, our post walks through setting up CI/CD pipelines for CUDA applications with GPU runners on GitLab and GitHub Actions. We covered GPU runner configuration, Docker image selection, and key verification steps using nvidia-smi to ensure smooth testing. The guide detailed tips on optimizing pipeline performance and troubleshooting common issues to maintain reliable production workflows. We aimed to simplify complex steps while keeping costs predictable and efficiency high. This approach to ci/cd for cuda applications (testing on gpu runners) inspires faster iterations and steady progress.

FAQ