Debugging Failed Runner Execution Environments

In this lesson, you will learn how to gain visibility into isolated runner containers during live pipeline failures to diagnose environment-specific execution errors.

When a CI/CD job fails within an ephemeral Kubernetes pod or a Docker-based runner, the lack of persistence makes post-mortem analysis difficult. Because these environments are destroyed immediately upon job completion (or failure), standard logs often fail to capture the state of volatile memory, orphan processes, or corrupted filesystem artifacts.

The "Pause and Inspect" Strategy

To effectively debug, you must intercept the container before the CI/CD orchestration layer triggers the SIGTERM signal that kills the pod. The most powerful technique is to introduce a "trap" in your pipeline definition. By modifying your command to sleep upon failure, you keep the container alive for remote investigation:

# Example: Adding a failure trap in a Kubernetes-based runner
script:
  - ./run_integration_tests.sh || (sleep 3600)

Once the container is paused, you can use kubectl or docker exec to enter the environment:

1. Access the shell: Use kubectl exec -it <pod-name> -- /bin/sh to gain direct access to the filesystem where the job crashed.
2. Examine environment parity: Check the injected environment variables (env) and mounted secret volumes to ensure they match your Infrastructure as Code expectations.
3. Inspect volatile state: Check for hidden artifacts or lock files in the working directory that might be interfering with subsequent tasks.

Diagnostic Tools

Within the shell, verify the process tree using ps aux to identify zombies or hung background tasks. If you suspect network-level connectivity issues (such as failing to reach the internal container registry), use nc -zv <host> <port> or curl -v to test connectivity from within the isolated network namespace.

By shifting from passive log analysis to active remote shell access, you can observe the precise state of the ephemeral filesystem, allowing you to bridge the gap between "it works locally" and "it fails on the runner."

Key Takeaway: To troubleshoot ephemeral runner failures, inject a sleep command upon error to hold the environment open, allowing you to manually exec into the container and inspect the live filesystem and network state before the orchestration layer terminates it.