Instrumented container reachability

• The container has complex external dependencies such as databases, message queues, third-party services that you cannot realistically exercise in a local ephemeral run.
• You want profiling to happen in a realistic environment, such as staging, that mirrors production traffic.
• You already have integration or end-to-end tests and want reachability to reflect those tests.

​

Prerequisites

• Enable container scanning using the --os-reachability flag.
• Install and authenticate endorctl.
• You have installed Docker daemon dockerd on the host and it runs and is accessible to the current user without elevated privileges. For example, docker images should work without sudo.
• The negotiated Docker API version between the client and server is 1.48 or higher.
• You must run the scan from either a Linux or a macOS host machine. Endor Labs supports container reachability on both amd64 and arm64 architectures.

• Docker: Docker daemon and Docker CLI is available and you can run the instrumented image locally.
• Kubernetes: You have configured kubectl with access to your cluster so you can deploy and run the instrumented image in a pod.

​

Determine instrumented container reachability using Kubernetes

1. Run endorctl container instrument to create a new image with a lightweight sensor injected into the filesystem. The resulting image will have -instrumented appended to the tag.
   endorctl container instrument \
     --image=<original_image>:<tag> \
     --app-stop-signal=QUIT \
     --load-instrumented-image=true
   
   • --image: Original container image to instrument.
   • --app-stop-signal: Signal used to stop the application. The sensor needs this so it can flush profiling data before the container exits.
   • --load-instrumented-image: Loads the instrumented image into your local Docker runtime so Kubernetes can reference it.
   • To instrument a multi-arch image for specific platforms, run endorctl container instrument with the --platform flag.
     endorctl container instrument \
       --image=<image_name-tag> \
       --app-stop-signal=QUIT \
       --load-instrumented-image=true \
       --platform=linux/arm64,linux/amd64
     
2. Define how the instrumented image runs in a manifest. Create a manifest file such as demo-manifest-file.yaml to identify your workload. In the manifest, reference the instrumented image from step 1 and use a pod name and container name you can reuse later. You can also add env, volumes, and other options as needed for your application.
   apiVersion: v1
   kind: Pod
   metadata:
     name: <pod-name>
   spec:
     restartPolicy: OnFailure
     containers:
       - name: <container-name>
         image: <instrumented-image>
         ports:
           - containerPort: <container-port>
             hostPort: <host-port>
         securityContext:
           privileged: true
   
   • Set securityContext.privileged to true so the profiling sensor can run.
   • Set restartPolicy to OnFailure so that the pod does not restart automatically after you stop the app to generate the report.
3. Deploy the instrumented image to Kubernetes. The application runs normally while the sensor observes file access and process activity while the application runs.
   kubectl apply -f <manifest-file>
   kubectl get pods <pod-name>
   
   Replace <manifest-file>, <pod-name>, and <container-name> with the values from your manifest.
   • If you need to access the application locally, run:
     kubectl port-forward pod/<pod-name> <host-port>:<container-port>
     
   • You can also run your tests or interact with the application normally. The profiling sensor will capture runtime activity.
4. After you finish testing, send the --app-stop-signal, for example, QUIT, to stop the application gracefully. This signal triggers the profiling sensor to write the creport.json file and generate the profiling data.
   kubectl exec -it <pod-name> -c <container-name> -- sh -c "kill -QUIT 1"
   
5. The sensor writes a profiling report to a known artifacts directory inside the container.
   • Verify that the creport.json file exists in the container:
     kubectl exec -it <pod-name> -c <container-name> -- sh -c "ls -ls /opt/_instrumented/artifacts"
     
6. Create a local directory and copy the report to it.
   # Create output directory
   mkdir -p collect_output
   # Copy the profiling data
   kubectl cp <pod-name>:/opt/_instrumented/artifacts/creport.json collect_output/creport.json -c <container-name>
   
   • To verify the copy, run:
     ls -la collect_output/
     
   • Alternatively, you can use endorctl container collect to stop the running application and retrieve the profiling report from the instrumented container into a local directory. Skip steps 4, 5, and 6 if you are using this command.
     endorctl container collect \
       --dynamic-profiling-data=true \
       --output-dir=collect_output \
       --image=<instrumented-image>
     
   • Set --dynamic-profiling-data to true to collect profiling data from the instrumented container.
   • Set --output-dir to the local directory that stores the collected data. The command creates a subdirectory under this path cluster/pod/container. Use that path for --profiling-data-dir in the next step.
7. Run endorctl container scan with the path to the directory that contains the collected profiling data, and OS reachability enabled. Endor Labs loads the report, maps runtime files to OS packages, and marks the corresponding packages as reachable.
   endorctl container scan \
     --image=<original_image>:<tag> \
     --profiling-data-dir=collect_output \
     --project-name=<project-name> \
     --os-reachability
   
8. Remove the pod after completing the analysis:
   kubectl delete pod <pod-name>
   
9. Optionally, publish the instrumented image to a registry using the --publish flag. The command pushes the image only if the Docker daemon is already authenticated with the target registry. endorctl will not attempt to re-authenticate with the container registry.
   endorctl container instrument \
     --image=<image_name-tag> \
     --app-stop-signal=QUIT \
     --load-instrumented-image=true \
     --publish=true
   

​

Determine instrumented container reachability using Docker

1. Run endorctl container instrument to create a new image with a lightweight sensor injected into the filesystem. At runtime, the sensor writes creport.json to /opt/_instrumented/artifacts/. By default, the command saves the image as instrumented-image.tar. To use a different output path, pass --output-image-tar.
   endorctl container instrument \
     --image <original_image>:<tag> \
     --load-instrumented-image \
     --app-stop-signal <SIGNAL>
   
   • --image: Container image you want to instrument.
   • --app-stop-signal: Signal that stops the application so the sensor can flush profiling data before the container exits.
   • --load-instrumented-image: Loads the instrumented image into Docker so you can run it with docker run.
2. Run the instrumented image in Docker with --privileged so the sensor can use ptrace to watch file access. Docker blocks ptrace in unprivileged containers by default. You can run tests or interact with the application while the container runs. The sensor observes file access and process activity.
   docker run -d \
     --name <container-name> \
     --privileged \
     <original_image>:<tag>-instrumented
   
   You can also pass:
   • -e KEY=VALUE, for example -e PASSWORD=testpassword123, to provide environment variables your app needs to start.
   • -p <host-port>:<container-port>, for example -p 8080:8080, to expose ports when you send traffic from your host during profiling.
3. After testing, send the --app-stop-signal to stop the application gracefully. This signal triggers the profiling sensor to write the creport.json file. The sleep gives the sensor time to finish writing.
   docker kill --signal=<SIGNAL> <container-name>
   sleep 10
   
4. Create a local directory and copy the report to it.
   mkdir -p ./profile-data
   docker cp <container-name>:/opt/_instrumented/artifacts/creport.json ./profile-data/
   
5. Run endorctl container scan with the path to the directory that contains the collected profiling data, and OS reachability enabled. Endor Labs loads the report, maps runtime files to OS packages, and marks the corresponding packages as reachable. If you run the command outside a Git repository, pass --project-name <project-name> to avoid an initialization error.
   endorctl container scan \
     --image <original_image>:<tag> \
     --profiling-data-dir ./profile-data \
     --os-reachability \
     -n <your-namespace>
   
6. Remove the container after you complete the analysis.
   docker rm <container-name>
   

​

Instrumented reachability options

​

Troubleshoot issues