Endor Labs allows you to determine whether the OS packages present in a container image are actually used by your application at runtime. Use container reachability to distinguish dependencies that are merely installed from those that are actively exercised during execution, helping security teams prioritize the most critical security issues for remediation.

Prerequisites

To perform container reachability analysis, ensure you meet the following system requirements:

• The container must have sufficient CPU and memory resources to run successfully.

• The container must be runnable.

• The container must have network access if its startup process requires external communication.

• Docker daemon dockerd must be installed on the host, must be runnable and accessible to the current user without elevated privileges. For example, docker images should work without sudo.

• The negotiated Docker API version between the client and server must be 1.48 or higher.

• The scan must be run on either a Linux or macOS host machine. Container reachability is supported for both amd64 and arm64 architectures.

Determine container reachability

Endor Labs determines container reachability by extracting OS packages from the container image, profiling the container’s runtime behavior, and correlating the results to identify which dependencies are actually used during execution. The steps below describe how container reachability is determined.

1. Dependency extraction - Endor Labs extracts all packages and dependencies present in the container image by analyzing its file system to identify installed OS packages and their file path locations within the image.

2. Dynamic profiling - Endor Labs runs the container image in a controlled environment, monitoring which OS-level files and dependencies are accessed during execution. The profiling captures runtime behavior including system calls, process IDs, and file path access patterns. It also identifies the main process that starts the container as the entry point and uses it to determine which packages are reachable through their dependency relationships.

3. Path matching and reachability determination - Endor Labs correlates the results from both steps by comparing the extracted dependency file paths against the file access patterns captured during profiling, then assigns each dependency a reachability status based on whether it was accessed during execution.

Run the following command with the --os-reachability flag to include container reachability analysis in the scan.

endorctl container scan \
  --namespace=<your-namespace> \
  --image=<image_name:tag> \
  --project-name=<endor_project_name> \
  --os-reachability

You can also run container scans with OS reachability using GitHub Actions. See Scan containers with OS reachability for details.

Image qualification

Before dynamic profiling begins, Endor Labs performs a series of image qualification checks to determine if the container image is suitable for profiling. These checks include:

• Image size - Verifies that the uncompressed image size does not exceed the configured limit. The default limit is 10 GB. Use the --profiling-max-size flag to adjust this limit.

• Runnability - Runs the container to check that it starts without errors. If the container exits with an error, the error details are shown in the CLI output and surfaced in the Endor Labs user interface.

If the image fails any of the qualification checks, dynamic profiling is skipped and the scan proceeds without reachability analysis.

Container reachability options

You can run the endorctl container scan --os-reachability command with the following options.

Container reachability status

The container reachability status indicates whether a dependency was used during runtime profiling or whether its usage could not be determined.

• Reachable - The dependency is observed in runtime signals or confidently inferred through correlation analysis.

• Potentially reachable - The dependency has not been observed during profiling, and there is no correlation evidence of its usage. However, its usage cannot be definitively ruled out without additional analysis, such as extended runtime monitoring.

• Unreachable - The dependency was not observed during profiling and has no path from the container image’s entry point to it.

Recommended approach for prioritizing remediation

Use reachability information to prioritize vulnerability remediation effectively. The following table provides recommended actions based on the combination of vulnerability severity and reachability status.

Filter container findings by reachability

You can filter findings across all projects by their reachability status.

1. Select Findings from the left sidebar.
2. Select Attributes, and in the Reachable Dependency filter, select Yes, Potentially, or No to narrow down findings by reachability status.

Limitations

Keep the following limitations in mind when using container reachability analysis:

• Code coverage - The scan does not detect dependencies accessed after the profiling window ends.

• OS packages - Container reachability analysis applies only to OS-level packages. It identifies which OS packages are used at runtime but does not analyze specific vulnerable functions within those packages. Use Software Composition Analysis reachability to assess the runtime relevance of application dependencies.

• Windows not supported - Container scanning and reachability are not supported on Windows.

• Tar image paths - Dynamic profiling is not supported for container images referenced with a tar path.

Troubleshoot issues

Endor Labs enhances software supply chain security by providing transparent mechanisms for signing and verifying software artifacts.

• Integrity of container images and build artifacts: Using a cryptographic signature ensures that container images and other build artifacts are genuine and crafted by the organization. This adds an extra layer of security to the software supply chain, making sure that only authorized and unaltered items are scheduled for execution.

• Traces across workflows: Beyond just verification, the framework offers thorough traceability. Users can trace the roots of container images and build artifacts, navigating through workflows and environments. Complete traceability ensures transparency, enabling organizations to validate the entire lifecycle of their software, from creation to deployment.

• Certificate validity: Endor Labs uses a short-lived certificate with a validity period of 5 minutes to ensure that the build artifact has been signed during this time frame. To further guarantee the signing occurred within the valid window, a timestamp is added alongside the certificate and signature, confirming the signing within the specified time frame.

You can sign artifacts using the following methods.

• Using GitHub Action
• Using endorctl CLI

Sign artifacts using GitHub Action

Use the Endor Labs GitHub Actions to sign artifacts.

1. Set up authentication to Endor Labs.
   • (Recommended) If you are using GitHub Action keyless authentication, set an authorization policy in Endor Labs to allow your organization or repository to authenticate. See Keyless Authentication for more information.
   • Alternatively, authenticate with a GCP service account setup for keyless authentication from GitHub Actions or an Endor Labs API key added as a repository secret.
2. Checkout your code.
3. Install your build toolchain.
4. Build your code.
5. Sign your artifacts with Endor Labs.

Endor Labs GitHub Action

Use the GitHub Action endorlabs/github-action/sign@version to sign your artifacts. Set the following input parameters.

See the following example workflows to sign an artifact.

- name: Sign artifacts with Endor Labs
  on: [push, workflow_dispatch]
  name: build
  jobs:
  ko-publish:
    name: Release ko artifact
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      packages: write
      contents: read
    steps:
      - uses: actions/setup-go@v4
        with:
          go-version: '1.20.x'
      - uses: actions/checkout@v3
      - uses: ko-build/setup-ko@v0.6
      - run: ko build
      - name: Login to the GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Publish
        run: KO_DOCKER_REPO=ghcr.io/endorlabs/hello-sign ko publish --bare github.com/endorlabs/hello-sign
      - name: Get Image Digest to Sign
        run: |
          IMAGE_SHA=$(docker inspect ghcr.io/endorlabs/hello-sign:latest | jq -r '.[].Id')
          SIGNING_TARGET="ghcr.io/endorlabs/hello-sign@$IMAGE_SHA"
          echo ARTIFACT="$SIGNING_TARGET" >> $GITHUB_ENV
      - name: Sign with Endor Labs
        uses: endorlabs/github-action/sign@version
        with:
          namespace: "example"
          artifact_name: ${{ env.ARTIFACT }}

Provenance information in signed artifacts

The signed artifacts contain provenance metadata that describe the origin, history, and ownership of an artifact throughout its lifecycle. Including this information in signed artifacts enhances transparency, trustworthiness, and accountability.

The following provenance information is included in the signed artifacts.

Sign artifacts using endorctl

Use the endorctl CLI to sign an artifact. Ensure you have downloaded the latest endorctl binary.

To sign an artifact, run the following command.

endorctl artifact sign --name string --source-repository-ref string --certificate-oidc-issuer string

Specify the following options with the endorctl artifact sign command to include provenance information in your signed artifacts.

View the signed artifacts

To view the signed artifacts:

1. Sign in to Endor Labs and select Inventory from the left sidebar.

2. Select Artifacts. The list shows signed artifacts with Name, Created, and Last Updated details.

   

3. Use the search bar to find artifacts by name, description, or tags. You can use the following filters:

   • Artifact Types: Filter by artifact type, for example container image.
   • Created: Filter by when the artifact was created.

4. Select an artifact to see its signed artifact digests in the list and provenance information. The list shows Artifact Digest, Reference, Created, and Last Updated for each digest.

5. Select an artifact digest to open Artifact Digest Details and view the metadata, signature and certificate details, build configuration, and source repository information.

   

Understand the signing process

When you run the endorctl artifact sign <image> command, Endor Labs initiates the following processes:

• Authentication: Initiates regular authentication and retrieves a token from the OIDC or workflow provider while using an authentication option such as --enable-github-action-token or API keys.
• Key Generation: Generates a public and private key using ECDSA-256.
• Certificate Request: Sends a certificate request to the private Certificate Authority to obtain a short-lived certificate.
• Provenance Inclusion: Incorporates provenance information from the token (if available) or provided with the CLI, adding it as a set of extensions to the certificate using ASN.1 encoding.
• Image Signing: Uses the private key to actively sign the image.
• Certificate Storage: Stores the certificate containing provenance information along with the signature in the database.
• Timestamp: Adds a timestamp of the signing event.

Verify the signed artifacts

To verify a signed artifact, use the following command:

endorctl artifact verify --name <artifact> --certificate-oidc-issuer <issuer>

Use the following command-line options with endorctl artifact verify:

Understand the verification process

When you run the endorctl artifact verify --name <artifact> --certificate-oidc-issuer string command, Endor Labs initiates the following verification processes:

• Authentication: Initiates regular authentication and retrieves a token from the OIDC or workflow provider while using an authentication option such as --enable-github-action-token or API keys.
• Signature Retrieval: Retrieves a signature entry from the database using the artifact name.
  • If the entry is not found, the verification process fails.
• Certificate Authority Check: Checks for a trusted Certificate Authority.
• Image Signature Validation: Validates the image signature using the public key from the certificate.
• Timestamp Validation: Validates that the timestamp in the signature entry is within the certificate’s validity.
• OIDC Issuer Verification: Checks whether the issuer provided matches the contents of the certificate.
• Provenance Verification: Ensures that any provenance information from the CLI matches the ones in the certificate.

Revoke the artifact signature

You can revoke a signature of a signed artifact for reasons such as a precautionary measure to safeguard against security risks, to maintain compliance, or to uphold trust and integrity.

To revoke a signature linked to an artifact and prevent its usage, use the following command:

endorctl artifact revoke-signature --name <image> --source-repository-ref "ref"

Specify the following command-line options for endorctl artifact revoke-signature:

Revoking the artifact signature invalidates the corresponding database entry and ensures that any attempts to verify the signature will fail.

Best practices

• While specifying the artifact name during the signing process, for the container images, adhere to the structure registry.example.com/repository/image@sha256:digest.
• The signing process does not support tags. Ensure that you specify a SHA256 digest with the artifact you are signing to represent a cryptographic hash of the image’s content. This ensures a unique digest is created for every minor alteration in the image.

With the release of the new endorctl container scan commands, the old endorctl scan container commands and their related flags will be removed after a three-month deprecation period.

Use the new dedicated command to ensure continued compatibility.

Mapping of deprecated and new container scan commands

Examples

• To scan a basic container image:

  • Old: endorctl scan --container nginx:latest --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --namespace my-namespace

• To scan a container tar file:

  • Old: endorctl scan --container-tar /path/to/image.tar --namespace my-namespace
  • New: endorctl container scan --image-tar /path/to/image.tar --namespace my-namespace

• To scan a container with a project name:

  • Old: endorctl scan --container nginx:latest --project-name my-nginx --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --project-name my-nginx --namespace my-namespace

• To scan a container in a reference context:

  • Old: endorctl scan --container nginx:latest --container-as-ref --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --as-ref --namespace my-namespace