Outpost: The on-prem scheduler for monitoring scans

• 1: Outpost requirements
• 2: Outpost authentication
• 3: Outpost configuration

graph TD
    A(["Endor Labs App"]) -->|<span style='font-size: 12px'>Continuous monitoring</span>| B["Source Code Repositories"]
    B <--> F["Private Artifact Registry"]
    A -->|<span style='font-size: 12px'>Initiates scans</span>| C["Private Kubernetes cluster
    <span style='font-size: 12px'>Runs Outpost and the scans inside your firewall</span>"]
    B -->|<span style='font-size: 12px'>Clones repositories</span>| C
    C -->|<span style='font-size: 12px'>Pass scan data</span>| D(["Endor Labs Platform
    <span style='font-size: 12px'>Generate findings from scan results</span>"])

    subgraph E["Firewall"]
    A
    B
    C
    F
    end

    class A,D endor
    class B,C,F customer
    class E firewall
    classDef customer fill:#3FE1F3
    classDef endor fill:#5BF385
    classDef firewall fill:transparent,stroke-dasharray: 5 5,stroke:#FF4500,color:#FF4500

1 - Outpost requirements

You need a Kubernetes cluster with nodes that have at least 8 cores and 32 GB of RAM. Nodes should be running Linux.

You must create a namespace in your Kubernetes cluster to deploy Outpost. Use this namespace when you configure Outpost integration in Endor Labs.

Outpost currently supports the following Kubernetes distributions:

• Azure Kubernetes Service (AKS)
• Google Kubernetes Engine (GKE)
• Amazon Elastic Kubernetes Service (EKS)
• Self-hosted Kubernetes clusters

The total number of nodes in the cluster depends on the number of projects that you want to scan and the number of scans that you want to run in a day. On average, each scan is expected to take approximately one hour to complete.

The total scans per node per day (24 hours) is calculated based on the following formula.

Total scans per node per day = (CPU cores in the node ÷ 8) × 24

You can use the following formula to calculate the number of nodes required.

Number of nodes = Total projects ÷ Total scans per node per day

The following table shows the number of nodes required for different combinations of projects and scans.

Network requirements for Outpost

Ensure the following network requirements are met for Outpost:

• Egress Access: Required. Allow outbound traffic to Endor Labs platform, toolchains, and package managers.
• DNS Resolution: Required. Allow list of necessary domains.
• Network Policies: Required. Allow outbound traffic to Endor Labs platform, toolchains, and package managers.

2 - Outpost authentication

Outpost can use the following authentication mechanisms:

• API Key: You can generate an Endor Labs API key and secret and configure Outpost to use it.
• Azure Managed Identity: You can configure Outpost to use an Azure managed identity for authentication. Applicable if you use Azure Kubernetes Service (AKS). You must also configure a corresponding authorization policy in Endor Labs.
• GCP Service Account: You can configure Outpost to use a GCP service account for authentication. Applicable if you use Google Kubernetes Engine (GKE). You must also configure a corresponding authorization policy in Endor Labs.

API key authentication for Outpost

You can create an Endor Labs API key and secret to authenticate your Outpost configuration. See API keys for more information.

Ensure that you select On-prem Scheduler as the API key permissions.

Azure Managed Identity authentication for Outpost

Perform the following steps to configure Outpost to use an Azure managed identity for authentication.

1. Enable workload identity in the AKS cluster.

2. Enable OIDC provider in the AKS cluster.

3. Create an Azure managed identity.

   az identity create -g endor-group -n endor-identity
   

   The command creates a zero-permission managed identity. Store the clientId for later use.

4. Run the following command to retrieve the OIDC Issuer from AKS.

   OIDC_ISSUER=$(az aks show -g endor-group -n onprem-cluster --query "oidcIssuerProfile.issuerUrl" -o tsv)
   

   The command fetches the OIDC issuer URL for federated authentication. Ensure that you enable OIDC and Workload Identity in the AKS cluster.

5. Create federated credentials for workloads.

   Run the following command to create the scheduler federated credential.

   az identity federated-credential create \
    --name scheduler-federated-identity \
    --identity-name endor-identity \
    --resource-group endor-group \
    --issuer $OIDC_ISSUER \
    --subject system:serviceaccount:onprem-cluster:onprem-scheduler-account
   

   Run the following command to create the endorctl federated credential.

   az identity federated-credential create \
    --name endorctl-federated-identity \
    --identity-name endor-identity \
    --resource-group endor-group \
    --issuer $OIDC_ISSUER \
    --subject system:serviceaccount:onprem-cluster:onprem-scheduler-endorctl-account
   

   Warning

   The onprem-cluster specified in the commands is the name of the Kubernetes namespace where Outpost is to be deployed. Replace it with the actual namespace name where you want to deploy Outpost. Ensure that you create the namespace before running the commands.

   The commands link the managed identity to Kubernetes service accounts and enable secure access without static credentials.

6. Configure an authorization policy in Endor Labs with configuration from Azure.

7. Configure the Outpost integration with Managed Identity Client ID.

   See Outpost configuration for more information on how to configure the Outpost integration.

GCP Service Account authentication for Outpost

Perform the following steps to configure Outpost to use a GCP service account for authentication.

1. Enable workload identity in the GKE cluster.

2. Enable OIDC provider in the GKE cluster.

3. Create a new workload service account.

   gcloud iam service-accounts create endor-compute \
     --description="Endor Labs Compute Service Account" \
     --display-name="Endor Labs Compute Service Account"
   

4. Grant roles/iam.serviceAccountOpenIdTokenCreator to workload service account.

   gcloud projects add-iam-policy-binding endor-experiments \
    --member "serviceAccount:endor-compute@endor-experiments.iam.gserviceaccount.com" \
    --role "roles/iam.serviceAccountOpenIdTokenCreator"
   

5. Create a zero-permission service account for Endor Labs to perform federation.

   gcloud iam service-accounts create endor-federation \
    --description="Endor Labs Keyless Federation Service Account" \
    --display-name="Endor Labs Federation Service Account"
   

6. Allow the Kubernetes service accounts to impersonate the IAM workload service account.

   gcloud iam service-accounts add-iam-policy-binding endor-compute@endor-experiments.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:endor-experiments.svc.id.goog[onprem-scheduler/onprem-scheduler-account]"
   

   gcloud iam service-accounts add-iam-policy-binding endor-compute@endor-experiments.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:endor-experiments.svc.id.goog[onprem-scheduler/onprem-scheduler-endorctl-account]"
   

7. Configure an authorization policy in Endor Labs with configuration from Google Cloud.

8. Configure the Outpost integration with Service Account Name.

   See Outpost configuration for more information on how to configure the Outpost integration.

   Warning

   Replace endor-experiments with the actual project name where you want to deploy Outpost. Replace endor-federation and endor-compute with the actual service account names.

3 - Outpost configuration

After you set up your Kubernetes cluster and set up authentication, you can configure the Outpost integration.

Configure Outpost integration

Perform the following steps to configure Outpost:

1. Select Integrations from the left sidebar.

2. Click Configure under On-Prem Integration.

   

3. Choose from the following authentication methods:

   • API: Use the Endor Labs API key authentication for Outpost. You need to enter the API key and API secret. See API key authentication for Outpost for more information.
   • Azure: Use the Azure managed identity authentication for Outpost. You need to enter the Azure managed identity client ID. See Azure Managed Identity authentication for Outpost for more information.
   • Google: Use the GCP service account authentication for Outpost. You need to enter the GCP service account name. See GCP Service Account authentication for Outpost for more information.

4. Click Advanced to configure the following parameters:

   • Select Enable Build Tool Caching to enable build tool caching. Bazel remote cache is installed and the build tools are cached in the cluster.
   • Enter the number of concurrent scans that Outpost can run in Max Running Scans.
   • Enter the maximum duration of a scan in Max Duration.

5. Click Enable Scheduler to store the configuration and enable Outpost.

6. Click Download Helm Values to download the endor-outpost-values.yaml file.

   You can choose to customize the values before you deploy Outpost.

   You can also extract the chart from oci://endorcipublic.azurecr.io/charts/onprem-scheduler and refer the default values.yaml for all the available options. See Helm Chart Values for more information.

7. Run the following command to deploy Outpost in your Kubernetes cluster.

   helm install endorlabsscheduler oci://endorcipublic.azurecr.io/charts/onprem-scheduler \
   -n <your Kubernetes namespace> \
   -f endor-outpost-values.yaml
   

   The command installs the Outpost scheduler on your Kubernetes cluster.

   Important

   If you use GCP service account authentication, you need to configure annotations for the service account in Helm values.

   Add the following annotations to the Helm chart values before you run the Helm install command.

   scheduler:
    .
    .
     serviceAccount:
       create: true
       annotations:
         iam.gke.io/gcp-service-account: "endor-scanner-sa@project-name-123456.iam.gserviceaccount.com"
   endorctl:
    .
    .
     serviceAccount:
       create: true
       annotations:
         iam.gke.io/gcp-service-account: "endor-scanner-sa@project-name-123456.iam.gserviceaccount.com"
   

   If you run the Helm command directly, update the generated Helm command to set the annotations with the following options:

   --set scheduler.serviceAccount.annotations.iam.gke.io/gcp-service-account="<GCP_SERVICE_ACCOUNT_NAME>@<GCP_PROJECT_NAME>.iam.gserviceaccount.com"
   --set endorctl.serviceAccount.annotations.iam.gke.io/gcp-service-account="<GCP_SERVICE_ACCOUNT_NAME>@<GCP_PROJECT_NAME>.iam.gserviceaccount.com"
   

   For example:

   helm install endorlabsscheduler oci://endorcipublic.azurecr.io/charts/onprem-scheduler \
       -n <k8s-ns> \
       --set endorAPI=https://api.staging.endorlabs.com \
       --set endorNamespace=nryn \
       --set auth.gcpServiceAccountName=endor-scanner-sa@project-name-123456.iam.gserviceaccount.com \
       --set scheduler.serviceAccount.annotations.iam.gke.io/gcp-service-account="endor-scanner-sa@project-name-123456.iam.gserviceaccount.com" \
       --set endorctl.serviceAccount.annotations.iam.gke.io/gcp-service-account="endor-scanner-sa@project-name-123456.iam.gserviceaccount.com" \
       --set bazelremote.install=false
   

   The annotations are automatically added to the Helm chart values if you choose Azure managed identity authentication.

8. If you do not want to customize the values, the Helm command with your configured values appears in the user interface. You can copy the command and run it on your Kubernetes cluster.

   For example, the following command appears on the user interface when you configure the integration on the endor Kubernetes namespace with the Azure managed identity authentication and build tool caching enabled. The root Endor Labs namespace is endor.

   helm install endorlabsscheduler oci://endorcipublic.azurecr.io/charts/onprem-scheduler \
        -n endor \
        --set endorAPI=https://api.endorlabs.com \
        --set endorNamespace=endor \
        --set auth.azureManagedIdentityClientID=12a34b56-7c89-0d1e-2f34-567g890h1234 \
        --set bazelremote.install=true
   

   You can copy the command and run it on your Kubernetes cluster to deploy Outpost.

Update Outpost configuration

To update the Outpost configuration, you need to uninstall the existing Helm chart and install a new one with the updated values.

Run the following command to uninstall the existing Helm chart.

helm uninstall endorlabsscheduler -n <your namespace>

You can update the configuration in the user interface to generate a new Helm chart or command, or you can manually update the values in the endor-outpost-values.yaml file. We recommend that you update the configuration in the user interface even if you manually update and install the Helm chart.

Perform the following steps to update the configuration in the user interface:

1. Select Integrations from the left sidebar.
2. Click Manage under On-Prem Integration.
3. Update the configuration and click Enable Scheduler to update the configuration.
4. Apply the updated values with the helm install command as described in Configure Outpost integration.

Generally, you need to update the configuration when the authentication expires. API keys have a maximum validity period of one year. The expiry of Azure managed identity and GCP service accounts depends on the expiry of the corresponding authorization policy.

View Outpost logs

You can view the Outpost logs in the Endor Labs platform.

Perform the following steps to view the Outpost logs:

1. Select Integrations from the left sidebar.

2. Click View Logs under On-Prem Integration.

   

   You can copy the logs to the clipboard or download the logs.

   By default, the logs are brief logs are displayed. You can select Show Verbose Logs to view the detailed logs.

   The log level is set as All by default. You can select Info to view the info logs and Debug to view the debug logs.

Helm Chart Values

Run the following command to extract the default values for the Outpost Helm chart.

helm pull oci://endorcipublic.azurecr.io/charts/onprem-scheduler --untar

The values.yaml file in the onprem-scheduler directory contains the default values for the Outpost Helm chart.

The following yaml file shows the default values in the values.yaml file.

# Default values for onprem-scheduler.
# This file is YAML-formatted.

# Base URL for the Endor Labs platform [Do not modify]
endorAPI: "https://api.endorlabs.com"

# Your organization's namespace in Endor Labs [Do not modify unless there is a change in your tenant]
endorNamespace: "required"

# Log level for scheduler and endorctl. Optional.
logLevel: "info"

# Log output format for scheduler. Optional.
logOutput: "json"

# Authentication configuration - use ONE of the following methods.
# NOTE: Only one authentication method (apiKey & apiSecret, gcpServiceAccountName,
# or azureManagedIdentityClientID) must be set.
auth:
  # Option 1: API Key authentication. Enter the Endor Labs API key and secret.
  apiKey: ""
  apiSecret: ""

  # Option 2: GCP Service Account authentication. Enter the GCP service account name.
  # NOTE: Ensure service accounts are created with workload identity annotations.
  gcpServiceAccountName: ""

  # Option 3: Azure Managed Identity authentication. Enter the Azure managed identity client ID.
  # NOTE: Ensure service accounts are created with workload identity annotations.
  azureManagedIdentityClientID: ""

scheduler:
  # Maximum number of scans that you want to run concurrently. Optional.
  maxRunningJobs: 20

  # Scheduler container image settings.
  image:
    # Container repository for the scheduler image [Do not modify]
    repository: "endorcipublic.azurecr.io/scheduler"

    # Image version to use [Do not modify]
    tag: "latest"

    # Image pull policy [Do not modify]
    pullPolicy: "Always"

  # Labels for the scheduler deployment. Optional.
  labels: {}

  # Annotations for the scheduler deployment. Optional.
  annotations: {}

  # Labels for the scheduler pod. Optional.
  podLabels: {}

  # Annotations for the scheduler pod. Optional.
  podAnnotations: {}

  serviceAccount:
    # Specifies whether a service account should be created. Optional.
    create: false

    # Name of the service account to use for scheduler. Optional.
    name: ""

    # Labels for the scheduler service account. Optional.
    labels: {}

    # Annotations for the scheduler service account. Optional.
    annotations: {}

  # Pod-level security context for the scheduler. Optional.
  podSecurityContext: {}

  # Container-level security context for the scheduler. Optional.
  securityContext: {}

  # Resource constraints for the scheduler pod. Optional.
  resources: {}

  healthProbes:
    # Port used to perform health checks. Optional.
    port: 8080

    # Readiness probe for the scheduler pod. Optional.
    readinessProbe:
      enabled: true
      failureThreshold: 2
      successThreshold: 1
      periodSeconds: 5
      timeoutSeconds: 1
      initialDelaySeconds: 10

    # Liveness probe for the scheduler pod. Optional.
    livenessProbe:
      enabled: true
      failureThreshold: 4
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
      initialDelaySeconds: 0

  # Node selector for the scheduler pod. Optional.
  nodeSelector: {}

  # Tolerations for the scheduler pod. Optional.
  tolerations: []

  # Affinity settings for the scheduler pod. Optional.
  affinity: {}

  # Volumes for the scheduler pod. Optional.
  volumes: []

  # Volume mounts for the scheduler pod. Optional.
  volumeMounts: []

  # Additional environment variables for the scheduler pod. Optional.
  additionalEnvs: []

endorctl:
  # Maximum runtime duration in minutes for a scan. Optional. Default value is 60.
  maxDuration: 1440

  bazelRemote:
    # Bazel remote cache service name.
    # Refer bazelremote values below. Optional.
    serviceName: "bazel-remote-cache"

    # Bazel remote cache GRPC service port.
    # Refer bazelremote values below. Optional.
    servicePort: 9092

  # Endorctl container image settings.
  image:
    # Container repository for the endorctl image [Do not modify]
    repository: "endorcipublic.azurecr.io/endorctl_bare"

    # Image version to use [Do not modify]
    tag: "latest"

    # Image pull policy [Do not modify]
    pullPolicy: "Always"

  # Labels for the endorctl job. Optional.
  labels: {}

  # Annotations for the endorctl job. Optional.
  annotations: {}

  # Labels for the endorctl pod. Optional.
  podLabels: {}

  # Annotations for the endorctl pod. Optional.
  podAnnotations: {}

  serviceAccount:
    # Specifies whether a service account should be created. Optional.
    create: false

    # Name of the service account to use for endorctl. Optional.
    name: ""

    # Labels for the endorctl service account. Optional.
    labels: {}

    # Annotations for the endorctl service account. Optional.
    annotations: {}

  # Pod-level security context for the endorctl. Optional.
  podSecurityContext: {}

  # Container-level security context for the endorctl. Optional.
  securityContext: {}

  # Resource constraints for the endorctl job. Optional.
  resources: {}

  # Node selector for the endorctl pod. Optional.
  nodeSelector: {}

  # Tolerations for the endorctl pod. Optional.
  tolerations: []

  # Affinity settings for the endorctl pod. Optional.
  affinity: {}

  # Volumes for the endorctl pod. Optional.
  volumes: []

  # Volume mounts for the endorctl pod. Optional.
  volumeMounts: []

  # Backoff limit for the endorctl job. Optional.
  backoffLimit: 0

  # TTL seconds after finished for the endorctl job. Optional.
  ttlSecondsAfterFinished: 100

  # Additional environment variables for the endorctl pod. Optional.
  additionalEnvs: []

#
# DEPENDENCIES
#

# Bazel remote cache configuration. Optional. Not enabled by default.
bazelremote:
  # Whether to install the Bazel remote cache component
  install: false

  image:
    # Container repository for the Bazel remote cache image [Do not modify]
    repository: "buchgr/bazel-remote-cache"

    # Specific version of the Bazel remote cache to use [Do not modify]
    tag: "v2.4.1"

    # Image pull policy [Do not modify]
    pullPolicy: "IfNotPresent"

  # Full name of the chart. Optional.
  fullnameOverride: "bazel-remote-cache"

  # Bazel-remote config to provision inside of the container. Optional.
  conf: |-
    # https://github.com/buchgr/bazel-remote#example-configuration-file
    dir: /data
    max_size: 500
    experimental_remote_asset_api: true
    access_log_level: all
    port: 8080
    grpc_port: 9092    

  ## For advanced bazel-remote configuration options,
  ## Refer https://github.com/slamdev/helm-charts/tree/master/charts/bazel-remote#readme

Post-deployment configuration

After you deploy Outpost, you can set up Endor Labs apps depending on your source code manager. You can install Endor Labs apps for the following source code managers:

• GitHub Cloud
• GitLab Cloud and self-hosted
• Bitbucket Data Center
• Bitbucket Cloud
• Azure DevOps Cloud