Deploy Endor Labs

• 1: Monitoring or supervisory scans
• 1.1: Deploy Endor Labs GitHub App
• 1.2: Deploy Endor Labs Azure DevOps App
• 1.3: Set up Jenkins pipeline for supervisory scans
• 2: CI Scans
• 2.1: Set up keyless authentication
• 2.2: Scanning with GitHub Actions
• 2.3: Scanning with Jenkins
• 2.4: Scanning in GitLab Pipelines
• 2.5: Scanning in Azure Pipelines
• 2.6: Scanning in Bitbucket Pipelines
• 2.7: Scanning with CircleCI
• 2.8: Scanning with Google Cloud Build
• 3: Scan from your IDE

1 - Monitoring or supervisory scans

Perform monitoring scans to gain fast and broad visibility over open source risks across the application portfolio without requiring integrations into application pipelines. These scans are conducted periodically and can also establish baselines that are subsequently used during CI scans.

• GitHub App monitoring scan: You can use the Endor Labs GitHub App to scan your GitHub organizations. It provides broad visibility over your GitHub organizations. Once installed, the GitHub App will automatically clone and scan all the repositories every 24 hours, providing continuous monitoring for open source vulnerabilities. These repositories are temporarily cloned and retained only during the scan. See Scan using the GitHub App for more information.

• Azure DevOps App monitoring scan: You can use the Endor Labs Azure DevOps App to scan your Azure projects organizations. It provides broad visibility over your GitHub organizations. Once installed, the Azure DevOps App will automatically clone and scan all Azure repos every 24 hours, providing continuous monitoring for open source vulnerabilities. These repositories are temporarily cloned and retained only during the scan. See Deploy Endor Labs Azure DevOps App for more information.

• Local monitoring scan: Perform periodic scans in your local environment. You must provide the necessary computing resources to run the scans. These scans are not restricted to GitHub and can support any type of Git repository. See Set up Jenkins pipeline for supervisory scans.

1.1 - Deploy Endor Labs GitHub App

Endor Labs provides a GitHub App that continuously monitors users’ projects for security and operational risk. You can use the GitHub App to selectively scan your repositories for SCA, secrets, RSPM, or CI/CD tools.

Prerequisites for GitHub App

Before installing and scanning projects with Endor Labs GitHub App, make sure you have:

• A GitHub cloud account and organization. If you don’t have one, create one at GitHub.
• Administrative permissions to your GitHub organization. Installing the Endor Labs GitHub App in your organization requires approval or permissions from your GitHub organizational administrator. If you don’t have the permissions, use the command line utility, endorctl, while you wait for the approval.
• Endor Labs GitHub App requires read permissions to Dependabot alerts, actions, administration, checks, code, commit statuses, issues, metadata, packages, pull requests, repository hooks, and security events. It does not need write access to any resources.

Install the GitHub App

To automatically scan repositories using the GitHub App:

1. Sign in to Endor Labs.

2. Choose Projects and click Add Project.

3. From GITHUB, choose GitHub App

4. Click Install GitHub App.

   You will be redirected to GitHub to install the GitHub App.

   

5. Click Install.

6. Select a user to authorize the app.

7. Select the organization in which you want to install the app.

8. Select whether to install and authorize Endor Labs on all your repositories or select the specific repositories that you wish to scan.

   

9. Review the permissions required for Endor Labs and click Install and Authorize.

   If the button to install says Install and Request instead of Install and Authorize, you don’t have permission to install the GitHub App. Use the endorctl command line interface or select Install and Request to notify your organizational administrator of your request to install.

10. Choose a namespace and click Next.

    

11. Based on your license, select and enable from the following list of available scanners.

    • SCA- Perform software composition analysis.
    • CI/CD - Scan the repository and identify all the CI/CD tools used in the repository.
    • RSPM - Scan the repository for misconfigurations.
    • Secret - Scan the repository for exposed secrets.

    

12. Click Continue. You have successfully installed the GitHub App.

Endor Labs GitHub App scans your repositories every 24 hours and reports any new findings or changes to release versions of your code.

Manage GitHub Apps on Endor Labs

You can edit or delete the GitHub App integrations.

To edit the GitHub App integration:

1. Sign in to Endor Labs.

2. Select Manage > Integrations from the left navigation menu.

3. Click Manage next to GitHub under Source Control Managers.

   

4. Click the ellipsis on the right side, and select Edit Integration.

   

5. Based on your license, select and enable from the available list of scanners and click Save. The changes are applicable from the next scanning cycle.

6. Use Reset to clear your selection.

To delete a GitHub App integration, click the ellipsis on the right side, and select Delete Integration.

To manually trigger a scan, click Rescan Org. Endor Labs GitHub App scans your repositories every 24 hours, use Rescan Org to manually schedule outside the 24-hour period.

Click Scan More Repositories to go to Projects page, from which you can add more repositories to scan through the GitHub App.

Set up package repositories

You can improve your experience with the GitHub App by setting up package repositories. This will help you create a complete bill of materials and perform static analysis. Without setting package repositories, you may not be able to get an accurate bill of materials. See Set up package manager integration for more information.

Technical limitations of the GitHub App

The Endor Labs GitHub App provides visibility across a GitHub organization, but it has technical limitations that do not account for the unique requirements of your application.

Here are some of these limitations.

Bill of materials variance

The Endor Labs GitHub App approximates software package builds to create a bill of materials and perform static analysis on your software dependencies. This requires building packages with specific versions of the package manager and runtime environment.

If there are differences in the build environment, it can result in variances in the bill of materials. For the most accurate information, use Endor Labs CLI as a post-build step in your software delivery process.

The following factors contribute to variances in the bill of materials:

1. The time a software package was built.
2. The version of a software package manager.
3. The type of package manager being used.
4. The version of the runtime environment on which a package is installed.

Custom package build steps

Endor Labs requires executing custom build steps outside of standard package manager commands to build software packages and get an accurate bill of materials and perform static analysis. Sometimes, a complete bill of materials may not be generated, or static analysis may not be performed.

Custom resource profiles

Large applications may require significant memory allocations to perform static analysis on a package. The services scanning the GitHub App use 16 GB of memory by default. Applications that require more memory may not obtain vulnerability prioritization information using the GitHub App. Scan large applications in a CI environment using a runner with sufficient resource allocations.

Authentication for private software components

Private software components hosted in an internal package repository may require authentication credentials to create a complete bill of materials or perform static analysis.

If your authentication information to your private package repository is hosted outside the repository, you will need to configure a package manager integration. See Set up package manager integration for more details.

1.2 - Deploy Endor Labs Azure DevOps App

Endor Labs provides an Azure DevOps App that continuously scans Azure repos in your projects for security risks. You can selectively scan your repositories for SCA, secrets, or CI/CD tools.

When you add an Azure DevOps project to an Endor Labs namespace, Endor Labs scans all the Azure repos contained in the project. As a best practice, we recommend that you add only one Azure project to one Endor Labs namespace so that the Azure repos of that project are mapped to an Endor Labs namespace.

Prerequisites for Azure DevOps App

Ensure the following prerequisites are in place before you install the Endor Labs Azure DevOps App.

• An Azure DevOps cloud account and organization. If you don’t have one, create one at Azure DevOps.
• Endor Labs Azure DevOps App requires read permissions to in your project. You can grant these permissions by providing read access to the Code category when you create an Azure DevOps personal access token for Endor Labs.

Install the Azure DevOps App

To automatically scan repositories using the Azure DevOps App:

1. Sign in to Endor Labs.

2. Choose Projects and click Add Project.

3. From AZURE, select Azure DevOps App.

   

4. Enter the host URL of your Azure project.

   The URL must be in the format, https://dev.azure.com/<ORG_NAME>/<PROJECT_NAME>.

5. Enter your personal access token from Azure.

   You must have at least read permissions in the Code category for your Azure DevOps personal access token.

6. Click Scanners and select the scan types to enable.

   • SCA- Perform software composition analysis.
   • Secret - Scan Azure repos for exposed secrets.
   • CI/CD - Scan Azure repos and identify all the CI/CD tools used.

   The available scan types depend upon your license.

   

7. Click Create.

Endor Labs Azure DevOps App scans your Azure repos every 24 hours and reports any new findings or changes to release versions of your code.

Manage Azure DevOps Apps on Endor Labs

You can edit or delete the Azure DevOps App integrations.

To edit the Azure DevOps App integration:

1. Sign in to Endor Labs.
2. Select Manage > Integrations from the left navigation menu.
3. Click Manage next to Azure under Source Control Managers.
4. Click the ellipsis on the right side, and select Edit Integration. You can update your personal access token.
5. Click SCANNERS and based on your license, select and enable from the available list of scanners.
6. Click Save. The changes are applicable from the next scanning cycle.

To delete an Azure DevOps App integration, click the ellipsis on the right side, and select Delete Integration.

To manually trigger a scan, click Rescan Org. Endor Labs Azure DevOps App scans your repositories every 24 hours, use Rescan Org to manually schedule outside the 24-hour period.

Click Scan More Repositories to go to the Projects page, from which you can add more projects to scan through the Azure DevOps App.

1.3 - Set up Jenkins pipeline for supervisory scans

Use the Endor Labs Jenkins pipeline to scan all the repositories in your organization and view consolidated findings. This pipeline runs on your organization’s Jenkins infrastructure and enables administrators to run organization-level supervisory scans easily. It is designed to work in GitHub Cloud and GitHub enterprise server environments.

The Jenkins pipeline carries out the following actions.

• Pulls the Endor Labs Docker image required to perform the scan.
• Synchronizes GitHub organization repositories to a specified namespace on the Endor Labs platform.
• Retrieves the project list or the GitHub repositories for the given tenant’s namespace.
• Groups the projects into batches to optimize scan execution.
• Runs endorctl scans on each batch of projects simultaneously.

Scan the repositories in your organization

The Jenkins Pipeline script is available in the github-org-scan-docker.groovy file.

To scan the repositories in your organization:

1. Generate Endor Labs API credentials
2. Configure GitHub cloud or GitHub enterprise server credentials
3. Configure the Jenkins job

Configure GitHub credentials

Configure the required credentials needed to access GitHub and Endor Labs in the Jenkins pipeline script. You can configure these values from the Jenkins user interface.

• GITHUB_TOKEN- Enter the GitHub token that has permission to access all the repositories in the organization.
• ENDOR_LABS_API_KEY- Enter the Endor Labs API key that you generated.
• ENDOR_LABS_API_SECRET- Enter the Endor Labs API secret generated while creating the Endor Labs API key.

Configure GitHub cloud credentials

Configure the following GitHub cloud parameters in the Jenkins pipeline script.

Required Parameters for GitHub cloud

• AGENT_LABEL- This is a string parameter. Enter the label used to identify the Jenkins agents. The Jenkins job will run on the agents that have this label.
• GITHUB_ORG- This is a string parameter. Enter the organization name in GitHub.
• ENDOR_LABS_NAMESPACE- This is a string parameter. The namespace of your organization tenant in Endor Labs.

Optional Parameters for GitHub cloud

• ENDOR_LABS_API- This is a string parameter. This is only required if the tenant namespace is configured on the Endor Labs staging environment.
• ADDITIONAL_ARGS- This is a string parameter. Use this field to pass any additional parameter to the endorctl scan.
• NO_OF_THREADS- This is a string parameter. Enter the number of Jenkins agents that can be used in parallel for the endorctl scan. If you have 10 Jenkins agents configured with the given AGENT_LABEL, you can enter this value as 9, 1 agent is used for the main job. If not specified, this value defaults to 5.
• ENDORCTL_VERSION- This is a string parameter. Specify the version of the endorctl Docker container. Defaults to the latest version.
• SCAN_TYPE- This is a string parameter. Set this to git to scan commits or github to fetch info from the GitHub API. Defaults to [git, analytics].
• SCAN_SUMMARY_OUTPUT_TYPE- This is a string parameter. Use this field to set the desired output format. Supported formats: json, yaml’, table, summary. Defaults to table.
• LOG_LEVEL- This is a string parameter. Use this field to set the log level of the application. Defaults to info.
• LOG_VERBOSE- This is a string parameter. Use this field to make the log verbose.
• LANGUAGES- This is a string parameter. Use this field to set programming languages to scan. Supported languages: c#,go, java, javascript, php, python, ruby, rust, scala, typescript. Defaults to all supported languages.
• ADDITIONAL_ARGS- This is a string parameter. Use this field to pass any additional parameters to the endorctl scan.

Configure GitHub enterprise server credentials

Configure the following GitHub enterprise server parameters in the Jenkins pipeline script.

Required Parameters for GitHub enterprise server

• AGENT_LABEL - This is a string parameter. Enter the label used to identify the Jenkins agents. The Jenkins job will run on the agents that have this label.
• GITHUB_ORG - This is a string parameter. Enter the organization name in GitHub.
• ENDOR_LABS_NAMESPACE - This is a string parameter. The namespace of your organization tenant in Endor Labs.
• GITHUB_API_URL - This is a string parameter. Enter the API URL of the GitHub enterprise server. This is normally in the form of <FQDN of GitHub Enterprise Server>/api/v3. For example, https://ghe.endorlabs.in/api/v3.

Optional Parameters for GitHub enterprise server

• ENDOR_LABS_API - This is a string parameter. This is only required if the tenant namespace is configured on the Endor Labs staging environment.

• GITHUB_DISABLE_SSL_VERIFY - This is a boolean parameter. This should be used when you want to skip SSL Verification while cloning the repository.

• GITHUB_CA_CERT - This is a multi-line string parameter. This should be used to provide the content of the CA Certificate (PEM format) of the SSL Certificate used on the GitHub Enterprise Server.

• PROJECT_LIST - This is a multi-line string parameter. This should be used to provide a list of projects to scan.

• SCAN_TYPE - This is a string parameter. Set this to git to scan commits or github to fetch info from the GitHub API. Defaults to [git, analytics].

• SCAN_SUMMARY_OUTPUT_TYPE - This is a string parameter. Use this field to set the desired output format. Supported formats: json, yaml*, table, summary. Defaults to table.

• LOG_LEVEL - This is a string parameter. Use this field to set the log level of the application. Defaults to info.

• LOG_VERBOSE - This is a string parameter. Use this field to generate verbose logs.

• LANGUAGES - This is a string parameter. Use this field to set programming languages to scan. Supported languages: c#, go, java, javascript, php, python, ruby, rust, scala, typescript. Defaults to all supported languages.

• ADDITIONAL_ARGS - This is a string parameter. Use this field to pass any additional parameters to the endorctl scan.

• PROJECT_LIST - This is a multi-line string parameter. List of projects to scan. Even though all projects are synchronized, scans run only on the provided projects.

• SCAN_PROJECTS_BY_LAST_COMMIT - This is a string parameter. This parameter is used to filter projects based on the date of the last commit. Enter a number (integer) value for this parameter. The value of 0 means that projects won’t be filtered based on last commit date. Any positive integer is used to calculate the duration in which a commit will add the project for further scanning. If a project did not have a commit in that interval, it will be skipped. If a proper SSL Certificate (a certificate issued by a well-known CA) is not used for GitHub Enterprise, the sync-org command fails and Endor Labs cannot fetch the projects or repositories to scan from the GitHub enterprise server. You can use this field to provide the list of projects or repositories to scan one per line. For example:

          https://github-test.endorlabs.in/pse/vuln_rust_callgraph.git
          https://github-test.endorlabs.in/pse/vulnerable-golang.git
          https://github-test.endorlabs.in/pse/java-javascript-vulnerable-repo.git
          https://github-test.endorlabs.in/pse/multi-lang-repo.git
  

• EXCLUDE_PROJECTS - This is a multi-line string parameter. Use this parameter to list projects or repositories to exclude from the scan.

• NO_OF_THREADS - This is a string parameter. Enter the number of Jenkins agents that can be used in parallel for the endorctl scan. If you have 10 Jenkins agents configured with the given AGENT_LABEL, you can enter this value as 9. If not specified, this value defaults to 5.

Configure the Jenkins job

Use the following procedure to configure the Jenkins pipeline and scan the repositories in your organization.

1. Sign in to Jenkins
2. Configure an Endor Labs API Key and GitHub credentials correctly for your environment.
3. Click + New Item, to create a new Jenkins job.
4. Enter the name of the new pipeline
5. Select Pipeline and click OK.
6. Select This project is parameterised and add the parameters based on your requirements.
7. From the Pipeline section, for Definition, select Pipeline script from SCM
8. For SCM select Git
9. For the Repository URL, enter either git@github.com:endorlabs/jenkins-org-scan.git or https://github.com/endorlabs/jenkins-org-scan.git.
10. For Credentials, enter the credentials required for cloning the repository entered in the previous step.
11. In Branches to build, enter */main.
12. For Script Path, enter github-org-scan-docker.groovy.
13. Select Lightweight checkout.
14. Click Save.

The Jenkins pipeline is highly customizable and adaptable to various GitHub environments and scanning requirements. It streamlines the process of running endorctl scans on your repositories efficiently.

2 - CI Scans

CI Scans are used to focus team’s attention and establish development workflows on the most actionable issues, prioritizing the development team’s time. CI Scans can be triggered directly from automated CI/CD pipelines, looking for new vulnerabilities relative to the baseline established for the target branch. These CI Scans provide immediate feedback to developers in the form of PR comments and can also enforce policies to break builds, block PRs, send notifications, open tickets, and more. CI scans are the most actionable method to prevent vulnerabilities from entering your repositories.

Perform CI scans using:

• endorctl CLI
• Scan with GitLab pipeline
• Scan with GitHub Actions
• Scan with Circle CI
• Scan with Jenkins
• Scan with Azure DevOps
• Scan with Bitbucket
• Scan with Google Cloud Build

See scanning strategies to learn techniques for effectively scanning and monitoring different versions of your projects with Endor Labs.

endorctl is a command line utility designed to bring the functionality of Endor Labs into your software delivery workflows. endorctl has several command flags to help you facilitate operational and security risk monitoring. Developers can integrate Endor Labs into Continuous Integration Workflows using the endorctl scan.

• endorctl scan - You can use endorctl scan to monitor your projects using Endor Labs, and you can update the scan information each time to keep monitoring the project for new findings. The endorctl scan command will scan a specific version of your repository, such as the default branch, a tagged release version, or a commit SHA.
• endorctl scan --pr - You can use the endorctl scan --pr command to scan a specific version of your source code for security and operational risks as part of your continuous integration workflows or CI runs. The endorctl scan --pr command performs a one-time evaluation of your project, focusing on security and operational risks, rather than providing continuous monitoring. CI runs are shown in the Scan History section of each project and are stored for 30 days so that you can analyze and review them on the Endor Labs user interface.

Any continuous integration workflows generally run using the endorctl scan --pr command unless a scan is run on a created tag release, a push to the default or specific branch, or a commit SHA that will be deployed to production.

Authenticating in CI with Keyless Authentication

Keyless Authentication enhances security and minimizes the expenses associated with secret rotation. Keyless authentication is Endor Labs recommended path to scan your projects in the CI workflows. See Keyless Authentication for more information.

2.1 - Set up keyless authentication

At Endor Labs, we believe that the most secure secret is one that doesn’t exist. That’s why in CI/CD environments we recommend using keyless authentication for machine authentication. Keyless Authentication uses OAuth for API authentication and removes the need to maintain and rotate an API key to Endor Labs.

Keyless Authentication is more secure and reduces the cost of secret rotation.

Configure keyless authentication using:

• Google Cloud
• GitHub OIDC
• AWS Cloud

Enabling Keyless Authentication in Google Cloud

To enable Keyless Authentication in GCP you’ll first need permissions to create service accounts and assign these accounts roles to GCP.

The workflow to enable keyless authentication is:

1. Create a service account with no permissions for federation.
2. If you do not attach a service account to compute resources or use the default service account, we recommend creating a new service account for the compute resources. Create a service account to attach to compute resources and impersonate the federation service account.
3. Create an authorization policy to allow the federation service account to authenticate to Endor Labs.
4. Provision the compute resources with the appropriate permissions.
5. Test Keyless authentication.

Creating GCP Service Accounts and Authorization Policies

To create your service accounts, first export your GCP project name as an environment variable:

export PROJECT=<insert-gcp-project>

Once you’ve set the environment variable you’ll create a service account that will be used to federate access to Endor Labs and will be provided permission to access the Endor Labs APIs:

Step 1: Create a federation service account called endorlabs-federation:

gcloud iam service-accounts create endorlabs-federation --description="Endor Labs Keyless Federation Service Account" --display-name="Endor Labs Federation Service Account"

We’ll also create a second service account, that will have access to impersonate the endorlabs-federation account. We’ll call this endorlabs-compute-service.

This is needed if you don’t already have service accounts for your compute resources. If you do, you need to modify the existing permissions to allow the existing service account to create a federation token.

Step 2: Create a keyless authentication service account to assign to compute resources called endorlabs-compute-service:

gcloud iam service-accounts create endorlabs-compute-service --description="Endor Labs Service account for keyless authentication" --display-name="Endor Labs Compute Instance SA"

Finally, we’ll assign endorlabs-compute-service permissions to impersonate the endorlabs-federation account to authenticate to Endor Labs through OIDC.

Step 3: Assign the serviceAccountOpenIdTokenCreator role to the endorlabs-compute-service service account:

gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com" --role="roles/iam.serviceAccountOpenIdTokenCreator"

Once we’ve created the necessary account permissions we will create an authorization policy in Endor Labs to allow the account endorlabs-federation to your Endor Labs tenant:

Use the following command to create an authorization policy in Endor Labs.

Note: Make sure to replace <your-tenant> with your Endor Labs tenant name and <insert-your-project> with your GCP project name in the following command.**

endorctl api create -r AuthorizationPolicy -d '{
    "tenant_meta": { "namespace": "<your-tenant>" },
    "meta": {
        "name": "Keyless Auth",
        "kind": "AuthorizationPolicy",
        "tags": ["gcp"]
     },
     "spec": {
        "clause": ["email=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com", "gcp"],
        "target_namespaces": ["<your-tenant>"],
        "propagate": true,
        "permissions": {
            "rules": {},
            "roles": [
                "SYSTEM_ROLE_CODE_SCANNER"
            ]
        },
    }
}'

You’ve now set up the foundation of keyless authentication. You’ll now need to provision your compute resources with the appropriate GCP scopes and service account.

See Provisioning and Testing Keyless Authentication for GKE workloads for instructions on setting up GKE for keyless authentication.

See Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances for instructions on setting up a virtual machine instance for keyless authentication.

Provisioning and Testing Keyless Authentication for GKE workloads

Prerequisites

The following prerequisites are required to setup keyless authentication on GKE workloads:

• Workload identity is enabled on the target GKE cluster. See the GCP documentation on using workload identity for instructions on migrating existing cluster node pools or creating new clusters to use GCP workload identity.
• The gcloud auth plugin is installed and operational on your machine. See the GCP instructions on enabling the gcloud auth plugin for more details.
• The kubectl CLI is installed. See the Kubernetes documentation for instructions.

Procedure

1. (Optional) Create a GKE cluster with workload identity enabled if you do not already have a GKE cluster
2. Authenticate to the GKE cluster
3. (Optional) Create a namespace for Endor Labs scans
4. Create a Kubernetes service account to impersonate your GKE compute service account
5. Bind your Kubernetes service account to your GCP compute service account
6. Annotate your Kubernetes service account with your GCP service account to complete your binding
7. Test a scanning workload using keyless authentication

Setting Up and Testing Keyless authentication in GKE

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT
• The GKE cluster as CLUSTER_NAME

export PROJECT=<Insert_GCP_Project>
export CLUSTER_NAME=<GKE_CLUSTER>

Optional Step 1: To create a GKE cluster with workload identity enabled if you do not already have a GKE cluster with workload identity enabled run the following command:

gcloud container clusters create keyless-test --workload-pool=endor-github.svc.id.goog --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To authenticate to your GKE cluster run the following command:

gcloud container clusters get-credentials $CLUSTER_NAME

Optional Step 3: To create a namespace for Endor Labs scans run the following command:

kubectl create namespace endorlabs

Step 4: To create a Kubernetes service account to impersonate your GKE compute service account run the following command:

kubectl create serviceaccount endorlabs-compute-service -n endorlabs

Step 5: To bind your Kubernetes service account to your GCP compute service account run the following command:

Note: Make sure to replace <insert-your-project> in the following command with your GCP project name.

gcloud iam service-accounts add-iam-policy-binding endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:<insert-your-project>.svc.id.goog[endorlabs/endorlabs-compute-service]"

Step 6: To annotate your Kubernetes service account with your GCP service account to complete your binding run the following command:

Note: If you have created a different service account name replace endorlabs-compute-service with the appropriate service account name.

kubectl annotate serviceaccount endorlabs-compute-service -n endorlabs iam.gke.io/gcp-service-account=endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com

Step 7: Test scan your project with keyless authentication

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication.

Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances

The following high-level procedure describes the required steps to use keyless authentication with a GCP virtual machine instance:

Procedure:

1. Create a virtual machine instance with the appropriate scopes
2. Download and install endorctl on the virtual machine instance
3. Launch a test scan with keyless authentication

Setting up and Testing Keyless authentication on a GCP virtual machine instance

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT

export PROJECT=<Insert_GCP_Project>

To successfully test keyless authentication first you’ll need to provision a compute resource with the service account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com and the scope https://www.googleapis.com/auth/cloud-platform:

Step 1: To create a virtual machine instance with the appropriate scopes run the following command:

gcloud compute instances create test-keyless --service-account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To download and install endorctl on the virtual machine instance run the following series of commands:

First, SSH to the virtual machine instance you’ve created to test:

gcloud compute ssh --zone "us-west1-b" "test-keyless"  --project $PROJECT

Then download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com for federated access to Endor Labs such as in the below example:

endorctl api list --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com -r Project -n <insert-your-tenant> --count

If this scan runs successfully you’ve tested and scanned a project with keyless authentication to Endor Labs.

Enabling Keyless Authentication in GitHub

To enable Keyless Authentication for GitHub Actions, you’ll need to perform the following steps:

1. Ensure you are using the Endor Labs GitHub Action in your GitHub workflow.
2. Edit your GitHub Action workflow to add permission settings for the GitHub id-token
3. Create an authorization policy for GitHub Action OIDC
4. Test that you can successfully scan a project using Github Action OIDC

Add a GitHub Action OIDC authorization policy

To ensure that the GitHub action OIDC identity can successfully login to Endor Labs you’ll need to create an authorization policy in Endor Labs.

To create an authorization policy:

1. Under Manage go to Access Control
2. Navigate to the “Auth Policy” tab
3. Click on the “Add Auth Policy” button
4. Select “GitHub Action OIDC” as your identity provider
5. Select the permission for the GitHub Action. This permission should be “Code Scanner”
6. For the claim use the key user and put in a matching value that maps to the organization of your GitHub repository.

Configuring your GitHub Action workflow

To configure your GitHub Action workflow with GitHub Action OIDC you can use the following example as a baseline.

The important items in this workflow are:

1. The Usage of the Endor Labs GitHub action.
2. Setting Job level permissions to allow writing to the GitHub id-token

name: Example Scan of OWASP Java
on: workflow_dispatch
jobs:
  create_project_owasp:
    permissions:
      id-token: write # This is required for requesting the JWT
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3
        with:
          repository: OWASP-Benchmark/BenchmarkJava
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Compile Package
        run: mvn clean install
      - name: Scan with Endor Labs
        uses: endorlabs/github-action@main # This workflow uses the Endor Labs GitHub action to scan.
        with:
          namespace: 'demo'
          scan_summary_output_type: 'json'
          pr: false
          scan_secrets: true
          scan_dependencies: true

Now that you’ve successfully configured your GitHub action workflow file you can use this workflow file or one of your own designs to run a test scan using Keyless authentication for GitHub actions.

Enabling Keyless Authentication in AWS

To enable Keyless Authentication in AWS you’ll first need permissions to create or modify the following roles and an instance profile with the appropriate roles configured.

1. An instance access role - The instance access role is assigned to the compute resource, which needs to access Endor Labs. Your instance access role may already exist and you must ensure this role provides the permissions to allow the role to assume the role of a dedicated federation role.
2. A dedicated federation role - The dedicated federation role should have no permissions in AWS. Endor Labs will authorize requests that come from this role.

Procedure:

1. Create or modify an existing Instance Profile that may be used to assign a role to your EC2 instance.
2. Create or modify a role an instance access role, which enables services to assume a dedicated federation role.
3. Assign this role to the Instance Profile
4. Create a dedicated federation role to provide access to Endor Labs, which will be assumed by the instance access role
5. Create an authorization policy in Endor Labs
6. Test Keyless Authentication

Create or select an Instance Profile

An instance profile allows users to attach a single role to an EC2 instance. If you do not already have a pre-defined role or instance profile used by your EC2 instances you should create an instance profile for Endor Labs access.

To create an instance profile using the AWS CLI:

aws iam create-instance-profile --instance-profile-name EndorLabsAccessProfile

Create or modify an instance access role

To successfully authenticate to Endor Labs you will need to assign an instance access role to the instance profile you created above.

The instance access role must at a minimum allow the compute resources that require access to perform the action sts:AssumeRole. If you already have a role you intend to assign to your instance profile, ensure that it has permissions to allow your compute resources to perform this action.

If you do not have an existing role you intend to use, create the role named endorlabs-instance-access-role using the instructions below.

Add the following json to a file called endorlabs-instance-access-role.json

cat > endorlabs-instance-access-role.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Use this file to create your instance access role using the following command:

aws iam create-role --role-name endorlabs-instance-access-role --assume-role-policy-document file://endorlabs-instance-access-role.json

Next, ensure that the instance access role is assigned to the instance profile using the following command:

aws iam add-role-to-instance-profile --instance-profile-name EndorLabsAccessProfile --role-name endorlabs-instance-access-role

Finally, create your EC2 instance and ensure that your instance profile is assigned to it.

Create a dedicated federation role

A dedicated federation role is leveraged to provide a least privileged role that enables access to Endor Labs. This role is designed to be assumed only by specific other roles and does not provide access to AWS resources.

To create your federation role you will need the name and AWS account number of the instance access role, which should look similar to the following policy json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$ACCOUNT:role/$ROLE_NAME"
            },
            "Action": "sts:AssumeRole"
        }
    ]
  }

First, get the account number of the role:

export ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')

Then define the name of the instance access role. For the following example, we will assume it is endorlabs-instance-access-role.

export ROLE_NAME=endorlabs-instance-access-role

Next, create the IAM policy document.

cat > endorlabs-federation-aws-role.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::${ACCOUNT}:role/${ROLE_NAME}"
          },
          "Action": "sts:AssumeRole"
      }
  ]
}
EOF

Next, apply the policy document as a role:

aws iam create-role --role-name endorlabs-federation --assume-role-policy-document file://endorlabs-federation-aws-role.json

Finally, fetch the ARN of the IAM role you’ve created using the following command and create an authorization policy for it in Endor Labs.

To fetch the ARN of the Endor Labs federation role use the following command:

aws iam list-roles | jq -r '.Roles[] | select(.Arn|contains("endorlabs-federation"))'.Arn

To add the authorization policy to Endor Labs:

1. Login to Endor Labs as an administrator.
2. Under Manage, navigate to Access Control > Auth Policy
3. Click Add Auth Policy
4. Under Identity Provider Select AWS role
5. Provide the appropriate permissions for your authorization policy.
6. Under claims use the Key User and the value of the ARN that you fetched in the previous command.
7. Click Save Auth Policy to finalize your keyless authentication setup.

Testing Keyless Authentication with AWS

On the EC2 instance you’ve configured for keyless authentication, download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --aws-role-arn=<insert-your-arn> for federated access to Endor Labs such as in the below example:

endorctl --aws-role-arn=<insert-your-arn> api list -r Project -n <insert-your-endorlabs-tenant> --page-size=1

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication with AWS.

2.2 - Scanning with GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can use GitHub Actions to include Endor Labs into your CI pipeline seamlessly.

Using this pipeline, developers can view and detect:

• Policy violations in the source code
• Secrets inadvertently included in the source code

The Endor Labs verifications are conducted as automated checks and help you discover violations before pushing code to the repository. Information about the violations can be included as comments on the corresponding pull request (PR). This enables developers to easily identify issues and take remedial measures early in the development life cycle.

• For policy violations, the workflow is designed to either emit a warning or return an error based on your action policy configurations.
• For secrets discovered in the commits, developers can view the PR comments and take necessary remedial measures.

To start using Endor Labs with GitHub:

• Install Software Prerequisites
• Example GitHub Action Workflow
• Authenticate with Endor Labs
  • Authentication Without Keyless Authentication for GitHub
• Endor Labs GitHub Action Configuration Parameters
  • Common parameters
  • Scanning parameters
• Configure Endor Labs Action Policies
• PR Comments
  • Enable PR Comments
  • PR Comments Example
  • Customize GitHub PR comments notification templates
    • Data model

Install Software Prerequisites

To ensure the successful execution of the Endor Labs GitHub action, the following prerequisites must be met:

• The GitHub action must be able to authenticate with the Endor Labs API.
• You must have the value of the Endor Labs namespace handy for authentication.
• You must have access to the Endor Labs API.
• If you use keyless authentication, you must set an authorization policy in Endor Labs. See Authorization policies for details.

Example GitHub Action Workflow

Endor Labs scanning workflow using GitHub actions that accomplishes the following tasks in your CI environment:

• Tests PRs to the default branch and monitors the most recent push to the default branch.
• Builds a Java project and sets up the Java build tools. If your project is not on Java, then configure this workflow with your project-specific steps and build tools.
• Authenticates to Endor Labs with GitHub Actions keyless authentication.
• Scan with Endor Labs.
• Comments on PRs if any policy violations occur.
• Generates findings and uploads results to GitHub in SARIF format.

The following example workflow shows how to scan with Endor Labs for a Java application using the recommended keyless authentication for GitHub actions:

name: Endor Labs Dependency and Secrets Scan
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload Sarif artifact to GitHub
      contents: read # Used to check out a private repository
      actions: read # Required for private repositories to upload Sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication with Endor Labs
      pull-requests: write # Required to automatically comment on PRs for new policy violations
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: Endor Labs Scan Pull Request
      if: github.event_name == 'pull_request'
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: true
        enable_pr_comments: true # Required to automatically comment on PRs for new policy violations
        github_token: ${{ secrets.GITHUB_TOKEN }} # Required for PR comments on new policy violations

  scan-main:
    permissions:
      id-token: write
      repository-projects: read
      pull-requests: read
      contents: read
    name: endorctl-scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: 'Endor Labs Scan Push to main'
      if: ${{ github.event_name == 'push' }}
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: false
        scan_summary_output_type: 'table'
        sarif_file: 'findings.sarif'
    - name: Upload findings to github
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'findings.sarif'

Authenticate with Endor Labs

Endor Labs recommends using keyless authentication in CI environments. Keyless authentication is more secure and reduces the cost of secret rotation. To set up keyless authentication see Keyless Authentication.

If you choose not to use keyless authentication, you can configure an API key and secret in GitHub for authentication as outlined in Managing API keys.

Authentication Without Keyless Authentication for GitHub

If you are not using keyless authentication for GitHub Actions, you must not provide id-token: write permissions to your GitHub token unless specifically required by a step in this job. You must also set enable_github_action_token: false in your Endor Labs GitHub action configuration.

The following example configuration uses the Endor Labs API key for authentication:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          api_key: ${{ secrets.ENDOR_API_CREDENTIALS_KEY }}
          api_secret: ${{ secrets.ENDOR_API_CREDENTIALS_SECRET }}
          enable_github_action_token: false

The following example configuration uses a GCP service account for keyless authentication to Endor Labs:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          gcp_service_account: '<Insert_Your_Service_Account>@<Insert_Your_Project>.iam.gserviceaccount.com'
          enable_github_action_token: false

Endor Labs GitHub Action Configuration Parameters

The following input configuration parameters are supported for the Endor Labs GitHub Action:

Common parameters

The following input global parameters are supported for the Endor Labs GitHub action:

Scanning parameters

The following input parameters are also supported for the Endor Labs GitHub action when used for scanning:

Configure Endor Labs Action Policies

Configure an action policy in the Endor Labs user interface to perform an action when a rule is triggered. See Action Policies for details on action policies.

• Set the Policy Template to Detected Secrets and select the Template Parameters as desired.
• Choose Enforce Policy and
  • Select Warn as the recommended action.
  • Select Break the Build to fail the build CI pipeline.

PR Comments

You can configure GitHub Actions to comment on PRs if there are any policy violations. When you raise a PR, Endor Labs scans and detects policy violations. A comment is added in the PR with the details of violation. The CI pipeline warns you or fails the build based on your action policy configuration. The PR comments also include recommendations to help you take necessary remedial actions.

You can customize the template of the comment according to your requirement.

Enable PR Comments

Make sure that your GitHub action workflow includes the following configuration.

• The workflow must have a with clause including: enable_pr_comments to true to publish new findings as review comments and github_token: ${{ secrets.GITHUB_TOKEN }}. This token is automatically provisioned by GitHub when using GitHub actions. See GitHub configuration parameters for more information.
• To grant Endor Labs the ability to comment on PRs you must include the permission pull-requests: write.

The following example configuration comments on PRs if a policy violation is detected.

      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example' # Update with your Endor Labs namespace
          scan_summary_output_type: 'table'
          scan_dependencies: true
          scan_secrets: true
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

PR Comments Example

You can view a live example of a GitHub repository with a workflow that enables PR comments.

The main.yaml file contains the following configuration to enable PR comments.

name: Build Release
on:
  pull_request:
    branches: [main]
  workflow_dispatch:
  push:
    branches: [main]
  schedule:
    - cron: "23 23 * * 0"
jobs:
  build:
    permissions:
      pull-requests: write
      security-events: write
      contents: read
      id-token: write
      actions: read
    runs-on: ubuntu-latest
    env:
      ENDOR_NAMESPACE: "endorlabs-hearts-github"
    steps:
      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1
        with:
          namespace: ${{ env.ENDOR_NAMESPACE }}
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

The PR #10 introduced a reachable vulnerability. Since the workflow has enable_pr_comments set as true, a comment is added to the PR on the policy violation.

You can expand the comment to view more details about the violation and take steps to resolve the issue.

Customize GitHub PR comments notification templates

Endor Labs provides a default template with standard information that will be included in your pull requests as comments. You can use the default template, or you can choose to edit and customize this template to fit your organization’s specific requirements. You can also create custom templates using Go Templates.

1. Sign in to Endor Labs and navigate to Manage>Integrations
2. Look for GitHub PR comments under Notifications.
3. Click Edit Template.
4. Make the required changes and click Save Template.
5. Click Restore to Default to revert the changes.
6. Use the download icon in the top right corner to download this template.
7. Use the copy icon to copy the information in the template.

Data model

To create custom templates for PR comments, you must understand the data supplied to the template.

See the following protobuf specification for the GithubCommentData message that this template uses.

syntax = "proto3";

package internal.endor.ai.endor.v1;

import "google/protobuf/wrappers.proto";
import "spec/internal/endor/v1/finding.proto";
import "spec/internal/endor/v1/package_version.proto";

option go_package = "github.com/endorlabs/monorepo/src/golang/spec/internal.endor.ai/endor/v1";
option java_package = "ai.endor.internal.spec";

// The list of finding UUIDs.
message FindingUuids {
  repeated string uuids = 1;
}

// The map of dependency name to findings.
message DependencyToFindings {
  map<string, FindingUuids> dependency_to_findings = 1;
}

// The map of PackageVersion UUID to DependencyToFindings.
message PackageToDependencies {
  map<string, DependencyToFindings> package_to_dependencies = 1;
}

message GithubCommentData {
  // The header of the PR comment. Identifies the PR comment published by Endor Labs.
  // It should always be at top of the template.
  google.protobuf.StringValue comment_header = 1;

  // The footer of the PR comment.
  google.protobuf.StringValue comment_footer = 2;

  // The map of finding UUID to finding object.
  map<string, internal.endor.ai.endor.v1.Finding> findings_map = 3;

  // The map of policy UUID to policy name.
  // This will contain only the policies that are triggered or violated.
  map<string, string> policies_map = 4;

  // The map of policy UUID to the list of finding UUIDs.
  map<string, FindingUuids> policy_findings_map = 5;

  // The map of PackageVersion UUID to PackageVersion object.
  map<string, internal.endor.ai.endor.v1.PackageVersion> package_versions_map = 6;

  // The data needs to be grouped as follows:
  //
  // - Policy 1
  // 		- Package 1
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 2
  //			- Dependency Package 2
  //				- Finding 3
  //				- Finding 4
  // 		- Package 2
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 5
  // - Policy 2
  //		....
  //
  //		Map 0[PolicyUUID]/Map 1[PkgVerUUID]/Map 2 [Dep Names]/Finding UUID
  map<string, PackageToDependencies> data_map = 7;

  google.protobuf.StringValue api_endpoint = 8;
}

To understand Finding and PackageVersion definitions that are used in this protobuf specification, see:

• Finding resource kind
• PackageVersion resource kind

See the following specification to understand the additional functions that are also available. You can access these functions by using their corresponding keys.

// FuncMap contains the additional functions that are available to GithubCommentTemplate.
var FuncMap = template.FuncMap{
	"now": toTime, // 'now' gives the current time

	// 'enumToString' coverts the enums for finding level, finding category and finding tags to string
	"enumToString": enumToString,

	// 'getPackageVersionURL' returns the URL for a given PackageVersion
	"getPackageVersionURL": func(apiURL string, packageVersion *endorpb.PackageVersion) string {
		result, err := common.GetPackageVersionURL(apiURL, packageVersion)
		if err != nil {
			return ""
		}
		return result
	},

	// 'getFindingURL' returns the URL for a given Finding
	"getFindingURL": func(apiURL string, finding *endorpb.Finding) string {
		result, err := common.GetFindingURL(apiURL, finding)
		if err != nil {
			return ""
		}
		return result
	},

	// 'add' returns the sum of two integers
	"add": func(n int, incr int) int {
		return n + incr
	},

	// 'getOtherFindingsPackageMarker' returns the key for _findingsWithNoPackages for lookup in DataMap
	// Not all findings are associated with a PackageVersion, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsPackageMarker": func() string { return _findingsWithNoPackages },

	// 'getOtherFindingsDependencyMarker' returns the key for _findingsWithNoDeps for lookup in DataMap
	// Not all findings are associated with a dependency, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsDependencyMarker": func() string { return _findingsWithNoDeps },

	// 'getFindingsCountString' returns a string with number of findings, example - "5 findings"
	"getFindingsCountString": func(dataMap *endorpb.PackageToDependencies) string {
		count := 0

		for _, depMap := range dataMap.PackageToDependencies {
			for _, findingMap := range depMap.DependencyToFindings {
				count += len(findingMap.Uuids)
			}
		}

		findingsStr := "findings"
		if count == 1 {
			findingsStr = "finding"
		}

		return fmt.Sprintf("%d %s", count, findingsStr)
	},
}

2.3 - Scanning with Jenkins

Jenkins is an open-source automation server widely used for building, testing, and deploying software. Specifically in the context of CI/CD pipelines, Jenkins serves as a powerful tool to automate various stages of the software development lifecycle.

To integrate Endor Labs into your Jenkins CI/CD processes:

1. Authenticate to Endor Labs
2. Install NodeJS plugin in Jenkins.
3. Install your build toolchain
4. Build your code
5. Scan with Endor Labs

Authenticate to Endor Labs

To configure keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in Jenkins for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.

1. In your Jenkins environment, navigate to Manage Jenkins.
2. Enter a credential name for reference such as endorlabs or re-use an existing context.
3. Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
4. In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
5. Select Add Environment Variable.

Install NodeJS plugin in Jenkins

See Jenkins documentation to install nodeJS plugin in Jenkins. You must have the nodeJS plugin to use npm and download endorctl.

Configure your Jenkins pipeline

To create a Jenkins pipeline:

1. Create a configuration pipeline file in your repository if you do not already have one using the pipeline project.
2. In your configuration pipeline file customize the job configuration based on your project’s requirements using one of the examples, simple Jenkins configuration or Jenkins pipeline using curl.
3. Ensure that the context you created is part of the workflow if you are not using keyless authentication.
4. Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
5. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
6. Update your default branch from main if you do not use main as the default branch name.
7. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Examples

Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.

Simple Jenkins configuration using npm

pipeline {
    agent any
    tools {nodejs "NodeJS"}
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY_1')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET_1')
    }
    stages {
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                script {
                    // Define the Node.js installation name configured in Jenkins
                    NODEJS_HOME = tool name: 'NodeJS', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
                    PATH = "$NODEJS_HOME/bin:${env.PATH}"
                }

                // Download and install endorctl.
                sh 'npm install -g endorctl'
                // Check endorctl version and installation.
                sh 'endorctl --version'
                // Run the scan.
                sh('endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET')

            }
        }

        stage('Results') {
            steps {
                // Publish or process the vulnerability scan results
                // Publish reports, fail the build on vulnerabilities, etc.
                echo 'Publish results'
            }
        }
    }

}

Jenkins pipe line for curl to download endorctl binary

The following example includes curl to download the endorctl binary.

pipeline {
    agent any

    // Endorctl scan uses following environment variables to the trigger endorctl scan
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET')
    }
    stages {
        // Not required if repository is allready cloned to trigger a endorctl scan
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            // Not required if project is already built
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                // Download and install endorctl.
                sh '''#!/bin/bash
                    echo "Downloading latest version of endorctl"
                    VERSION=$(curl $ENDOR_API/meta/version | jq -r '.Service.Version')
                    ENDORCTL_SHA=$(curl $ENDOR_API/meta/version | jq -r '.ClientChecksums.ARCH_TYPE_LINUX_AMD64')
                    curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
                    echo "$ENDORCTL_SHA  endorctl" | sha256sum -c
                    if [ $? -ne 0 ]; then
                        echo "Integrity check failed"
                        exit 1
                    fi
                    chmod +x ./endorctl
                    // Check endorctl version and installation.
                    ./endorctl --version
                    // Run the scan.
                    ./endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
                '''
            }
        }
    }
}

Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.

2.4 - Scanning in GitLab Pipelines

GitLab CI/CD pipelines are a part of GitLab’s integrated continuous integration and deployment features. They allow you to define and automate the different stages and tasks in your software development workflow.

This documentation page provides an overview and example job to integrate Endor Labs into your GitLab CI pipeline.

High Level Usage Steps

1. Setup authentication to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authentication to Endor Labs

Endor Labs recommends using keyless authentication in continuous integration environments.

Keyless Authentication is more secure and reduces the cost of secret rotation.

To setup keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in GitLab for authentication using the following steps. See managing API keys for more information on getting an API key for Endor Labs authentication

1. In your GitLab environment, select the project you want to scan.
2. Go to Settings > CI/CD.
3. Click Expand in the Variables section.
4. Click the Add variable button at the bottom of the section.
5. In the Key field, enter ENDOR_API_CREDENTIALS_SECRET.
6. In the Value field, enter your Endor Labs API secret.
7. Under Flags, make sure you select Mask variable.
8. Repeat the previous steps to add your API key as the variable ENDOR_API_CREDENTIALS_KEY.

Configuring your GitLab CI pipeline

1. Create a .gitlab-ci.yml file in the root directory of your project if you do not already have one.
2. In your .gitlab-ci.yml file customize the job configuration based on your project’s requirements using the example below.
3. Modify the image field to align with the build tools necessary for building your software packages.
4. Update the before_script section to include any additional steps required before executing the scan, such as installing dependencies or building your project.
5. Save and commit the .gitlab-ci.yml file to your GitLab repository.
6. GitLab will automatically detect the .gitlab-ci.yml file and trigger the defined job whenever there are changes pushed to the repository.
7. Monitor the progress and results of the CI pipeline in the GitLab CI/CD interface.

Below is an example job to get you started. Please ensure to customize this job with your specific build environment and build steps as needed.

# You can copy and paste this template into a new `.gitlab-ci.yml` file.
# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword.
#
stages:
  - Scan
Endor Labs Dependency Scan:
  stage: test
  image: node # Modify this image to align with the build tools nessesary to build your software packages
  dependencies: []
  variables:
   ## Scan scoping section
   #
   ## Use the following environment variables for custom paths, inclusions and exclusions.
   # ENDOR_SCAN_PATH: "Insert a custom path to your git repository. Defaults to your pwd"
   # ENDOR_SCAN_EXCLUDE_PATH: "Insert a Glob style pattern of paths to exclude in the scan. Generally used for monorepos."
   # ENDORCTL_SCAN_INCLUDE_PATH: "Insert a Glob style pattern of paths to include in the scan. Generally used for monorepos."
   #
   ## Authentication to Endor Labs
   #
   ## Use the following environment variables for keyless authentication with your cloud provider. For more information visit: https://docs.endorlabs.com/continuous-integration/keyless-authentication/
   #
   # ENDOR_GCP_CREDENTIALS_SERVICE_ACCOUNT: "endorlabs@<yourproject.iam.gserviceaccount.com"
   # ENDOR_AWS_CREDENTIALS_ROLE_ARN: "arn:aws:iam::123456789012:role/my-role"
   #
   ## Follow the below instructions to use an API key and secret instead of keyless authentication.
   ## In your GitLab environment, select the project you want to scan.
   ## Go to `Settings` > `CI/CD`.
   ## Click `Expand` in the `Variables` section.
   ## Click the `Add variable` button at the bottom of the section.
   ## In the `Key` field, enter ENDOR_API_CREDENTIALS_SECRET.
   ## In the `Value` field, enter your Endor Labs API secret.
   ## Under `Flags` make sure you select `Mask variable`.
   ## Repeat to add your API key as the variable ENDOR_API_CREDENTIALS_KEY
   #
    ENDOR_ENABLED: "true"
    ENDOR_ALLOW_FAILURE: "false"
    ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    ENDOR_PROJECT_DIR: "."
    ENDOR_ARGS: |
      --path=${ENDOR_PROJECT_DIR}
      --detached-ref-name=$CI_COMMIT_REF_NAME
      --output-type=summary
      --exit-on-policy-warning
      --dependencies --secrets --git-logs      
  before_script:
    - npm install yarn # Replace with the build steps for your Endor Labs job.
  script:
    - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl;
    - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
      if [ $? -ne 0 ]; then
       echo "Integrity check failed";
       exit 1;
      fi
    - chmod +x ./endorctl
    - if [ "$DEBUG" == "true" ]; then
        export ENDOR_LOG_VERBOSE=true;
        export ENDOR_LOG_LEVEL=debug;
      fi
    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi
    - ./endorctl scan ${ENDOR_ARGS}
  rules:
  - if: $ENDOR_ENABLED != "true"
    when: never
  - if: $ENDOR_ALLOW_FAILURE == "true"
    allow_failure: true
  - if: $ENDOR_ALLOW_FAILURE != "true"
    allow_failure: false

2.5 - Scanning in Azure Pipelines

Azure Pipelines is a continuous integration and continuous delivery (CI/CD) service available in Azure DevOps ecosystem. It facilitates continuous integration, continuous testing, and continuous deployment for seamless building, testing, and delivery of software.

To integrate Endor Labs into an Azure pipeline:

• Complete the Prerequisites
  • Set up an Endor Labs tenant
  • Configure Endor Labs authentication
  • Enable Advanced Security in Azure
  • Configure Endor Labs variables in the pipeline
• Configure your Azure pipeline
  • Azure Pipeline Examples
• View scan results in Azure

Complete the Prerequisites

Ensure that you complete the following prerequisites before you proceed.

Set up an Endor Labs tenant

You must have an Endor Labs tenant set up for your organization. You can also set up namespaces according to your requirements. See Set up namespaces

Configure Endor Labs authentication

Configure an API key and secret for authentication. See managing API keys for more information on generating an API key for Endor Labs. Store API key and secret as environment variables, ENDOR_API_CREDENTIALS_KEY and ENDOR_API_CREDENTIALS_SECRET.

Enable Advanced Security in Azure

You need to enable Advanced Security in your Azure repository to view results in Azure.

1. Log in to Azure and open Project Settings.
2. Navigate to Repos > Repositories in the left navigation panel.
3. Select your repository.
4. Enable Advanced Security.

Configure Endor Labs variables in the pipeline

You can manage Endor Labs variables centrally by configuring them within your Azure project. You can assign these variables to various pipelines.

1. Log in to Azure and select Pipelines > Library.
2. Click +Variable Group to add a new variable group for Endor Labs.
3. Enter a name for the variable group, for example, tenant-variables, and click Add under Variables.
4. Add the following variables.
   • ENDOR_API_CREDENTIALS_KEY
   • ENDOR_API_CREDENTIALS_SECRET
   • NAMESPACE
5. Select the variable group that you created.
6. Click Pipeline Permissions.
7. Click + to add the pipelines in which you want to use the variable group.

Configure your Azure pipeline

1. Create azure-pipelines.yml file in your project, if it doesn’t exist.
2. In the azure-pipelines.yml file, customize the job configuration based on your project’s requirements.
3. Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project. For example, update the node pool settings based on your operating system.

• Windows
• Ubuntu
• macOS

pool:
name: Default
vmImage: "windows-latest"

pool:
name: Default
vmImage: "ubuntu-latest"

pool:
name: Default
vmImage: "macOS-latest"

4. Update your default branch from main if you do not use main as the default branch name.
5. Modify any dependency or artifact caches to align with the languages and caches used by your project.
6. Enter the following steps in the azure-pipelines.yml file to download endorctl.

• Windows
• Ubuntu
• macOS

- bash: |
    echo "Downloading latest version of endorctl"
    VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
    curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
    echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe)  endorctl" | sha256sum -c
    if [ $? -ne 0 ]; then
      echo "Integrity check failed"
      exit 1
    fi    

- bash: |
    echo "Downloading latest version of endorctl"
    VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
    curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
    echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
    if [ $? -ne 0 ]; then
      echo "Integrity check failed"
      exit 1
    fi    

- bash: |
    echo "Downloading latest version of endorctl"
    VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
    curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
     echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64)  endorctl" | shasum -a 256 --check
    if [ $? -ne 0 ]; then
      echo "Integrity check failed"
      exit 1
    fi    

7. Enter the steps to build your project if your project needs building and setup steps.

8. Enter the following step in the azure-pipelines.yml file to run endorctl scan to generate the SARIF file.

   You can run endorctl scan with options according to your requirement, but you must include the -s option to generate the SARIF file.

   For example, use the --secrets flag to scan for secrets.

• Windows
• Ubuntu
• macOS

- script: |
    .\endorctl.exe scan -n $(NAMESPACE) -s scanresults.sarif
    env:
      ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
      ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)    

- script: |
    .\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
    env:
      ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
      ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)    

- script: |
    .\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
    env:
      ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
      ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)    

1. Enter the following task in the azure-pipelines.yml to publish the scan results.

   - task: AdvancedSecurity-Publish@1
       displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
       inputs:
         SarifsInputDirectory: $(Build.SourcesDirectory)\
   

After a successful run of the pipeline, you can view the results in Azure.

Azure Pipeline Examples

• Windows
• Ubuntu
• macOS

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "windows-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
    - bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
       echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe)  endorctl" | sha256sum -c
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
    ```    
  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    .\endorctl.exe scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)\

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "ubuntu-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
    - bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
        ## Modify the permissions of the binary to ensure it is executable
        chmod +x ./endorctl
        ## Create an alias of the endorctl binary to ensure it is available in other directories
        alias endorctl="$PWD/endorctl"    

  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    ./endorctl scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)/

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "macos-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64)  endorctl" | shasum -a 256 --check
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
        ## Modify the permissions of the binary to ensure it is executable
        chmod +x ./endorctl
        ## Create an alias of the endorctl binary to ensure it is available in other directories
        alias endorctl="$PWD/endorctl"        
  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    ./endorctl scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)/

View scan results in Azure

After the pipeline runs, you can view the scan results in Azure.

1. Log in to Azure and navigate to your projects.
2. Select Repos > Advanced Security to view the scan results.
3. Click an alert to view more details.
4. If you ran endorctl with --secrets flag, you can view if there are any secret leaks. Click the entry to view more details.

2.6 - Scanning in Bitbucket Pipelines

Bitbucket Pipelines is a continuous integration and continuous delivery (CI/CD) service built into Bitbucket. It allows developers to automatically build, test, and deploy their code based on a configuration file bitbucket-pipelines.yml defined in the root of their repository.

To integrate Endor Labs into a Bitbucket pipeline:

1. Authenticate to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authenticate to Endor Labs

Configure an API key and secret in the bitbucket-pipelines.yml file for authentication. See managing API keys for more information on generating an API key for Endor Labs.

Configure your Bitbucket pipeline

To create a Bitbucket pipeline reference the following steps:

1. Create a bitbucket-pipelines.yml file in your repository if you do not already have one.
2. In your bitbucket-pipelines.yml file customize the job configuration based on your project’s requirements using the following example.
3. Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project.
4. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
5. Update your default branch from main if you do not use main as the default branch name.
6. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Example

Use the following example to get started. Make sure to customize this job with your specific build environment and build steps.

Bitbucket configuration

simage: maven:3.6.3-jdk-11

pipelines:
  branches:
    main:
      - step:
          name: "Build and Test"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
  pull-requests:
    '**':
      - step:
          name: "Build and Test on PR to Main"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs PR Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan --pr --pr-baseline=main --languages=java --output-type=json -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET | tee output.json
            #Optional - Comment on the PR
            # - apt-get update
            # - apt-get install -y python3 python3-pip
            # - pip3 install -r requirements.txt
            # - python3 add-bitbucket-pr-comments.py output.json

Once you’ve set up Endor Labs, you can test your CI implementation to ensure it is successful and then proceed with your scans.

View PR comments for policy violations

You can also use the Insights feature in Bitbucket Pipelines to indicate if the changes in your pull requests violated any policies set in Endor Labs.

import json
import os
import sys
import requests
import argparse

# Check for required environment variables
BITBUCKET_REPO_OWNER = os.getenv('BITBUCKET_REPO_OWNER')
BITBUCKET_REPO_SLUG = os.getenv('BITBUCKET_REPO_SLUG')
BITBUCKET_COMMIT = os.getenv('BITBUCKET_COMMIT')

if not all([BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT]):
     sys.exit("Error: One or more required environment variables (BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT) are not set.")

#This is an internal proxy running in the BitBucket environment to accept Code Insights
proxies = {"http": "http://localhost:29418"}

def load_json_with_unescaped_characters(file_path):
    """Load and return JSON data from a file, replacing unescaped characters if necessary."""
    try:
        with open(file_path, 'r', encoding='utf-8') as file:
            json_str = file.read().strip()
        return json.loads(json_str)
    except json.decoder.JSONDecodeError as e:
        print(f"Failed to parse JSON: {e}")
        return None
    except FileNotFoundError:
        print(f"File not found: {file_path}")
        sys.exit()

def construct_report_payload(endor_findings):
    """Construct and return the payload for creating a Bitbucket report."""
    warning_findings_count = len(endor_findings.get('warning_findings', []))
    blocking_findings_count = len(endor_findings.get('blocking_findings', []))
    total_violations = warning_findings_count + blocking_findings_count
    result = "PASSED" if total_violations == 0 else "FAILED"
    report_payload = {
        "title": "Endor Labs Policy Violations",
        "details": f"Endor Labs detected {total_violations} policy violations associated with this pull request.\n\n{endor_findings['warnings'][0]}",
        "report_type": "SECURITY",
        "reporter": "Endor Labs",
        "link": f"https://app.endorlabs.com/t/{namespace}/projects/{project_uuid}/pr-runs/{report_id}",
        "logo_url": "https://avatars.githubusercontent.com/u/92199924",
        "result": result,
        "data": [
            {"title": "Warning Findings", "type": "NUMBER", "value": warning_findings_count},
            {"title": "Blocking Findings", "type": "NUMBER", "value": blocking_findings_count}
        ]
    }
    return report_payload

def construct_annotation_payload(finding):
    """Construct and return the payload for creating an annotation in Bitbucket."""
    title = "Endor Labs Policy Violation"
    summary = finding['meta']['description']
    details =  f"{finding['spec']['summary']}\n\n{finding['spec']['remediation']}"
    severity = "CRITICAL" if finding['spec']['level'] == "FINDING_LEVEL_CRITICAL" else \
               "HIGH" if finding['spec']['level'] == "FINDING_LEVEL_HIGH" else \
               "MEDIUM" if finding['spec']['level'] == "FINDING_LEVEL_MEDIUM" else "LOW"
    affected_paths = finding['spec'].get('dependency_file_paths', [])
    path = affected_paths[0] if affected_paths else "Unknown file"
    annotation_payload = {
        "external_id": finding['uuid'],
        "title": title,
        "annotation_type": "VULNERABILITY",
        "summary": summary,
        "details": details,
        "severity": severity,
        "path": path
    }
    return annotation_payload

def send_report(report_payload):
    """Send the constructed report payload to the Bitbucket API."""
    report_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports/{report_id}"
    response = requests.put(report_url, json=report_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Report created or updated successfully")
    else:
        print(f"Failed to create or update report: {response.text}")

def send_annotation(annotation_payload):
    """Send the constructed annotation payload to the Bitbucket API."""
    annotation_url = f"{base_url}/{report_id}/annotations/{annotation_payload['external_id']}"
    response = requests.put(annotation_url, json=annotation_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Annotation added successfully")
    else:
        print(f"Failed to add annotation: {response.text}")

def process_findings(filename):
    """Load findings from JSON, create a report, and add annotations for each finding."""
    endor_findings = load_json_with_unescaped_characters(filename)
    if endor_findings is None:
        print("Failed to load findings. Exiting.")
        return

    global report_id, project_uuid, namespace

    # Define the order of keys to check
    finding_types = ['all_findings', 'warning_findings', 'blocking_findings']

    # Iterate over finding types and extract the first one found
    for finding_type in finding_types:
        if endor_findings.get(finding_type):
            first_finding = endor_findings[finding_type][0]
            report_id = first_finding['context']['id']
            project_uuid = first_finding['spec']['project_uuid']
            namespace = first_finding['tenant_meta']['namespace']
            break  # Stop after finding the first non-empty list

    if not report_id:
        print("No findings found.")
        sys.exit()

    # Prepare the base URL for Bitbucket API requests
    global base_url
    base_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports"

    # Create the report
    report_payload = construct_report_payload(endor_findings)
    send_report(report_payload)

    # Iterate over findings and create annotations
    for finding in endor_findings.get('blocking_findings', []) + endor_findings.get('warning_findings', []):
        annotation_payload = construct_annotation_payload(finding)
        send_annotation(annotation_payload)

def main():
    """Main function to parse arguments and process findings."""
    parser = argparse.ArgumentParser(description="Script to process findings and update Bitbucket via API.")
    parser.add_argument("filename", help="Filename containing the JSON findings.")
    args = parser.parse_args()

    process_findings(args.filename)

if __name__ == "__main__":
    main()

2.7 - Scanning with CircleCI

CircleCI CI/CD pipelines allow you to configure your pipeline as code. Your entire CI/CD process is orchestrated through a single file called config.yml. The config.yml file is located in a folder called .circleci at the root of your project which defines the entire pipeline.

To integrate Endor Labs into your CircleCI CI/CD processes:

1. Authenticate to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authenticate to Endor Labs

Endor Labs recommends using keyless authentication in continuous integration environments. Keyless Authentication is more secure and reduces the cost of secret rotation but is only available on self-hosted runners in CircleCI.

To configure keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in CircleCI for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.

1. In your CircleCI environment, navigate to Organizational Settings.
2. From Contexts and select Create Context.
3. Enter a context name for reference such as endorlabs or re-use an existing context.
4. Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
5. In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
6. Select Add Environment Variable.
7. Repeat the previous 3 steps to add your API key secret as the environment variable ENDOR_API_CREDENTIALS_SECRET. Have the name of the context handy to reference in the workflows later.

Configure your CircleCI pipeline

To create a CircleCI pipeline reference the following steps:

1. Create a .cirlceci/config.yml file in your repository if you do not already have one.
2. In your config.yml file customize the job configuration based on your project’s requirements using one of the examples, simple CircleCI configuration or advanced CircleCI configuration.
3. Create two workflows called build_and_watch_endorlabs and build_and_test_endorlabs.
4. Ensure that the context you created is part of the workflow if you are not using keyless authentication.
5. Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
6. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
7. Update your default branch from main if you do not use main as the default branch name.
8. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Examples

Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.

Simple CircleCI configuration

version: 2.1

jobs:
  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            
  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - watch-endorlabs-scan:
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - test-endorlabs-scan:
          context:
            - endorlabs

Advanced CircleCI configuration

The following example is an advanced implementation of Endor Labs in circleCI which includes several optional performance optimizations and job maintainability updates.

This includes:

1. Caching and restoring caches of jobs and artifacts to improve performance. Caches should be modified to reflect the build artifacts and dependencies of your project.
2. Segmenting jobs and scans.

# You can copy and paste portions of this `config.yml` file as an easy reference.
#
version: 2.1

jobs:
  build:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build steps
    steps:
      - checkout
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Build Your Project"
          command: |
            mvn clean install            
      - persist_to_workspace:
          root: .
          paths:
            - target/ # Persist artifact across job. Change this if you are creating your artifact in a location outside of the target directory.
      - save_cache:
          paths:
            - ~/.m2/repository
          key: maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}

  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            

  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example" #Replace with your namespace in Endor Labs
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - build
      - watch-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - build
      - test-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs

Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.

2.8 - Scanning with Google Cloud Build

Google Cloud Build is a fully managed continuous integration and continuous delivery (CI/CD) service offered by Google Cloud Platform.

To integrate Endor Labs with Google Cloud Build:

• Authenticate to Endor Labs
• Set up Google Cloud prerequisites
• Set up repositories on Google Cloud Build
• Create Cloud Build triggers
  • Baseline scan
  • PR scan
  • Release scan
  • Example configuration file

Authenticate to Endor Labs

Generate API credentials to authenticate to Endor Labs. Configure the API key and secret in the cloudbuild.yaml file for authentication. See managing API keys for more information on generating an API key for Endor Labs.

You can enable keyless authentication to Google Cloud. See Enabling Keyless Authentication in Google Cloud for more information.

Set up Google Cloud prerequisites

Ensure the following prerequisites are in place in Google Cloud Build before integrating with Endor Labs.

• GCP Service Account: Create a service account to operate Google Cloud Build.
• APIs:
  • Enable the Google Cloud Build API.
  • Enable the Secrets Manager API.
• Secrets:
  • Create secrets in Secret Manager to store the Endor Labs API credentials: endor-api-key and endor-api-secret.
• Permissions: Grant the service account the following roles:
  • Secret Manager Secret Accessor: Allows the service account to access API credentials from Secret Manager.
  • Logging Admin: Allows the service account to write build logs to Cloud Logging.

Set up repositories on Google Cloud Build

1. Sign in to the Google Cloud Build console.
2. Navigate to Repositories.
3. Follow the instructions in Connecting GitHub Repositories to Cloud Build to add the repositories you want to scan with Cloud Build.

Create Cloud Build triggers

Triggers initiate Cloud Build for different types of scans. You can set up triggers for the following scan types:

• Baseline scan
• PR scan
• Release scan

Baseline scan

• Purpose: Scans the baseline or the default branch to identify existing security vulnerabilities. Future code and dependencies will be evaluated against this baseline.
• Trigger Type: Push to branch.
• Setup: Create a trigger for the required repository and branch, for example, main, or develop.
• Cloud Build Configuration: Create a cloudbuild.yaml file using the configuration file examples as a reference. Include this file for baseline scans in the required GitHub repository.

PR scan

• Purpose: Scans the pull requests that could include new code and dependencies for vulnerabilities and security risks. This scan compares the new code against the baseline or the default branch and raises results based on findings and admission policies.
• Trigger Type: Pull request.
• Setup: Create a trigger for the required repository and branch.
• Additional Parameters: Pass extra parameters as part of the endorctl arguments.
• Cloud Build Configuration: Create a cloudbuild.yaml file using the configuration file examples as a reference. Include this file for baseline scans in the required GitHub repository,

Release scan

• Purpose: Scans code before it lands in production or pre-production environments. This is similar to a baseline scan, however, it is triggered when you push the code to a release branch or create a new release tag.
• Trigger Type: Push to branch or push to new tag.
• Setup: Create a trigger for the release branch or tag.
• Cloud Build Configuration: Create a cloudbuild.yaml file using the configuration file examples excluding the --as-default-branch argument for release scans, and add this file to the required GitHub repository.

Example configuration file

Here is an example cloudbuild.yaml configuration file to perform a baseline scan for Java project repository.

steps:
  # Step 1: Fetch The Trigger Branch
  # This step addresses a known issue where Cloud Build renames the pulled branch to main.
  # If you are not encountering this issue with your build, you can skip this step.
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        echo "Fetching all branches..."
        git fetch origin
        echo "Checking out branch: ${BRANCH_NAME}"
        git checkout ${BRANCH_NAME}
  # Step 2: Build With Maven
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'mvn'
    args: ['clean', 'install']
    id: 'Build'
  # Step 3: Install latest version of endorctl
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
        chmod +x ./endorctl
        ./endorctl --version
    id: 'Install latest version of endorctl'
  # Step 4: SCA Scan With EndorLabs
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'bash'
    args: ["-c", "./endorctl scan -n $$ENDOR_NAMESPACE --api-key=$$ENDOR_API_CREDENTIALS_KEY --api-secret=$$ENDOR_API_CREDENTIALS_SECRET --as-default-branch=true"]
    secretEnv: ['ENDOR_API_CREDENTIALS_KEY', 'ENDOR_API_CREDENTIALS_SECRET']
    env:
      - 'ENDOR_NAMESPACE=demo'
    id: 'SCA Scan With EndorLabs'

# Fetch Endor Labs API Token and Secret From Secrets Manager
availableSecrets:
  secretManager:
  - versionName: projects/{your-project-id}/secrets/endor-api-key/versions/1
    env: 'ENDOR_API_CREDENTIALS_KEY'
  - versionName: projects/{your-project-id}/secrets/endor-api-secret/versions/1
    env: 'ENDOR_API_CREDENTIALS_SECRET'

options:
  # Choose your log configuration
  logging: 'CLOUD_LOGGING_ONLY'
  # Select a private pool if the default runners do not meet the minimum requirements.
  pool:
    name: 'projects/{your-project-id}/locations/{your_location}/workerPools/{your_worker_pool_id}'

Check the example configuration files and customize them for your requirements.

3 - Scan from your IDE

Endor Labs provides developers with plug-ins that they can directly install in their Integrated Development Environments (IDE). The extension helps developers fix code at its origin phase and during the early stages of development without running the endorctl scan.

Developers can successfully perform early security reviews and mitigate the need for expensive fixes during later stages of development. It accelerates the process of creating, delivering, and shipping secure applications.

Endor Labs provides a plug-in for Visual Studio Code that developers can install from Visual Studio’s marketplace and get started with early vulnerability and dependency scanning.

Endor Labs Visual Studio Code Extension

The Endor Labs extension for Visual Studio Code scans your repositories and highlights issues that may exist in the open-source dependencies. You can use the extension with Endor Labs API credentials.

Prerequisites

The following prerequisites must be fulfilled to use the Endor Labs VS Code extension:

• The minimum supported version of Visual Studio Code is 1.71 and higher.

• See the following table for supported languages, package managers, and file extensions. The extension reads the manifest files to fetch the list of dependencies and displays the results in both manifest and source code files.

  Supported Language	Manifest file	Source code file
  JavaScript	package.json	.js, .ts, .jsx, .tsx, .mjs, .cjs extensions
  Python	requirements.txt	.py extension
  Golang	go.mod	.go extension

• Generate Endor Labs API keys and have them handy. You must enter these details in the VS Code extension. See Managing API Keys for details.

Install the Endor Labs extension

Developers can install the extension from the Visual Studio marketplace and configure it with Endor Labs API keys.

1. Launch Visual Studio Code and click Extensions.
2. Look for the Endor Labs using the search bar and click Install. See Visual Studio Extension documentation for details on managing the extension.
3. Select the Endor Labs extension, click the Settings icon, and choose Extension Settings.
4. Enter the API Key and API Secret of the Endor Labs application.

View scan results

The Endor Labs Visual Studio extension reads all the manifest files in your project and fetches the list of dependencies.

• Hover over a dependency to view the package version, released date, findings, and Endor Labs scores in a pop-up.
• For effective prioritization, issues with dependencies are classified into four severity levels: Critical, High, Medium, and Low.
• Click a specific version to view the same results in the Endor Labs application’s user interface.
• The dependencies are color-coded in the following ways:
  • Red underline - Has critical findings and is also on an outdated version
  • Orange underline - Has critical findings and is on the latest version
  • Yellow underline - Has no critical findings but is an outdated version
  • No Underline - Has no critical findings and is on the latest version
• Use Update to latest version to update the package to its latest version.

Note: The manifest file is updated with the latest version however, the package is not automatically upgraded.