Monitoring or supervisory scans

• 1: Deploy Endor Labs GitHub App
• 2: Deploy Endor Labs Azure DevOps App
• 3: Set up Jenkins pipeline for supervisory scans

1 - Deploy Endor Labs GitHub App

Endor Labs provides a GitHub App that continuously monitors users’ projects for security and operational risk. You can use the GitHub App to selectively scan your repositories for SCA, secrets, RSPM, or CI/CD tools.

Prerequisites for GitHub App

Before installing and scanning projects with Endor Labs GitHub App, make sure you have:

• A GitHub cloud account and organization. If you don’t have one, create one at GitHub.
• Administrative permissions to your GitHub organization. Installing the Endor Labs GitHub App in your organization requires approval or permissions from your GitHub organizational administrator. If you don’t have the permissions, use the command line utility, endorctl, while you wait for the approval.
• Endor Labs GitHub App requires read permissions to Dependabot alerts, actions, administration, checks, code, commit statuses, issues, metadata, packages, pull requests, repository hooks, and security events. It does not need write access to any resources.

Install the GitHub App

To automatically scan repositories using the GitHub App:

1. Sign in to Endor Labs.

2. Choose Projects and click Add Project.

3. From GITHUB, choose GitHub App

4. Click Install GitHub App.

   You will be redirected to GitHub to install the GitHub App.

   

5. Click Install.

6. Select a user to authorize the app.

7. Select the organization in which you want to install the app.

8. Select whether to install and authorize Endor Labs on all your repositories or select the specific repositories that you wish to scan.

   

9. Review the permissions required for Endor Labs and click Install and Authorize.

   If the button to install says Install and Request instead of Install and Authorize, you don’t have permission to install the GitHub App. Use the endorctl command line interface or select Install and Request to notify your organizational administrator of your request to install.

10. Choose a namespace and click Next.

    

11. Based on your license, select and enable from the following list of available scanners.

    • SCA- Perform software composition analysis.
    • CI/CD - Scan the repository and identify all the CI/CD tools used in the repository.
    • RSPM - Scan the repository for misconfigurations.
    • Secret - Scan the repository for exposed secrets.

    

12. Click Continue. You have successfully installed the GitHub App.

Endor Labs GitHub App scans your repositories every 24 hours and reports any new findings or changes to release versions of your code.

Manage GitHub Apps on Endor Labs

You can edit or delete the GitHub App integrations.

To edit the GitHub App integration:

1. Sign in to Endor Labs.

2. Select Manage > Integrations from the left navigation menu.

3. Click Manage next to GitHub under Source Control Managers.

   

4. Click the ellipsis on the right side, and select Edit Integration.

   

5. Based on your license, select and enable from the available list of scanners and click Save. The changes are applicable from the next scanning cycle.

6. Use Reset to clear your selection.

To delete a GitHub App integration, click the ellipsis on the right side, and select Delete Integration.

To manually trigger a scan, click Rescan Org. Endor Labs GitHub App scans your repositories every 24 hours, use Rescan Org to manually schedule outside the 24-hour period.

Click Scan More Repositories to go to Projects page, from which you can add more repositories to scan through the GitHub App.

Set up package repositories

You can improve your experience with the GitHub App by setting up package repositories. This will help you create a complete bill of materials and perform static analysis. Without setting package repositories, you may not be able to get an accurate bill of materials. See Set up package manager integration for more information.

Technical limitations of the GitHub App

The Endor Labs GitHub App provides visibility across a GitHub organization, but it has technical limitations that do not account for the unique requirements of your application.

Here are some of these limitations.

Bill of materials variance

The Endor Labs GitHub App approximates software package builds to create a bill of materials and perform static analysis on your software dependencies. This requires building packages with specific versions of the package manager and runtime environment.

If there are differences in the build environment, it can result in variances in the bill of materials. For the most accurate information, use Endor Labs CLI as a post-build step in your software delivery process.

The following factors contribute to variances in the bill of materials:

1. The time a software package was built.
2. The version of a software package manager.
3. The type of package manager being used.
4. The version of the runtime environment on which a package is installed.

Custom package build steps

Endor Labs requires executing custom build steps outside of standard package manager commands to build software packages and get an accurate bill of materials and perform static analysis. Sometimes, a complete bill of materials may not be generated, or static analysis may not be performed.

Custom resource profiles

Large applications may require significant memory allocations to perform static analysis on a package. The services scanning the GitHub App use 16 GB of memory by default. Applications that require more memory may not obtain vulnerability prioritization information using the GitHub App. Scan large applications in a CI environment using a runner with sufficient resource allocations.

Authentication for private software components

Private software components hosted in an internal package repository may require authentication credentials to create a complete bill of materials or perform static analysis.

If your authentication information to your private package repository is hosted outside the repository, you will need to configure a package manager integration. See Set up package manager integration for more details.

2 - Deploy Endor Labs Azure DevOps App

Endor Labs provides an Azure DevOps App that continuously scans Azure repos in your projects for security risks. You can selectively scan your repositories for SCA, secrets, or CI/CD tools.

When you add an Azure DevOps project to an Endor Labs namespace, Endor Labs scans all the Azure repos contained in the project. As a best practice, we recommend that you add only one Azure project to one Endor Labs namespace so that the Azure repos of that project are mapped to an Endor Labs namespace.

Prerequisites for Azure DevOps App

Ensure the following prerequisites are in place before you install the Endor Labs Azure DevOps App.

• An Azure DevOps cloud account and organization. If you don’t have one, create one at Azure DevOps.
• Endor Labs Azure DevOps App requires read permissions to in your project. You can grant these permissions by providing read access to the Code category when you create an Azure DevOps personal access token for Endor Labs.

Install the Azure DevOps App

To automatically scan repositories using the Azure DevOps App:

1. Sign in to Endor Labs.

2. Choose Projects and click Add Project.

3. From AZURE, select Azure DevOps App.

   

4. Enter the host URL of your Azure project.

   The URL must be in the format, https://dev.azure.com/<ORG_NAME>/<PROJECT_NAME>.

5. Enter your personal access token from Azure.

   You must have at least read permissions in the Code category for your Azure DevOps personal access token.

6. Click Scanners and select the scan types to enable.

   • SCA- Perform software composition analysis.
   • Secret - Scan Azure repos for exposed secrets.
   • CI/CD - Scan Azure repos and identify all the CI/CD tools used.

   The available scan types depend upon your license.

   

7. Click Create.

Endor Labs Azure DevOps App scans your Azure repos every 24 hours and reports any new findings or changes to release versions of your code.

Manage Azure DevOps Apps on Endor Labs

You can edit or delete the Azure DevOps App integrations.

To edit the Azure DevOps App integration:

1. Sign in to Endor Labs.
2. Select Manage > Integrations from the left navigation menu.
3. Click Manage next to Azure under Source Control Managers.
4. Click the ellipsis on the right side, and select Edit Integration. You can update your personal access token.
5. Click SCANNERS and based on your license, select and enable from the available list of scanners.
6. Click Save. The changes are applicable from the next scanning cycle.

To delete an Azure DevOps App integration, click the ellipsis on the right side, and select Delete Integration.

To manually trigger a scan, click Rescan Org. Endor Labs Azure DevOps App scans your repositories every 24 hours, use Rescan Org to manually schedule outside the 24-hour period.

Click Scan More Repositories to go to the Projects page, from which you can add more projects to scan through the Azure DevOps App.

3 - Set up Jenkins pipeline for supervisory scans

Use the Endor Labs Jenkins pipeline to scan all the repositories in your organization and view consolidated findings. This pipeline runs on your organization’s Jenkins infrastructure and enables administrators to run organization-level supervisory scans easily. It is designed to work in GitHub Cloud and GitHub enterprise server environments.

The Jenkins pipeline carries out the following actions.

• Pulls the Endor Labs Docker image required to perform the scan.
• Synchronizes GitHub organization repositories to a specified namespace on the Endor Labs platform.
• Retrieves the project list or the GitHub repositories for the given tenant’s namespace.
• Groups the projects into batches to optimize scan execution.
• Runs endorctl scans on each batch of projects simultaneously.

Scan the repositories in your organization

The Jenkins Pipeline script is available in the github-org-scan-docker.groovy file.

To scan the repositories in your organization:

1. Generate Endor Labs API credentials
2. Configure GitHub cloud or GitHub enterprise server credentials
3. Configure the Jenkins job

Configure GitHub credentials

Configure the required credentials needed to access GitHub and Endor Labs in the Jenkins pipeline script. You can configure these values from the Jenkins user interface.

• GITHUB_TOKEN- Enter the GitHub token that has permission to access all the repositories in the organization.
• ENDOR_LABS_API_KEY- Enter the Endor Labs API key that you generated.
• ENDOR_LABS_API_SECRET- Enter the Endor Labs API secret generated while creating the Endor Labs API key.

Configure GitHub cloud credentials

Configure the following GitHub cloud parameters in the Jenkins pipeline script.

Required Parameters for GitHub cloud

• AGENT_LABEL- This is a string parameter. Enter the label used to identify the Jenkins agents. The Jenkins job will run on the agents that have this label.
• GITHUB_ORG- This is a string parameter. Enter the organization name in GitHub.
• ENDOR_LABS_NAMESPACE- This is a string parameter. The namespace of your organization tenant in Endor Labs.

Optional Parameters for GitHub cloud

• ENDOR_LABS_API- This is a string parameter. This is only required if the tenant namespace is configured on the Endor Labs staging environment.
• ADDITIONAL_ARGS- This is a string parameter. Use this field to pass any additional parameter to the endorctl scan.
• NO_OF_THREADS- This is a string parameter. Enter the number of Jenkins agents that can be used in parallel for the endorctl scan. If you have 10 Jenkins agents configured with the given AGENT_LABEL, you can enter this value as 9, 1 agent is used for the main job. If not specified, this value defaults to 5.
• ENDORCTL_VERSION- This is a string parameter. Specify the version of the endorctl Docker container. Defaults to the latest version.
• SCAN_TYPE- This is a string parameter. Set this to git to scan commits or github to fetch info from the GitHub API. Defaults to [git, analytics].
• SCAN_SUMMARY_OUTPUT_TYPE- This is a string parameter. Use this field to set the desired output format. Supported formats: json, yaml’, table, summary. Defaults to table.
• LOG_LEVEL- This is a string parameter. Use this field to set the log level of the application. Defaults to info.
• LOG_VERBOSE- This is a string parameter. Use this field to make the log verbose.
• LANGUAGES- This is a string parameter. Use this field to set programming languages to scan. Supported languages: c#,go, java, javascript, php, python, ruby, rust, scala, typescript. Defaults to all supported languages.
• ADDITIONAL_ARGS- This is a string parameter. Use this field to pass any additional parameters to the endorctl scan.

Configure GitHub enterprise server credentials

Configure the following GitHub enterprise server parameters in the Jenkins pipeline script.

Required Parameters for GitHub enterprise server

• AGENT_LABEL - This is a string parameter. Enter the label used to identify the Jenkins agents. The Jenkins job will run on the agents that have this label.
• GITHUB_ORG - This is a string parameter. Enter the organization name in GitHub.
• ENDOR_LABS_NAMESPACE - This is a string parameter. The namespace of your organization tenant in Endor Labs.
• GITHUB_API_URL - This is a string parameter. Enter the API URL of the GitHub enterprise server. This is normally in the form of <FQDN of GitHub Enterprise Server>/api/v3. For example, https://ghe.endorlabs.in/api/v3.

Optional Parameters for GitHub enterprise server

• ENDOR_LABS_API - This is a string parameter. This is only required if the tenant namespace is configured on the Endor Labs staging environment.

• GITHUB_DISABLE_SSL_VERIFY - This is a boolean parameter. This should be used when you want to skip SSL Verification while cloning the repository.

• GITHUB_CA_CERT - This is a multi-line string parameter. This should be used to provide the content of the CA Certificate (PEM format) of the SSL Certificate used on the GitHub Enterprise Server.

• PROJECT_LIST - This is a multi-line string parameter. This should be used to provide a list of projects to scan.

• SCAN_TYPE - This is a string parameter. Set this to git to scan commits or github to fetch info from the GitHub API. Defaults to [git, analytics].

• SCAN_SUMMARY_OUTPUT_TYPE - This is a string parameter. Use this field to set the desired output format. Supported formats: json, yaml*, table, summary. Defaults to table.

• LOG_LEVEL - This is a string parameter. Use this field to set the log level of the application. Defaults to info.

• LOG_VERBOSE - This is a string parameter. Use this field to generate verbose logs.

• LANGUAGES - This is a string parameter. Use this field to set programming languages to scan. Supported languages: c#, go, java, javascript, php, python, ruby, rust, scala, typescript. Defaults to all supported languages.

• ADDITIONAL_ARGS - This is a string parameter. Use this field to pass any additional parameters to the endorctl scan.

• PROJECT_LIST - This is a multi-line string parameter. List of projects to scan. Even though all projects are synchronized, scans run only on the provided projects.

• SCAN_PROJECTS_BY_LAST_COMMIT - This is a string parameter. This parameter is used to filter projects based on the date of the last commit. Enter a number (integer) value for this parameter. The value of 0 means that projects won’t be filtered based on last commit date. Any positive integer is used to calculate the duration in which a commit will add the project for further scanning. If a project did not have a commit in that interval, it will be skipped. If a proper SSL Certificate (a certificate issued by a well-known CA) is not used for GitHub Enterprise, the sync-org command fails and Endor Labs cannot fetch the projects or repositories to scan from the GitHub enterprise server. You can use this field to provide the list of projects or repositories to scan one per line. For example:

          https://github-test.endorlabs.in/pse/vuln_rust_callgraph.git
          https://github-test.endorlabs.in/pse/vulnerable-golang.git
          https://github-test.endorlabs.in/pse/java-javascript-vulnerable-repo.git
          https://github-test.endorlabs.in/pse/multi-lang-repo.git
  

• EXCLUDE_PROJECTS - This is a multi-line string parameter. Use this parameter to list projects or repositories to exclude from the scan.

• NO_OF_THREADS - This is a string parameter. Enter the number of Jenkins agents that can be used in parallel for the endorctl scan. If you have 10 Jenkins agents configured with the given AGENT_LABEL, you can enter this value as 9. If not specified, this value defaults to 5.

Configure the Jenkins job

Use the following procedure to configure the Jenkins pipeline and scan the repositories in your organization.

1. Sign in to Jenkins
2. Configure an Endor Labs API Key and GitHub credentials correctly for your environment.
3. Click + New Item, to create a new Jenkins job.
4. Enter the name of the new pipeline
5. Select Pipeline and click OK.
6. Select This project is parameterised and add the parameters based on your requirements.
7. From the Pipeline section, for Definition, select Pipeline script from SCM
8. For SCM select Git
9. For the Repository URL, enter either git@github.com:endorlabs/jenkins-org-scan.git or https://github.com/endorlabs/jenkins-org-scan.git.
10. For Credentials, enter the credentials required for cloning the repository entered in the previous step.
11. In Branches to build, enter */main.
12. For Script Path, enter github-org-scan-docker.groovy.
13. Select Lightweight checkout.
14. Click Save.

The Jenkins pipeline is highly customizable and adaptable to various GitHub environments and scanning requirements. It streamlines the process of running endorctl scans on your repositories efficiently.