Scan projects

• 1: Scan for open source risks
• 1.1: Java
• 1.2: Kotlin (Beta)
• 1.3: Go
• 1.4: Python
• 1.5: JavaScript/TypeScript
• 1.6: Ruby
• 1.7: .NET
• 1.8: Bazel
• 1.9: Swift/Objective-C (Beta)
• 1.10: Scala
• 1.11: PHP
• 1.12: Rust
• 2: Scanning strategies
• 3: Scoping scans
• 4: Working with monorepos
• 5: SCM Posture Management
• 6: Perform scans in CI pipelines
• 6.1: Set up keyless authentication
• 6.2: Scanning with GitHub Actions
• 6.3: Scanning in GitLab Pipelines
• 6.4: Scanning with CircleCI
• 6.5: Scanning with Jenkins
• 7: Scan artifacts and binaries (Beta)
• 8: Scan from your IDE
• 9: Rescan projects

1 - Scan for open source risks

Endor Labs supports the following major capabilities to help teams reduce the risk and expense of software dependencies across their lifecycle:

• SCA - Software composition analysis is the identification of the bill of materials for first-party software packages and the mapping of vulnerabilities to these software component versions. SCA helps teams to maintain compliance and get visibility into the risks of their software inventory.
• Endor Scores - Endor Labs provides a holistic risk score that includes the security, quality, popularity and activity of a package. Risk scores help in identifying leading indicators of risk in addition to if a software component is outdated, or unmaintained. Risk analysis helps teams to go beyond vulnerabilities and approach the risk of their software holistically.
• Reachability Analysis - Reachability analysis is Endor Labs’ capability to perform static analysis on your software packages to give context to how each vulnerability may be reached in the context of your code. This includes mapping vulnerabilities back to vulnerable functions so that deep static analysis can target vulnerabilities with higher levels of granularity as well as the identification of unused software dependencies.

The resource requirements, both minimum and recommended, for build runners or workers executing scans using endorctl are listed here.

Note: Large applications may require additional resources to complete or enhance the scan performance.

Minimum Resources

Recommended Resources

Supported languages

The following table shows Endor Labs language coverage:

Complete support matrix

The following comprehensive matrix lists the supported languages, build tools, manifest files, and supported requirements.

See the detailed procedure for all supported languages:

1.1 - Java

Java is a high-level, object-oriented programming language widely used by developers. Endor Labs supports scanning and monitoring of Java projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Java projects

To successfully scan your Java applications:

1. Install software prerequisites
2. Build your Java projects
3. Run a scan
4. Understand the scan process
5. Troubleshoot errors

Install software prerequisites

• Endor Labs requires the following pre-requisite software to be installed to successfully perform a scan:
  • Maven Package Manager: Any version
  • Gradle build system: Version 6.0.0 and higher. To support lower versions of Gradle, see Scan projects on Gradle versions between 4.7 and 6.0.0.
  • Java runtime environment: JDK versions between 11 and 19
  • For JDK 8, see Scan the projects on JDK version 8
• When performing deep scans make sure that you have the following minimum system specification requirements:
  • 4-core processor with 16 GB RAM for small projects
  • 8-core processor with 32 GB RAM for mid-size projects
  • 16-core processor with 64 GB RAM for large projects

Build Java projects

You must build your Java projects before running a scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

Use Gradle

To analyze your software built with Gradle, Endor Labs requires that the software be able to be successfully built. To perform a quick scan, dependencies must be located in the local package manager cache. The standard $GRADLE_USER_HOME/caches or /User/«username»/.gradle/caches must exist and contain successfully downloaded dependencies. To perform a deep scan the target artifact must be generated on the filesystem as well.

To build your project with Gradle, use the following procedure:

1. If you would like to run a scan against a custom configuration, specify the Gradle configuration by setting an environment variable.

      export endorGradleJavaConfiguration="<configuration>"
   

   The default configuration is runtimeClasspath if none is provided.

2. Ensure that you can resolve the dependencies for your project without errors by running the following command:

   For Gradle wrapper:

      ./gradlew dependencies
   

   For Gradle:

      gradle dependencies
   

3. Run ./gradlew assemble or gradle assemble to resolve dependencies and to create an artifact that may used for deep analysis.

Override sub project level configuration

In a multi-build project, if you set the environment variable endorGradleJavaConfiguration=[GlobalConfiguration], the specified configuration is used for dependency resolution across all projects and sub-projects in the hierarchy below.

\--- Project ':samples'
     +--- Project ':samples:compare'
     +--- Project ':samples:crawler'
     +--- Project ':samples:guide'
     +--- Project ':samples:simple-client'
     +--- Project ':samples:slack'
     +--- Project ':samples:static-server'
     +--- Project ':samples:tlssurvey'
     \--- Project ':samples:unixdomainsockets'

To override the configuration only for the :samples:crawler and :samples:guide sub-projects, follow these steps:

1. Navigate to the root workspace, where you execute endorctl scan, and run ./gradlew projects to list all projects and their names.

2. Run the following command at the root of the workspace:

   echo ":samples:crawler=testRuntimeClasspath,:samples:guide=macroBenchMarkClasspath" >> .endorproperties
   

   This creates a new file named .endorproperties in your root directory. This enables different configurations for the specified sub-projects in the file.

3. Run endorctl scan as usual.

At this point, all other projects will adhere to the GlobalConfiguration. However, the :samples:crawler sub-project will use the testRuntimeClasspath configuration, and the :samples:guide sub-project will use the macroBenchMarkClasspath configuration.

Use Maven

To analyze your software built with Maven, Endor Labs requires that the software be able to be successfully built. To perform a quick scan, dependencies must be located in the local package manager cache. The standard .m2 cache must exist and contain successfully downloaded dependencies. To perform a deep scan the target artifact must be generated on the filesystem as well.

To build your project with Maven, use the following procedure:

1. Ensure that you can resolve the dependencies for your project without error by running the following command.

mvn dependency:tree

2. Run mvn install and make sure the build is successful.

mvn install -DskipTests

3. If you have multiple Java modules not referenced in the root pom.xml file, make sure to run mvn install separately in all the directories.

Configure Maven private registries

If you have a private registry and internal dependencies on other projects, you must configure private registries. In addition to scanning public Java projects and repositories, Endor Labs provides the support to integrate with private registries.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Maven and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan will not perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When deep analysis is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but will not impact subsequent scans.

Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --disable-private-package-analysis

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Scan the projects on JDK version 8

Endor Labs supports JDK versions between 11-19, however, you can scan projects on JDK 8 using the following procedure:

1. Build your Java project on JDK 8.
2. After building, switch your Java home to JDK 11 or higher versions.

export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home

3. Run a scan

Scan projects on Gradle versions between 4.7 and 6.0.0

To scan Java projects on Gradle versions between 4.7 and 6.0.0, make sure to

1. Check the version of your project using:

./gradlew --version

2. The project must have a Gradle wrapper. You can generate the Gradle wrapper using:

--gradle-version <your required version>.

Endor Labs prioritizes Gradle wrapper over Gradle and it is a recommended best practice to use Gradle Wrapper.

3. Before executing the endorctl scan, ensure the project can be built in your required version.

Execute ./gradlew assemble.

4. Use --bypass-host-check during endorctl scan to execute scans on projects that have gradle versions lower than 6.0.0.

Understand the scan process

Endor Labs analyzes your Java code and dependencies to detect known security issues, including open-source vulnerabilities and generates call graphs.

How Endor Labs resolves dependencies for package versions

Endor Labs resolves the dependencies for Java packages based on the following factors:

• For packages built using Maven, it leverages the Maven cache in the .m2 directory of your file system to resolve the package’s dependencies and mirrors Maven’s build process for the most accurate results.
• For Maven, Endor Labs respects the configuration settings present in the settings.xml file. If this file is included in your repository, you need not provide any additional configuration.
• For packages built using Gradle, it uses Gradle and Gradle wrapper files to build packages and resolve dependencies.
• Endor Labs supports EAR, JAR, RAR, and WAR files.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis on the Java code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known limitations

• If a package can not be successfully built in the source control repository, static analysis will fail.
• Spring dependencies are analyzed based on spring public entry points to reduce the impact of Inversion of Control (IOC) frameworks. Dependencies and functions are identified as reachable and unreachable in the context of a spring version and its entry points.
• Annotation processing is limited only to the usage of the code they annotate.
• Static analysis of reflection and callbacks are not supported.
• Endor Labs requires JDK 11 to generate call graphs for Java projects. Gradle versions lacking JDK 11 support are not compatible.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:

  • Java is not installed or not present in the PATH environment variable. Install Java and try again. See Java documentation for more information.
  • The installed version of Java is lower than the required version. Install JDK versions between 11-19 and try again.
  • Java is installed but Maven or Gradle is not installed. In such cases, the dependency resolution may not be complete.

• Unresolved dependency errors: Maven is not installed properly or the system is unable to build root pom.xml. Run mvn dependency:tree in the root of the project and try again. In such cases, the dependency resolution may not be complete.

• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

• Gradle variant incompatibility message:
  Gradle will perform JVM toolchain checks for sub projects or dependencies and may raise errors indicating a Java version mismatch between dependencies declared in Gradle manifest and Java home setup.

  Example error message: Incompatible because this component declares a component for use during compile-time, compatible with Java version 21 and the consumer needed a component for use during runtime, compatible with Java version 17
  

  • To resolve this and taking advantage of Java’s backward compatibility, instruct Gradle to use the higher version of JDK detected in the error message. For example, for the message above, specify org.gradle.java.home=<path of java> in .gradle/gradle.properties. The path needs to be to the root of the directory bin/java. For example, if your Java is at /Users/Downloads/jdk-21/Contents/Home/bin/java, specify org.gradle.java.home=/Users/Downloads/jdk-21/Contents/Home.
  • If you are scanning a purely Java 8 Gradle project and if you encounter the above error, set org.gradle.java.home to point to Java 8 home, before you execute the endorctl scan.
  • A general guideline for determining which Java version to use, is to match the Java version specified in .gradle/gradle.properties with the one used for building your Gradle project.

• Call graph errors:

  • The project can not be built because a dependency cannot be located in the repository.
  • Sometimes, the project is not built, if a Java version discrepancy exists between the required repository version and the version on the system running the scan. For example, the Java required version is 1.8 but the system has 12 installed. Install the required version and try again.

• If you have a private registry and internal dependencies on other projects, you must configure the credentials of the registry. See Configure Maven private registries.

• If you have a large repository, you may need to increase the size of the JVM through a scan parameter. Export the ENDOR_SCAN_JVM_PARAMETERS environment variable with additional JVM parameters before performing the scan as shown below:

  export ENDOR_SCAN_JVM_PARAMETERS="-Xmx32G"
  

• If you use a remote repository configured to authenticate with a client-side certificate, you must add the certificate through an endorctl parameter. Export the ENDOR_SCAN_JVM_PARAMETERS parameter before performing a scan. See Maven documentation for details.

  export ENDOR_SCAN_JVM_PARAMETERS="-Xmx16G,-Djavax.net.ssl.keyStorePassword=changeit,
  -Djavax.net.ssl.keyStoreType=pkcs12,
  -Djavax.net.ssl.keyStore=/Users/myuser/Documents/nexustls/client-cert1.p12"
  

1.2 - Kotlin (Beta)

Kotlin is a statically-typed programming language that runs on the Java Virtual Machine (JVM), known for its concise syntax, null safety, and seamless integration with Java. Endor Labs supports scanning and monitoring of Kotlin projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Kotlin projects

To successfully scan your Kotlin projects:

1. Prerequisites
2. Build your Kotlin projects
3. Run a scan
4. Understand the scan process

Prerequisites

• Install software: To ensure a successful scan using Endor Labs, it is essential to have the following prerequisite software installed:

  • Maven Package Manager: Version 3.6.1 and higher
  • Gradle Build System: Version 6.0.0 and higher
  • Java Runtime Environment: JDK versions between 11 and 19
  • Kotlin version 1.4 through 1.9.x

  For users on JDK 8, specific instructions can be found in the section Scan the Projects on JDK Version 8.

• Minimum system specifications for deep scans: When performing deep scans, ensure that you have the following minimum system specifications:

  • For small projects: 4-core processor with 16 GB RAM
  • For mid-size projects: 8-core processor with 32 GB RAM
  • For large projects: 16-core processor with 64 GB RAM

Build Kotlin projects

Before initiating a scan with Endor Labs, ensure that your Kotlin projects are built successfully prior to initiating the scan. Additionally, ensure that the packages are downloaded into local package caches and build artifacts are present in their standard locations. Follow the guidelines to use Gradle and Maven:

Use Gradle

To analyze your software built with Gradle, Endor Labs requires:

• The software must be successfully built with Gradle.
• For quick scans, dependencies must be located in the local package manager cache. The standard $GRADLE_USER_HOME/caches or /User/<<username>>/.gradle/caches cache must exist.
• For deep scans, the target artifact must be generated on the filesystem.

To build your project with Gradle, run the following commands:

1. Specify the Gradle configuration by setting an environment variable.

   export endorGradleKotlinConfiguration="compileClasspath"
   

   If you need to override the default configuration, use the command:

   export endorGradleKotlinConfiguration="<configuration>"
   

   The default configuration is runtimeClasspath if none is provided.

   For Android projects, you can set the configuration using:

   export endorGradleAndroidConfiguration="<configuration>"
   

   The default configuration for an Android application or library uses the same structure established by Android Studio. For applications, all possible combinations of application variants are examined. For libraries, all possible combinations of library variants are examined. The first variant from the alphabetically sorted list is then selected for runtimeClasspath.

   If neither approach provides a value, the default configuration is set to releaseRuntimeClasspath.

2. Confirm an error-free dependency resolution for your project.

   gradle dependencies
   

   or, with a Gradle wrapper.

   ./gradlew dependencies
   

3. Generate the artifact for deep analysis.

   gradle assemble
   

   or, with a Gradle wrapper.

   ./gradlew assemble
   

Override sub project level configuration

In a multi-build project, if you set the environment variable endorGradleKotlinConfiguration=[GlobalConfiguration] and/or endorGradleAndroidConfiguration=[GlobalConfiguration], the specified configuration is used for dependency resolution across all projects and sub-projects in the hierarchy below.

\--- Project ':samples'
     +--- Project ':samples:compare'
     +--- Project ':samples:crawler'
     +--- Project ':samples:guide'
     +--- Project ':samples:simple-client'
     +--- Project ':samples:slack'
     +--- Project ':samples:static-server'
     +--- Project ':samples:tlssurvey'
     \--- Project ':samples:unixdomainsockets'

To override the configuration only for the :samples:crawler and :samples:guide sub-projects, follow these steps:

1. Navigate to the root workspace, where you execute endorctl scan, and run ./gradlew projects to list all projects and their names.

2. Run the following command at the root of the workspace:

   echo ":samples:crawler=testRuntimeClasspath,:samples:guide=macroBenchMarkClasspath" >> .endorproperties
   

   This creates a new file named .endorproperties in your root directory. This enables different configurations for the specified sub-projects in the file.

3. Run endorctl scan as usual.

At this point, all other projects will adhere to the GlobalConfiguration. However, the :samples:crawler sub-project will use the testRuntimeClasspath configuration, and the :samples:guide sub-project will use the macroBenchMarkClasspath configuration.

Use Maven

To analyze your software built with Maven, Endor Labs requires:

• The software must be successfully built with Maven.
• For quick scans, dependencies must be located in the local package manager cache. The standard .m2 cache must exist.
• For deep scans, the target artifact must be generated on the filesystem.

To build your project with Maven, run the following commands:

1. Confirm an error-free dependency resolution for your project.

   mvn dependency:tree
   

2. Run mvn install and ensure the build is successful.

   info

   If you want to skip the execution of tests during the build, you can use -DskipTests to quickly build and install your projects.

   mvn install -DskipTests
   

3. If you have multiple Kotlin modules not referenced in the root pom.xml file, ensure to run mvn install separately in each directory.

Configure Maven private registries

If you have a private registry with internal dependencies on other projects, you must configure private registries. Endor Labs not only supports scanning public Kotlin projects and repositories but also allows integration with private registries.

Follow these steps to configure Maven private registries:

1. Sign in to Endor Labs and navigate to Integrations under Manage from the left sidebar.
2. In the Package Managers > Maven section, click Manage.
3. On the Maven integrations page, click Add Package Manager.
4. Specify a package manager URL.
5. (Optional) To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and specify the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

To scan your repositories with Endor Labs, you can use the following options after building your Kotlin projects.

Option 1 - Quick scan

To quickly gain insight into your software composition, initiate a quick scan using the following command:

endorctl scan --quick-scan

This scan offers a quick overview without performing reachability analysis, helping you prioritize vulnerabilities.

Save local results

To scan a Git project repository from the root directory and save the results locally in the results.json file, use the following command:

endorctl scan --quick-scan -o json | tee /path/to/results.json

This generates comprehensive results and analysis information, accessible from the Endor Labs user interface.

Access results

To access and review detailed results, sign in to the Endor Labs user interface. Navigate to Projects on the left sidebar, and locate your project for a thorough examination of the scan results.

Option 2 - Deep scan

To perform dependency resolution and reachability analysis, use deep scan with Endor Labs. This option is recommended only after successfully completion of quick scan.

endorctl scan

Save local results

To save the local results to a results.json file, use the following flag.

endorctl scan -o json | tee /path/to/results.json

This generates comprehensive results and analysis information, accessible from the Endor Labs user interface.

Analyze private packages

During deep analysis, Endor Labs thoroughly analyzes all private software dependencies that have not been previously scanned. While this initial operation may slow down scans, subsequent scans remain unaffected.

If your organization does not own specific software parts and related findings are non-actionable, you can choose to disable this analysis using the disable-private-package-analysis flag. Disabling private package analysis enhances scan performance but may result in a loss of insights into how applications interact with first-party libraries.

To disable private package analysis, use the following command flag:

endorctl scan --disable-private-package-analysis

Access results

To access and review detailed results, sign in to the Endor Labs user interface. Navigate to Projects on the left sidebar, and locate your project for a thorough examination of the scan results.

Scan the projects on JDK version 8

While Endor Labs primarily supports JDK versions between 11-19, you can still scan projects on JDK 8 by following these steps:

1. Build your Java project on JDK 8.

2. After successful build, switch your Java home to JDK 11 or higher versions.

   export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home
   

3. Run a scan.

Understand the scan process

Endor Labs analyzes your Kotlin code and dependencies to identify known security issues, including open-source vulnerabilities.

How does Endor Labs resolve dependencies for package versions?

Endor Labs resolves Kotlin package dependencies by considering the following factors:

• For packages built with Maven, it leverages the Maven cache in the .m2 directory of your file system. This mirrors Maven’s build process for precise results.
• For packages built with Maven, it respects the configuration settings present in the settings.xml file. If the file is included in your repository, any additional configuration is not necessary.
• For packages built with Gradle, it leverages Gradle and Gradle wrapper files to build and resolve dependencies.
• Endor Labs supports AAR, EAR, JAR, RAR, and WAR files.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis on the code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known limitations

• If a package can not be successfully built in the source control repository, static analysis will fail.
• Spring dependencies are analyzed based on spring public entry points to reduce the impact of Inversion of Control (IOC) frameworks. Dependencies and functions are identified as reachable and unreachable in the context of a spring version and its entry points.
• Annotation processing is limited only to the usage of the code they annotate.
• Static analysis of reflection and callbacks are not supported.
• If Endor Labs fails to resolve dependencies using default Kotlin configurations, the Kotlin configuration must be specified.
• Static analysis for Kotlin projects using Gradle is only supported when the Kotlin plugin for Gradle versions 1.5.30 to 1.9.x

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:
  • Java is not installed or not present in the PATH environment variable. Install Java and try again. See Java documentation for more information.
  • For android applications, $ANDROID_HOME must be specified as an environment variable.
  • The installed version of Java is lower than the required version. Install JDK versions between 11-19 and try again.
  • Java is installed but Maven or Gradle is not installed. In such cases, the dependency resolution may not be complete.
• Unresolved dependency errors: Maven is not installed properly or the system is unable to build root pom.xml. Run mvn dependency:tree in the root of the project and try again. In such cases, the dependency resolution may not be complete.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.
• Call graph errors:
  • If the project is not compiled, call graphs are not generated. Run gradlew compileKotlin or gradlew compileReleaseKotlin for android based projects before running the scan.
  • Sometimes, the project is not compiled, if a Kotlin version discrepancy exists between the required repository version and the version on the system running the scan. For example, the Kotlin required version is 1.4 but the system has lower version installed. Install the required version and try again.
• If you have a private registry and internal dependencies on other projects, you must configure the credentials of the registry. See Configure Maven private registries.
• If you use a remote repository configured to authenticate with a client-side certificate, you must add the certificate through an endorctl parameter. Export the ENDOR_SCAN_JVM_PARAMETERS parameter before performing a scan. See Maven documentation for details.

  export ENDOR_SCAN_JVM_PARAMETERS="-Xmx16G,-Djavax.net.ssl.keyStorePassword=changeit,
  -Djavax.net.ssl.keyStoreType=pkcs12,
  -Djavax.net.ssl.keyStore=/Users/myuser/Documents/nexustls/client-cert1.p12"
  

1.3 - Go

Go or Golang is a software development programming language widely used by developers. Endor Labs supports scanning and monitoring of Go projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Go projects

To successfully scan your repositories for Go:

• Scan Go projects
  • Install software prerequisites
  • Build Go projects
  • Run a scan
    • Option 1 - Quick scan
    • Option 2 - Deep scan
  • Understand the scan process
    • Known Limitations
  • Troubleshoot errors

Install software prerequisites

Make sure that you have Go 1.12 or higher versions.

Build Go projects

You must build your Go projects before running the scan. Additionally, ensure that the packages are downloaded into the local package caches and that go.mod file well formed and is available in the standard location.

To ensure that your go.mod file is well formed, run the following command:

go mod tidy

go get ./

This removes any dependencies that are not required by your project and ensures to resolve the dependencies without errors.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan will not perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs resolves your Golang-based dependencies by leveraging built-in Go commands to replicate the way a package manager would install your dependencies.

To discover package names for Go packages Endor Labs uses the command:

go list -e -mod readonly -json -m

To analyze the dependency graph of your package Endor Labs uses the command:

go list -e -deps -json -mod readonly all

To assess external dependencies, specifically third-party packages or libraries that your Go project relies on, Endor Labs uses the command:

go list -e -deps -json -mod vendor all

These commands allow us to assess packages’ unresolved dependencies, analyze the dependency tree, and resolve dependencies for your Go projects.

Known Limitations

Endor Labs creates go.mod files for you when projects do not have a go.mod file. This can lead to inconsistencies with the actual package created over time and across versions of the dependencies.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:

  • Go is not installed or not present in the PATH environment variable. Install Go and try again.
  • The installed version of Go is lower than 1.12. Install Go version 1.12 or higher and try again.

• Resolved dependency errors:

  • A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.
  • If the go.mod file is not well-formed then dependency resolution may return errors. Run go mod tidy and try again.

• Call graph errors:

  These errors often mean the project will not build. Please ensure any generated code is in place and verify that go build ./... runs successfully.

1.4 - Python

Python is a high-level, interpreted programming language widely used by developers. Endor Labs supports the scanning and monitoring of Python projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Python projects

To successfully scan your repositories for Python:

1. Install software prerequisites
2. Build Python projects
3. Run a scan
4. Understand the scan process
5. Troubleshoot errors

Install software prerequisites

• Make sure the following pre-requisites are installed:
  • Python 3.6 and higher versions. See the Python documentation on how to install Python.
  • The package manager Pip or Poetry used by your projects to build your software packages. For Python 3.12, Setuptools must be installed along with Pip and Poetry.
  • Any build, code generation, or other dependencies that will be required to install your project’s packages.
  • The project is organized as one or more packages using setup.py, setup.cfg, pyproject.toml, or requirements.txt package manifest files.
• Make sure that you have the following minimum system specification requirements:

  Repository size	Processor	Memory
  Small	8-core	32 GB
  Medium	16-core	64 GB
  Large	32-core	128 GB

Build Python projects

You must create a virtual environment and build your Python projects before running the endorctl scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Configure any private repositories
   • If you use dependencies from a PyPI-compatible repository other than pypi.org, configure it in the Integrations section of the Endor Labs web application. See Configure Python private repositories for more details.
2. Clone the repository and create a virtual environment inside it
   1. Clone the repository using git clone or an equivalent workflow.
   2. Enter the working copy root directory that’s created.
   3. Create a virtual environment: Use python3 -m venv venv. Set up the virtual environment in the root folder that you want to scan and name it venv or .venv.
   4. Activate the virtual environment using source venv/bin/activate or your project’s equivalent to activate the virtual environment.
   5. Install your project’s dependencies using venv/bin/python3 -m pip install or venv/bin/poetry install.
   6. If the virtual environment is created outside of the project, use one of the ways defined in Virtual environment support to specify the path of the Python virtual environment to Endor Labs.

Virtual environment support

Use one of the following ways to specify the virtual environment details of your Python projects for both quick and deep scans:

• Set up the virtual environment in the root folder that you want to scan and name it venv or .venv, it is automatically picked up by the Endor Labs application.

export PYTHONPATH=/usr/tmp/venv:/usr/tmp/another-venv

• Set the environment variable ENDOR_SCAN_PYTHON_VIRTUAL_ENV to the path of the virtual environment of your Python project.

export ENDOR_SCAN_PYTHON_VIRTUAL_ENV=/usr/tmp/venv

• Set the environment variable ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES to true to indicate that a virtual environment is not present and Endor Labs can use the system-wide Python installation packages and modules.

export ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES=true

• If the virtual environment is not set up, Endor Labs attempts to set it up with all the code dependencies, however, it is recommended that all dependencies be installed in a virtual environment for the most accurate results.

• If you are using custom scripts without manifest files to assemble your dependencies, make sure to set up the virtual environment and install the dependencies.

Configure private Python repositories

In addition to scanning public Python projects, Endor Labs provides support to fetch and scan private Python repositories. Endor Labs will fetch the resources from the authenticated endpoints and perform the scan and you can view the dependencies and findings.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select PyPI and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan will not perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully. The deep scan performs the following operations for the Python projects.

• Discovers explicitly declared dependencies,
• Discovers project dependant OSS packages present in the venv/global and scope/python.
• Performs reachability analysis and generates call graphs.
• Detects dependencies used in source code but not declared in the package’s manifest files called phantom dependencies.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but will not impact subsequent scans.

Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs uses the following two methods to analyze your Python code.

• Dependency resolution using manifest files
• Dependency resolution using static analysis

Endor Labs uses the results from both these methods to perform superior dependency resolution, identify security issues, detect open-source vulnerabilities, and generate call graphs.

Dependency resolution using manifest files

In this method, Endor Labs analyzes the manifest files present in a project to detect and resolve dependencies. The manifest files are analyzed in the following priority.

On initialization of a scan, Endor Labs identifies the package manager by inspecting files such as the pyproject.toml, poetry.lock, setup.py, and requirements.txt. When the files, pyproject.toml, or poetry.lock files are discovered, Endor Labs will use the poetry package manager to build the project, otherwise, it will use Pip3.

Example

This is an example that demonstrates scanning a Python repository from GitHub on your local system using the endorctl scan. Here we are assuming that you are running the scan on a Linux or Mac operating system environment and that you have the following Endor Labs API key and secret stored in the environment variables. See endorctl flags and variables.

• ENDOR_API_CREDENTIALS_KEY set to the API key
• ENDOR_API_CREDENTIALS_SECRET set to the API secret
• ENDOR_NAMESPACE set to your namespace (you can find this when logged into Endor Labs by looking at your URL: https://app.endorlabs.com/t/NAMESPACE/...; it is typically a form of your organization’s name)

git clone https://github.com/HybirdCorp/creme_crm.git
cd creme_crm
python3 -m venv venv
source venv/bin/activate
venv/bin/python3 -m pip install
endorctl scan

The scan for this repository is expected to be completed in approximately 10 minutes. You can now visit app.endorlabs.com, navigate to Projects, and choose the helloflas/flask-examples project to see your scan results.

Dependency resolution using static analysis

All Python projects do not always include manifest files. A project can be a series of install statements that are assembled by custom scripts. Even when manifest files are present, the dependency information and version declared in the manifest file may be drastically different from what is used in a project.

To solve this problem, Endor Labs has developed a unique method for dependency resolution by performing a static analysis on the code, giving you complete visibility of what is used in your code.

• Endor Labs enumerates all Python packages and recognizes the import statements within the project. An import statement is a Python code statement that is used to bring external modules or libraries into your Python script.
• It performs a static analysis of the code to match the import statements with the pre-installed packages and recursively traverses all files to create a dependency tree with the actual versions that are installed in the virtual environment.
• It detects the dependencies at the system level to identify which ones are resolved and retrieves the precise name and version information from the library currently in use.
• It gives you accurate visibility into your project components and helps you understand how the components depend on one another.

Through this approach, Endor Labs conducts comprehensive dependency management, assesses reachability, and generates integrated call graphs.

Known Limitations

• Endor Labs specifically looks for the requirements.txt file for a Python project using Pip. If you use a different file name, it will not be automatically discovered.
• Python versions older than 3.7 are not supported but may work as expected.
• If a virtual environment is not provided, Python version constraints are not assumed based on the runtime environment of CI. Dependencies are shown for all possible versions of Python at runtime. If a virtual environment is provided Endor Labs respects what is installed in the virtual environment.
• Symbolic links into manifest files may result in the same package being duplicated in the project.
• If a dependency is not available in the PyPI repository or in a configured private package repository, Endor Labs will be unable to build the software and scans may fail without first building the package in the local environment successfully.

Call Graph Limitations

• Function calls using dispatch table calls might not be included in the call graph.
• Function calls using unresolved variables might not be included in the call graph.
• Dynamically modified or extended function calls used to declare methods or attributes at run time might not be included in the call graph.
• Functions called indirectly through a function pointer and not by their direct name, might not be included in the call graph.
• Type stubs that provide hints or type annotations for functions, methods, and variables in your Python modules or libraries have to be installed manually before performing a scan.
• If your project has a pyproject.toml file that includes tools.pyright section, it overrides Endor Labs settings for pyright and may result in incorrect call graph results. You will need to remove the tools.pyright section from the pyproject.toml file.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Virtual environment errors: You can identify the errors that may occur during virtual environment installation by looking for the following message in the error logs; failed to create virtual environment or failed to install dependencies.
• Missing environment dependency: If your code depends on packages such as psycopg2, environment dependencies such as Postgresql are also required. The endorctl scan may fail if the environment where it is running does not have Postgresql installed.
• Incompatible Python version: The default Python version in the environment where the endorctl scan is running is incompatible with one or more of the dependencies that are needed by the code.
• Incompatible architecture: One or more dependencies are not compatible with the operating system architecture of the local system on which you are running the endorctl scan. For example, projects with dependency on pyobjc can be run on Mac-based systems, but not Linux systems. A few Python libraries are incompatible with x32 architectures and can only be run on x64 architectures.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.
• Call graph errors: These errors come if Pip or Poetry are unable to build the project because a required dependency cannot be located.

1.5 - JavaScript/TypeScript

JavaScript is a high-level, interpreted programming language primarily used for creating interactive and dynamic web content widely used by developers. Endor Labs supports the scanning and monitoring of JavaScript projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan JavaScript projects

To successfully scan your JavaScript or TypScript applications:

• Install software prerequisites
• Build JavaScript projects
• Run a scan
• Understand the scan process
  • Known Limitations
• Troubleshoot errors

Install software prerequisites

• Endor Labs requires the following pre-requisite software to be installed to successfully perform a scan:

  • Yarn: Any version
  • NPM: 6.14.18 or higher versions
  • PNPM: 3.0.0 or higher versions

• Make sure that you have the following minimum system specification requirements:

  Processor	Memory
  4-core	16 GB

Build JavaScript projects

You can choose to build your JavaScript projects before running a scan. This will ensure that either a package-lock.json, yarn.lock, or pnpm-lock.yaml file is created enhancing the scan speed.

Ensure your repo has package.json and run the following command making sure it builds the project successfully.

For NPM:

npm install

For Yarn:

yarn install

For PNPM:

pnpm install

If the project is not built, endorctl builds the project during the scan and generate either package-lock.json, yarn.lock, or pnpm-lock.yaml file. Make sure that either NPM, Yarn, or PNPM is installed on your system. If your repository includes a lock file, endorctl uses the existing file for dependency resolution and does not create it again.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

To detect dependencies when using PNPM:

If you are using PNPM, set the environment variable ENDOR_PNPM_ENABLED to true and then run the scan.

export ENDOR_PNPM_ENABLED=true

endorctl scan

To detect phantom dependencies:

To detect dependencies used in source code but not declared in the package’s manifest files, set the flag --call-graph-languages with javascript,typescript.

endorctl scan --call-graph-languages=javascript,typescript

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Dependency analysis tools analyze the lock file of an npm, yarn, or pnpm based package and attempt to resolve dependencies. To resolve dependencies from private repositories, the settings of the .npmrc file in the repository is considered.

Endor Labs surpasses mere manifest file analysis by expertly resolving JavaScript dependencies and identifies:

• Dependencies listed in the manifest file but not used by the application
• Dependencies used by the application but not listed in the manifest file
• Dependencies listed in the manifest as transitive but used directly by the application
• Dependencies categorized as test in the manifest, but used directly by the application

Developers can eliminate the false positives, false negatives, and easily identify test dependencies with this analysis. The dependencies used in source code but not declared in the package’s manifest files are tagged as Phantom.

Endor Labs also supports NPM, Yarn, and PNPM workspaces out-of-the-box. If your JavaScript frameworks and packages use workspaces, Endor Labs will automatically take the dependencies from the workspace to ensure that the package successfully builds.

Scan speed is enhanced if the lock file exists in the repository. endorctl does not perform a build and uses the existing files in the repository for analysis.

Known Limitations

• Endor Labs doesn’t currently support local package references
• If a dependency can not be resolved in the lock file, building that specific package may be unsuccessful. This package may have been removed from npm or the .npmrc file is not properly configured. Other packages in the workspace are scanned as usual.
• Call graph generation is not supported for JavaScript

Troubleshoot errors

• Unresolved dependency errors: The manifest file package.json is not buildable. Try running npm install, yarn install, or pnpm install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.6 - Ruby

Ruby is a widely used open-source programming language. Endor Labs supports scanning and monitoring of Ruby projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Ruby projects

To successfully scan your Ruby applications:

• Install software prerequisites
• Build Ruby projects
  • Configure Ruby private Gem sources
• Run a scan
• Understand the scan process
  • Known limitations
• Troubleshoot errors

Install software prerequisites

The following prerequisites must be fulfilled:

• All applications monitored by Endor Labs must be on Ruby versions 2.6 or higher.
• A Gemfile or a *.gemspec file must be present in your Ruby project.

Build Ruby projects

You can choose to build your Ruby projects before running a scan. This will ensure that gemfile.lock is created.

Ensure your repo has Gemfile and run the following command making sure it builds the project successfully.

bundler install

If the project is not built, endorctl will build the project during the scan and generate Gemfile.lock. If the repository includes a Gemfile.lock, endorctl uses this file for dependency resolution and does not create it again.

Configure Ruby private Gem sources

If you have a private registry and internal dependencies on other projects, you must configure private registries. Endor Labs can fetch the gems that are hosted in a private gem repository which are not available publicly after authenticating the endpoints.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Ruby Gems and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs looks for Gemfile, *.gemspec, and Gemfile.lock files to find and monitor the dependency activity.

• A Gemfile is a configuration file used in Ruby projects to specify the required RubyGems (libraries or packages) for the project’s dependencies.
• A *.gemspec file is a RubyGems specification file used to define the metadata and dependencies for a RubyGem.
• The Gemfile.lock file is automatically generated by Bundler and is used to lock down the specific versions of gems and their dependencies installed in your project’s environment.

If the Gemfile.lock is not present in your project, Endor Labs generates this file and stores it in a temp directory. The file is deleted after extracting dependency information.

Endor Labs’ dependency resolution mechanism assesses multiple factors, including compatibility, stability, and availability, to determine the most suitable version for usage. The resolved dependency version is used during the build or execution of your Ruby project. By utilizing the dependency graph, you can access significant information regarding the dependencies. This includes determining whether a dependency is direct or transitive, checking its reachability, verifying source availability, and more. The dependency graph provides a visual representation that allows you to examine the graphical details of these dependencies.

Known limitations

• Call graphs are not supported for Ruby projects.
• If a dependency can not be resolved in the Gemfile, building that specific package may not be successful. This package may have been removed from the Gem package manager. Other packages in the workspace are scanned.

Troubleshoot errors

• Unresolved dependency errors: The Gemfile is not buildable. Try running bundler install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.7 - .NET

.NET is a free, cross-platform, open-source developer platform for building different types of applications. Endor Labs supports the scanning and monitoring of projects built on the .NET platform.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan .NET projects

To successfully scan your .NET projects

1. Install software prerequisites
2. Run a scan
3. Understand the scan process
4. Troubleshoot errors

Install software prerequisites

The following prerequisite must be fulfilled:

• One or more *.csproj files must be present in your repository.
• The .NET command or Nuget command must be installed and available on the host system.
• At least one .NET SDK installed on the system must be compatible with the project’s global.json file settings.

Call graph requirements

• The .NET SDK 7.0.101 or higher versions within the 7.x major version range must be installed on the host system.
• Endor Labs’ call graph support for .NET is based on Microsoft’s Common Intermediate Language (CIL). Artifacts such as .exe or .dll files must be available in the project’s standard workspace through a build and restore or a restored cache.
• When performing deep scans make sure that you have the following minimum system specification requirements:
  • 4-core processor with 16 GB RAM for small projects
  • 8-core processor with 32 GB RAM for mid-size projects
  • 16-core processor with 64 GB RAM for large projects

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan will not perform reachability analysis to help you prioritize vulnerabilities.

You must restore your .NET projects before running a quick scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Run the following commands to resolve dependencies and create the necessary files to scan your .NET project.

To ensure that the build artifacts project.assets.json file is generated and dependencies are resolved run:

dotnet restore

If you use Nuget instead run:

nuget restore

To create a packages.lock.json file if your project uses a lock file run:

dotnet restore --use-lock-file

If project.assets.json or packages.lock.json are not present and if the project is buildable, endorctl will restore the project and create a project.assets.json or a packages.lock.json file to resolve dependencies.

2. You can run a quick scan with the following commands:

endorctl scan --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

You must restore and build your .NET projects before running a deep scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Run the following commands to restore and build your project. This may vary depending on your project’s configuration.

dotnet restore
dotnet build

2. You can run a deep scan with the following commands:

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but will not impact subsequent scans.

Organizations might not own some parts of the software internally and findings are actionable by another team. These organizations can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --disable-private-package-analysis

To perform dependency resolution and reachability analysis on .NET projects using the Central Package Management feature of Nuget, you must first set the environment variable ENDOR_ENABLE_DOTNET_PROPS_RESOLUTION to true and then run the endorctl scan.

export ENDOR_ENABLE_DOTNET_PROPS_RESOLUTION=true

endorctl scan

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Configure Nuget private repositories

You can configure Endor Labs to fetch dependencies from private NuGet feeds after authenticating the endpoints.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Nuget and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Understand the scan process

A *.csproj file is an XML-based C# project file that contains information about the project, such as its source code files, references, build settings, and other configuration details. The dependencies and findings are listed individually for every *.csproj file. The scan discovers all *.csproj files and uses these files to resolve the appropriate dependency graph of your project.

(Beta) Endor Labs scans the .NET projects that are using the Central Package Management feature of Nuget for the packages declared as:

• Package references in Directory.Build.props or Directory.Packages.props files.
• Package references in any *.props file and the prop file is imported in the *.csproj file.
• Package references in *.Targets file.

Endor Labs enriches your dependency graph to help you understand if your dependencies are secure, sustainable, and trustworthy. This includes Endor Labs risk analysis and scores, if a dependency is direct or transitive, and if the source code of the dependency is publicly auditable.

Software composition analysis for .NET is performed in the following ways:

• Using Project.assests.json
• Using package.lock.json

How does dependency resolution happen using Project.assets.json

The Project.assets.json file is used in .NET projects to store metadata and information about the project’s dependencies and assets.

Endor Labs fetches resolved package versions, paths to the dependencies’ assets, such as assemblies and resources, and other related information from this file. If a project does not include a project.assets.json file, it is generated through the dotnet restore or the nuget restore command. This command uses all the configured sources to restore dependencies as well as project-specific tools that are specified in the project file.

How does dependency resolution happen using package.lock.json

The package.lock.json file is used in .NET projects to lock dependencies and their specific versions. It is a snapshot of the exact versions of packages installed in a project, including their dependencies and sub-dependencies, requested versions, resolved versions, and contentHash. The lock file provides a more dependable, uniform, and accurate representation of the dependency graph.

In Endor Labs’ dependency management, the resolution of dependencies is primarily based on package.lock.json, which takes precedence over projects.assets.json to resolve dependencies.

Endor Labs fetches the dependency information from package.lock.json and creates a comprehensive dependency graph. The vulnerabilities associated with the dependencies are listed on the Endor Labs’ user interface.

If the package.lock.json file is not present in the repository, Endor Labs triggers the restore process to generate the package.lock.json file and uses it to perform the dependency scans.

How Endor Labs performs static analysis on the code (Beta)

Endor Labs performs static analysis on the C# code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs looks for the project’s .dll files typically located within the bin directory.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• The following .NET programming languages are not currently supported for dependency resolution or call graph generation.
  • Projects written in F#
  • Projects written in Visual Basic
• When using the GitHub app, either resolve all the private and internal dependencies, or Configure Nuget private repositories before running a scan.
• Call graph binary discovery may fail if you use environment variables to define the name of the package in your *.csproj file.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors: .NET or Nuget is not installed or not present in the PATH environment variable. Install Nuget and try again.
• Unresolved dependency errors: This error occurs when the .csproj file can not be parsed or if it has syntax errors.

1.8 - Bazel

Bazel is an open-source build and test tool, which is commonly used in monorepos. Bazel is used to quickly build software across multiple languages.

Using Endor Labs, organizations relying heavily on Bazel can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan your Bazel projects

To successfully scan using Bazel:

• Scan your Bazel projects
  • Install software prerequisites
  • Select and build your Bazel targets
  • Run a scan
    • Option 1 - Quick scan
    • Option 2 - Deep scan
  • Understand the scan process

Install software prerequisites

The following prerequisite must be fulfilled:

• One or more WORKSPACE files must be present in your repository.
• The bazel command must be installed and available on the host system. To install Bazel, see the Bazel documentation.
• When performing deep scans make sure that you have the following minimum system specification requirements:
  • 4-core processor with 16 GB RAM for small projects
  • 8-core processor with 32 GB RAM for mid-size projects
  • 16-core processor with 64 GB RAM for large projects
• Bazel versions 5.x.x and 6.x.x are supported.

Select and build your Bazel targets

Endor Labs supports scanning targets using the following language-specific Bazel rules:

• Bazel Rules for Java support included for 4.1 or higher versions.

  • java_library

• Bazel Rules for Python support included for 0.9.0 or higher versions.

  • py_binary
  • py_library

• Bazel Rules for Golang support included for 0.40.1 or higher versions with Bazel versions 5.x.x, 6.x.x. 0.42.0 or higher versions with Bazel versions 7.x.x.

  • go_binary
  • go_library

• To scan with Endor Labs you’ll need to specify a list of targets to scan or a Bazel query for which all targets matching your query criteria are scanned.

• To select your Bazel targets you can search using the Bazel query language for specific targets to scan, or scan all targets that match a given query.

Here are some examples.

• To get all targets created with a java_binary build rule you can use the query:

bazel query 'kind(java_binary, //...)'

• To get all targets created with a py_binary build rule you can use the query:

bazel query 'kind(py_binary, //...)'

• To get all targets created with a go_binary build rule under the golang directory you can use the query:

bazel query 'kind(go_binary, //golang/...)'

In general, refine your query to select the most important targets in your monorepo and align with your existing build workflows.

You can choose to build the targets before running the scan. Use the bazel build commands to do this by passing a comma-separated list of targets. For example, //:test and //:test2 run bazel build //:test,//:test2. endorctl will attempt a scan if the targets are not built. endorctl uses bazel build //:test and bazel query 'deps( //:test)' --output graph to build your targets.

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects. See

Once you’ve selected your targets you can define the targets for scanning using one of three command line arguments:

To include or exclude specific targets you can pass a comma-separated list of targets to --bazel-exclude-targets or --bazel-include-targets.

To scan a specific list of targets, use the command:

endorctl scan --use-bazel --bazel-include-targets=//your-target-name

To scan a list of targets using the bazel query language use the following command and replace the query below with your own:

endorctl scan --use-bazel --bazel-targets-query='kind(java_binary, //...)

Finally, if your workspace file is not located at the root of the repository you must define the WORKSPACE file location for the targets you would like to scan. For example:

endorctl scan --use-bazel --bazel-targets-query='kind(java_binary, //...) --bazel-workspace-path=./src/java

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan will not perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

You can run a deep scan with the following commands:

endorctl scan --use-bazel --bazel-include-targets-query=//your-target-name

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --use-bazel --bazel-include-targets-query=//your-target-name -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but will not impact subsequent scans.

Organizations might not own some parts of the software internally and findings are actionable by another team. These organizations can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --disable-private-package-analysis

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

To understand the scan process for Java projects, see Endor Labs for Java.

To understand the scan process for Python projects, see Endor Labs for Python.

To understand the scan process for GO projects, see Endor Labs for GO.

Known limitations

• Scanning Java binaries using Bazel is not supported.

1.9 - Swift/Objective-C (Beta)

CocoaPods is a popular package manager for Swift and Objective-C. It simplifies the integration of Swift and Objective-C dependencies through Podfile declaration and automated installation.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan CocoaPods projects

To successfully scan your CocoaPods projects:

• Install software prerequisites
• Build CocoaPods projects
• Run a scan
• Understand the scan process
  • Known limitations

Install software prerequisites

The following prerequisites must be fulfilled:

• All applications monitored by Endor Labs must be on CocoaPods versions 0.9.0 or higher.
• A Podfile and a Podfile.lock must be present in your CocoaPods project.

Build CocoaPods projects

If the Podfile.lock is not present in your repository, run the following command to create the Podfile.lock for your Podfile.

pod install

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies. Use the flag --languages=swift along with the endorctl scan to discover Swift and Objective-C dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs looks for the Podfile and Podfile.lock files to discover the dependencies used by an application.

• A Podfile is a configuration file used in CocoaPods projects to specify the required libraries or packages for the project’s dependencies.
• A Podfile.lock file is a CocoaPods specification file used to define the metadata and dependencies.

To successfully discover Swift and Objective-C dependencies, both Podfile and Podfile.lock files must be present in your project for each Podfile.

Known limitations

• Call graphs are not supported for the CocoaPods projects.
• If a Podfile.lock file is not present, Endor Labs will skip analyzing the project and present a warning that the package was skipped.

1.10 - Scala

Scala is a general-purpose and scalable programming language widely used by developers. Endor Labs supports the scanning and monitoring of Scala projects managed by the interactive build tool sbt.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Scala projects

To successfully scan your Scala applications:

• Install software prerequisites
• Run a scan
• Understand the scan process
• Known Limitations
• Troubleshoot errors

Install software prerequisites

The following prerequisites must be fulfilled:

• A manifest file for the Scala build tool, build.sbt must be present in your repository.
• If your sbt version is lower than 1.4, you must install the sbt-dependency-graph plugin. The sbt-dependency-graph plug-in is by default integrated into the sbt versions 1.4 and higher.
• Make sure that the project/build.properties file has the required sbt version.
• Make sure that your system has a minimum 8-core processor with 32 GB RAM to successfully scan Scala projects.

Option 1 - Quick scan

You must be able to build your Scala projects before running a scan. The standard .sbt cache must exist and contain successfully downloaded dependencies.

1. Ensure your repo has build.sbt file and run the following commands making sure it builds the project successfully.

sbt compile

sbt projects

2. Make sure sbt dependencyTree runs successfully inside the project directory.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 1 - Deep scan

You must be able to build your Scala projects before running a scan. The build artifact as well as the standard .sbt cache must exist and contain successfully downloaded dependencies.

1. Ensure your repo has build.sbt file and run the following commands making sure it builds the project successfully.

sbt package

sbt projects

2. Make sure sbt dependencyTree runs successfully inside the project directory.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs scans Scala projects by executing sbt plugins and inspecting the build.sbt file to retrieve information about direct and transitive dependencies.

• The build.sbt file is a configuration file used in Scala projects with sbt to define project settings, dependencies, and build tasks. This file provides the necessary configuration and instructions to sbt on resolving and managing project dependencies.

• The sbt dependency graph plugin visualizes the dependencies between modules in a Scala project.

Endor Labs analyzes information from both these methods to determine different components, binary files, manifest files, images, and more in the Scala codebase and presents finding policy violations, identifying, and resolving dependencies.

Using Endor Labs users can gain significant insights into the structure and relationships of their Scala project’s dependencies, aiding in managing dependencies effectively, identifying potential issues, and ensuring a well-organized and maintainable codebase.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• Software composition analysis for Scala on Microsoft Windows operating systems is currently unsupported.
• Scala packages built with Gradle are not currently supported.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors: These errors occur if:
  • If sbt is not installed or present in the path variable. Install SBT 1.4 or higher versions and try again.
  • If the sbt version mentioned in the project or the build.properties file is lower than 1.4 and the sbt-dependency-graph plug-in is not installed. Install the sbt-dependency-graph and try again.
• Dependency graph errors - Scala by default imports MiniDependencyTreePlugin which is a mini version of the sbt-dependency-graph plugin and it supports only dependencyTree cmd. To get complete features of the sbt-dependency-graph plugin, add DependencyTreePlugin to your project/plugins.sbt file and run the scan again. See Scala documentation for details.

1.11 - PHP

PHP is a popular server-side scripting language primarily used for web development. Endor Labs supports the scanning and monitoring of PHP projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan PHP projects

To successfully scan your PHP applications:

• Scan PHP projects
  • Install software prerequisites
  • Build PHP projects
    • Configure private Composer package repositories
  • Run a scan
  • Understand the scan process
    • Known Limitations
  • Troubleshoot errors

Install software prerequisites

• One of the following prerequisites must be fulfilled:
  • The PHP project must contain a composer.json file. If the project includes the composer.lock file it is beneficial, but this is not a mandatory requirement.
  • If the composer.lock file is not present in the repository, it is necessary to have PHP and Composer installed before running a scan on your local system.
• The following versions are supported for PHP and Composer:
• PHP 5.3.2 and higher versions
• Composer 2.2.0 and higher versions

Build PHP projects

You can choose to build your PHP projects before running a scan. This will ensure that composer.lock is created.

Ensure your repo has composer.json and run the following command making sure it builds the project successfully.

composer install

If the project is not built, endorctl will build the project during the scan and generate composer.lock. If the repository includes a composer.lock, endorctl uses this file for dependency resolution and does not create it again.

Configure private Composer package repositories

If you have a private registry and internal dependencies on other projects, you must configure private registries.

To configure private Composer package repositories:

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Packagist and click Manage.
3. Click Add Package Manager.
4. In PACKAGE MANAGER HOST, enter the host domain of the package manager.
5. From the available options, choose the authentication method for private repositories.
6. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the required details of your private package manager repository.
7. Click Add Package Manager to save your configuration.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs discovers all composer.json files in your PHP project and uses these files to resolve the dependencies of your packages. Composer is a PHP dependency management tool that enables you to specify the libraries your project relies on and manages the process of installing or updating them. The dependencies and findings are listed in the Endor Labs application individually for every composer.json file.

In Endor Labs’ dependency management, the resolution of dependencies is based on both composer.json and composer.lock files. The composer.lock file is generated by Composer and includes information such as resolved versions, package information, transitive dependencies, and other details. Using the composer.lock file ensures deterministic dependency installation by recording the exact versions of installed dependencies and their transitive dependencies. If the composer.lock file is not present in the repository, Endor Labs generates the composer.lock file, and uses it to analyze the operational and security risks associated with your package’s dependencies. Endor Labs fetches the dependency information and creates a comprehensive dependency graph.

Known Limitations

Call graphs are not supported for PHP projects.

Troubleshoot errors

• Unresolved dependency errors: The composer.json is not buildable. Try running composer install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.12 - Rust

Rust is a software programming language widely used by developers. Endor Labs supports scanning and monitoring of Rust projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Scan Rust projects

To successfully scan your Rust applications:

1. Install software prerequisites
2. Build Rust projects
3. Run a scan
4. Understand the scan process
5. Troubleshoot errors

Install software prerequisites

• Make sure the following pre-requisites are installed:
  • Package Manager Cargo - Any version
  • Rust - Any version,
• Install Rust using the latest Rustup tool.
• Make sure that you have a minimum system requirement specification of an 8-core processor with 32 GB RAM. s
• Use a system equipped with either Mac OS X or Linux operating systems to perform the scans.

Build Rust projects

Ensure your repo has Cargo.toml file and run the following command making sure it builds the project successfully.

cargo build

If the project is not built, endorctl will build the project during the scan and generate the Cargo.lock file. If the repository includes a Cargo.lock file, endorctl uses this file for dependency resolution and does not create it again.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan will not perform the reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl comes bundled as a docker image for portable usage and integration into continuous integration (CI) pipelines. To configure endorctl docker image, you must perform the following steps:

1. Install Docker
2. Get Endor Labs API credentials
3. Export your environment variables
4. Scan Rust projects

Export your environment variables

endorctl requires four environment variables for your source control system and your Endor Labs tenant.

These environment variables are:

• ENDOR_API_CREDENTIALS_KEY - The API key used to authenticate against the Endor Labs API.
• ENDOR_API_CREDENTIALS_SECRET - The API key secret used to authenticate against the Endor Labs API.
• ENDOR_NAMESPACE - The Endor Labs namespace you want to scan against. Locate the namespace from the top left-hand corner, under the Endor Labs logo on the Endor Labs application,.
• SOURCE_PATH - The path to your source code that may be mounted to the docker container.

To export your environment variables run the following commands and insert the environment variable values in each command.

export ENDOR_NAMESPACE=name-of-your-namespace
export ENDOR_API_CREDENTIALS_KEY=endorlabs-api-key
export ENDOR_API_CREDENTIALS_SECRET=endorlabs-api-secret
export SOURCE_PATH=/path/to/your/source/code

Scan Rust projects

To scan your rust project run the following command:

docker run -it --rm \
 -e ENDOR_NAMESPACE=$ENDOR_NAMESPACE \
 -e ENDOR_API_CREDENTIALS_KEY=$ENDOR_API_CREDENTIALS_KEY \
 -e ENDOR_API_CREDENTIALS_SECRET=$ENDOR_API_CREDENTIALS_SECRET \
 -v $SOURCE_PATH:/root/endorlabs \
 us-central1-docker.pkg.dev/endor-ci/public/endorctl:latest \
 scan --path=/root/endorlabs

Use the following command to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

docker run -it --rm \
 -e ENDOR_NAMESPACE=$ENDOR_NAMESPACE \
 -e ENDOR_API_CREDENTIALS_KEY=$ENDOR_API_CREDENTIALS_KEY \
 -e ENDOR_API_CREDENTIALS_SECRET=$ENDOR_API_CREDENTIALS_SECRET \
 -v $SOURCE_PATH:/root/endorlabs \
 us-central1-docker.pkg.dev/endor-ci/public/endorctl:latest \
 scan --path=/root/endorlabs -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs performs the following steps when it scans Rust projects:

• Resolves dependencies for the package version
• Performs static analysis on your Rust code

Resolving Dependencies & Static Analysis for Rust

Endor Labs leverages the Cargo.toml file in Rust and uses this file to build the package version using cargo. Endor Labs uses the output from cargo metadata to resolve dependencies specified in Cargo.toml files and construct the dependency graph.

Perform Static Analysis on Your Rust Code

• After building the package, call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project. Use the call graphs to understand if vulnerabilities in your Rust code are reachable through a function associated with the known vulnerability.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• To perform static analysis on Rust packages, the package version must be able to be successfully built.
• Performing Endor Labs scans on the Microsoft Windows operating system is currently unsupported.

Troubleshoot errors

• Host system check failure errors: These errors occur when Rust is not installed or not present in the path variable. Install Rust and try again.
• Call graph errors: These errors occur when the project is not buildable and the required dependencies cannot be located.

2 - Scanning strategies

There are several key scanning strategies your team should be aware of as your deploy Endor Labs in your environemnt.

• Testing and monitoring different versions of your code
  • How to deploy a strategy for reporting
    • Scanning detatched refs
  • Implementing baseline scans
• Accelerating preventative control adoption with CI baselines
  • Implementing baseline scan into your program

Testing and monitoring different versions of your code

Across the software engineering lifecycle its important that continuous testing is separated from what is monitored and reported on regularly. Often, engineering organizations want to test each and every change that enters a code base, but if security teams reported on each test they would quickly find themselves overwhelmed with noise. Endor Labs enables teams to separate what should be reported on relative to what should be tested but not reported on. Endor Labs allows teams to select reporting strategies for their software applications when integrated into CI/CD pipelines.

Here are the primary scanning and reporting strategies:

• Reporting on the default branch - All pull request commits are tested and all pushes or merges to the default branch are reported on and monitored by security and management teams.
• Reporting on the latest release - All reporting and monitoring is performed against tagged release versions. This requires each team have a mature release tagging strategy.

How to deploy a strategy for reporting

The endorctl scan command by default will continuously monitor a version of your code for new findings such as unmaintained, outdated or vulnerable dependencies in the bill of materials for a package. In order to test a version of your code without monitoring and reporting on it, use the flag --pr or environment variable ENDOR_SCAN_PR as part of your scan.

When adopting a strategy such as reporting on the default branch, you will want to run any push or merge event to the default branch without the --pr flag and run any pull_request or merged_request event with the --pr flag. This allows you to test changes before they have been approved and report what has been merged to the default branch as your closest proxy to what is in production.

Lets use the following GitHub actions workflow as an example! In this workflow any push event will be scanned without the --pr flag but any pull_request event is scanned as a point in time test of that specific version of your code.

name: Endor Labs Scan
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload sarif artifact to GitHub
      contents: read # Used to check out a private repository but actions/checkout.
      actions: read # Required for private repositories to upload sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication to Endor Labs
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Build Package
        run: mvn clean install
      - name: Endor Labs Scan Pull Request
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
          pr_baseline: $GITHUB_BASE_REF
      - name: Endor Labs Reporting Scan
        if: github.event_name == 'push'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: false
          sarif_file: 'findings.sarif'
      - name: Endor Labs Testing Scan
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
      - name: Upload findings to github
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'findings.sarif'

Scanning detatched refs

In some CI/CD based environments, each time code is pushed to the default branch the exact commit SHA is checked out as a detatched Git Reference. This is notably the case with Jenkins, CircleCI and GitLab Pipelines.

In these senarios on push or merge events Endor Labs must be told that the reference should be monitored as the default branch. You can do this with the --detatched-ref-name flag or ENDOR_SCAN_DETACHED_REF_NAME environment variable. You should also couple this flag with the --as-default-branch flag or ENDOR_SCAN_AS_DEFAULT_BRANCH environment variable. This allows you to set this version of code as a version that should be monitored as well as define the name associated with the branch.

This strategy may be used for both a strategy reporting on the default branch on push events and a strategy reporting on tag creation event for that version of code.

You can see in the below GitLab Pipelines example defining the logic to manage a detatched reference on GitLab.

    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi

The full GitLab pipelines reference can be found below:

Endor Labs Dependency Scan:
  stage: Scan
  image: node # Modify this image to align with the build tools nessesary to build your software packages
  dependencies: []
  variables:
    ENDOR_ENABLED: "true"
    ENDOR_ALLOW_FAILURE: "true"
    ENDOR_NAMESPACE: "demo"
    ENDOR_PROJECT_DIR: "."
    ENDOR_ARGS: |
      --path=${ENDOR_PROJECT_DIR}
      --show-progress=false
      --detached-ref-name=$CI_COMMIT_REF_NAME
      --output-type=summary
      --exit-on-policy-warning
      --dependencies --secrets --git-logs      
  before_script:
    - npm install yarn
  script:
    - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl;
    - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
      if [ $? -ne 0 ]; then
       echo "Integrity check failed";
       exit 1;
      fi
    - chmod +x ./endorctl
    - if [ "$DEBUG" == "true" ]; then
        export ENDOR_LOG_VERBOSE=true;
        export ENDOR_LOG_LEVEL=debug;
      fi
    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi
    - ./endorctl scan ${ENDOR_ARGS}
  rules:
  - if: $ENDOR_ENABLED != "true"
    when: never
  - if: $CI_COMMIT_TAG
    when: never
  - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH && $ENDOR_FEATURE_BRANCH_ENABLED != "true"
    when: never
  - if: $ENDOR_ALLOW_FAILURE == "true"
    allow_failure: true
  - if: $ENDOR_ALLOW_FAILURE != "true"
    allow_failure: false

Implementing baseline scans

One of the common concerns software development teams have when adopting preventative controls is ownership of issues. Often, software has accrued significant technical debt, or new vulnerabilities arise that don’t directly impact their changes. Security teams want to have all known issues addressed while the development teams are focused on fixing issues or delivering core business value. They can’t be hindered each time a new issue impacts their entire code base.

To prevent new issues from entering the environment, security teams sometimes set policies that may break the build or return a non-zero exit code that can fail automated tests. This creates friction as there is no context around what changes a developer is responsible for versus what technical debt exists in a codebase on that day.

Establishing a baseline of what issues already exist in a software project and what issues may occur because of new updates is crucial to enabling preventative control adoption.

Accelerating preventative control adoption with CI baselines

The high-level steps to establish and measure policies against a baseline scan are as follows:

1. Establish a baseline scan of your default branch or any other branch that undergoes regular testing
2. Integrate baseline scans into your automated workflows
3. Evaluate policy violations within the context of the branches to which you routinely merge

Implementing baseline scan into your program

Development teams often have different delivery strategies. Some merge changes to a default branch. Others merge to a release branch that is then released to their environment. While these strategies differ across organizations, a baseline scan must exist to measure against attribute ownership.

To establish a baseline scan, your team must perform regular scans on the branch to which you merge. This often means that you scan each push of your default branch to monitor your environment and you test each pull request using the --pr and --pr-baseline flags.

The --pr flag is a user’s declaration that they are testing their code as they would in a CI pipeline. The --pr-baseline flag tells Endor Labs which git reference to measure any changes.

For this example, we will use the default branch as a merging strategy. In this strategy, you’ll want to scan the default branch on each push event to re-establish your baseline. You’ll also want to establish your CI baseline as the default branch.

The following GitHub workflow illustrates this strategy:

• See the GitHub Actions documentation for GitHub default variables.
• See the GitLab CI/CD documentation for the GitLab default variables.

name: Endor Labs Scan
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload sarif artifact to GitHub
      contents: read # Used to check out a private repository but actions/checkout.
      actions: read # Required for private repositories to upload sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication to Endor Labs
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Build Package
        run: mvn clean install
      - name: Endor Labs Scan Pull Request
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
          pr_baseline: $GITHUB_BASE_REF
      - name: Endor Labs Reporting Scan
        if: github.event_name == 'push'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: false
          sarif_file: 'findings.sarif'
      - name: Endor Labs Testing Scan
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
      - name: Upload findings to github
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'findings.sarif'

3 - Scoping scans

Setup scan inclusion and exclusion patterns

Exclude and include filters help your team to focus their attention on the open source packages that matter most and to improve scan performance. Inclusion and exclusion patterns must not be used together so it is important to pick a strategy that works best for you. We recommend using inclusion patterns when you have many packages that you want to scan separately and exclusion patterns when you want to filter out packages that are not important to you.

You can include or exclude packages using the following standard patterns:

1. Include or exclude specific packages
2. Include or exclude specific directories
3. Include or exclude with a Golang style regex pattern.

To include or exclude a package based on its file name:

• Include
• Exclude

endorctl scan --include=path/to/your/manifest/file/package.json

endorctl scan --exclude=path/to/your/manifest/file/package.json

To include or exclude a package based on its directory

• Include Directory
• Exclude Directory

endorctl scan --include="directory/path/"

endorctl scan --exclude="path/to/your/manifest/"

To exclude or include using a Golang style regex:

• Include Regex
• Exclude Regex

endorctl scan --include=".*/build/.*"

endorctl scan --exclude=".*/build/.*"

Best practises

Here are a few best practises that you must take into consideration while using include and exclude patterns:

• Ensure that you enclose your exclude pattern in double quotes to avoid shell expansion issues. For example, do not use --exclude=.*/build/.*, instead, use --exclude=".*/build/.*".
• When performing excludes and includes in the Windows environment
  • Use double backslashes \\ which is more common in regular expressions when dealing with Windows file paths. For example, the exclusion pattern for Windows is --exclude=".*\\build\\.*"
  • Do not use single quotes. Use double quotes. For example, do not use --exclude='.*\\build\\.*' instead use --exclude=".*\\build\\.*".
• Inclusion patterns are not designed for documentation or example directories. You can not explicitly include documentation or example directories:
  • docs/
  • documentation/
  • groovydoc/
  • javadoc
  • man/
  • examples/
  • demos/
  • inst/doc/
  • samples/
• If you are using Javascript workspaces, take special consideration when including and excluding the root package:
  • When using include or exclude patterns, it’s crucial to make sure you never exclude and always include the parent workspace package. Otherwise, all child packages won’t build properly.
  • You can always exclude child packages in the workspace if the root is included.
  • There is only one lock file for the workspace that exists in the workspace root directory. Make sure to include the lock file to perform a successful scan.

4 - Working with monorepos

Large monorepos are a reality for many organizations. Since monorepos can have anywhere from tens to even hundreds of packages scanning all packages in a monorepo can take significant periods of time. While the time requirements may vary based on your development team and pipeline times, in general, development teams need quick testing times to improve their productivity while security teams need full visibility across a monorepo. These two needs can conflict without performance engineering or an asynchronous scanning strategy. This documentation outlines some performance engineering and scanning strategies for large monorepos.

Asynchronous scanning strategies

When scanning a large monorepo, a common approach taken by security teams is to run an asynchronous cron job outside of a CI/CD-based environment. This is often the point of least friction but is prohibitive. With this approach, inline blocking of critical issues is not generally possible. We would be remiss not to mention this as a scanning strategy for monorepos but this approach is NOT recommended beyond a step to get initial visibility into a large monorepo.

Performance Enhancements for inline scanning strategies

The following performance enhancements may be used with Endor Labs to enable the scanning of large monorepos:

Scoping scans based on changed files

For many CI/CD systems path filters are readily available. For example, with GitHub actions, dorny path filters is a readily accessible way to establish a set of filters by a path. This is generally the most effective path to handle monorepo deployments but does require the highest level of investment in terms of human time. The human time investment is made up for by the time saved by reducing the need to scan everything on each change.

Based on the paths that change you can scope scans based on the files that have actually changed. For example, you can scan only the packages in a monorepo that are housed under the ui/ directory when this path has changed by running a scan such as endorctl scan --include=ui/ when this path has been modified.

Using a path filtering approach each team working in a monorepo would need to be responsible for the packages that they maintain, but generally, each team may be associated with one to several pre-defined directory paths.

Parallelizing scans for many packages

When scanning a large monorepo organizations can choose to regularly scan the whole monorepo based on the packages or directories they’d like to scan. Different jobs may be created that scan each directory simultaneously.

Parallelizing with scoped scans

Using scoped scans for monorepos with multiple parallel include patterns is a common performance optimization for monorepos.

Below is an example parallel GitHub action scan that can be used as a reference. In this example, the directory ui/ and backend/ are both scanned simultaneously and the results are aggregated by Endor Labs.

This approach can improve the overall scan performance across a monorepo where each directory can be scanned independently.

name: Parallel Actions
on:
  push:
    branches: [main]
jobs:
  scan-ui:
    runs-on: ubuntu-latest
    steps:
      - name: UI Endor Labs Scan
        run: endorctl scan --include=ui/
  scan-backend:
    runs-on: ubuntu-latest
    steps:
      - name: Backend Endor Labs Scan
        run: endorctl scan --include=backend/

To include or exclude a package based on its directory.

endorctl scan --include="directory/path/"

See scoping scans for more information on approaches to scoping scans.

Parallelizing across languages

For teams that work out of smaller monorepos, it is often most reasonable to parallelize scanning based on the language that is being scanned and performance optimize for individual languages based on need.

Below is an example parallel GitHub action scan that can be used as a reference. In this example, Javascript and Java are scanned at the same time and aggregated together by Endor Labs. This approach can improve the overall scan performance across a monorepo with multiple languages.

name: Parallel Actions
on:
  push:
    branches: [main]
jobs:
  scan-java:
    runs-on: ubuntu-latest
    steps:
      - name: Java Endor Labs Scan
        run: endorctl scan --languages=java
  scan-javascript:
    runs-on: ubuntu-latest
    steps:
      - name: Javascript Endor Labs Scan
        run: endorctl scan --languages=javascript,typescript

To scan a project for only packages written in typescript or javascript use the command:

endorctl scan --languages=javascript,typescript

To scan a project for only packages used for packages written in java use the command:

endorctl scan --languages=java

5 - SCM Posture Management

Secure critical components of your software supply chain, including code, open source libraries to ensure the security posture of your software development code and its repositories.

• Endor Labs comes with out-of-the box finding policies that help you detect misconfigurations, enforcing coding best practices, and staying compliant with industry standards such as CIS benchmarks for GitHub and more.

• Review the available finding SCPM policy templates.

• Endor Labs regularly updates its existing policies and also includes several new policies. Configure policy settings to ensure that you benefit from these regular updates.

• The policies provide provide up-to-date insights into critical risks, so you can manage security threats before your projects even start. They also include remediation advice that can help you fix and mitigate issues.

6 - Perform scans in CI pipelines

endorctl is a command line utility designed to bring the functionality of Endor Labs into your software delivery workflows.endorctl has several command flags to help you facilitate operational and security risk monitoring. Developers can integrate Endor Labs into Continuous Integration Workflows using the endorctl scan.

• endorctl scan - You can use endorctl scan to monitor your projects using Endor Labs, and you can update the scan information each time to keep monitoring the project for new findings. The endorctl scan command will scan a specific version of your repository, such as the default branch, a tagged release version, or a commit SHA.
• endorctl scan --pr - You can use the endorctl scan --pr command to scan a specific version of your source code for security and operational risks as part of your continuous integration workflows or CI runs. The endorctl scan --pr command performs a one-time evaluation of your project, focusing on security and operational risks, rather than providing continuous monitoring. CI runs are shown in the Scan History section of each project and are stored for 30 days so that you can analyze and review them on the Endor Labs user interface.

Any continuous integration workflows generally run using the endorctl scan --pr command unless a scan is run on a created tag release, a push to the default or specific branch, or a commit SHA that will be deployed to production.

Authenticating in CI with Keyless Authentication

Keyless Authentication enhances security and minimizes the expenses associated with secret rotation. Keyless authentication is Endor Labs recommended path to scan your projects in the CI workflows. See Keyless Authentication for details.

Configuring GitHub Actions workflows with Endor Labs

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. See the Endor Labs Configuring GitHub Actions workflows. for details.

Configuring GitLab CI/CD pipelines with Endor Labs

GitLab CI/CD pipelines are a part of GitLab’s integrated continuous integration and deployment features. They allow you to define and automate the different stages and tasks in your software development workflow. See the Endor Labs Configuring GitLab CI/CD for details.

6.1 - Set up keyless authentication

At Endor Labs, we believe that the most secure secret is one that doesn’t exist. That’s why in CI/CD environments we recommend using keyless authentication for machine authentication. Keyless Authentication uses OAuth for API authentication and removes the need to maintain and rotate an API key to Endor Labs.

Keyless Authentication is more secure and reduces the cost of secret rotation.

Configure keyless authentication using:

• Google Cloud
• GitHub OIDC
• AWS Cloud

Enabling Keyless Authentication in Google Cloud

To enable Keyless Authentication in GCP you’ll first need permissions to create service accounts and assign these accounts roles to GCP.

The workflow to enable keyless authentication is:

1. Create a service account with no permissions for federation.
2. If you do not attach a service account to compute resources or use the default service account, we recommend creating a new service account for the compute resources. Create a service account to attach to compute resources and impersonate the federation service account.
3. Create an authorization policy to allow the federation service account to authenticate to Endor Labs.
4. Provision the compute resources with the appropriate permissions.
5. Test Keyless authentication.

Creating GCP Service Accounts and Authorization Policies

To create your service accounts, first export your GCP project name as an environment variable:

export PROJECT=<insert-gcp-project>

Once you’ve set the environment variable you’ll create a service account that will be used to federate access to Endor Labs and will be provided permission to access the Endor Labs APIs:

Step 1: Create a federation service account called endorlabs-federation:

gcloud iam service-accounts create endorlabs-federation --description="Endor Labs Keyless Federation Service Account" --display-name="Endor Labs Federation Service Account"

We’ll also create a second service account, that will have access to impersonate the endorlabs-federation account. We’ll call this endorlabs-compute-service.

This is needed if you don’t already have service accounts for your compute resources. If you do, you need to modify the existing permissions to allow the existing service account to create a federation token.

Step 2: Create a keyless authentication service account to assign to compute resources called endorlabs-compute-service:

gcloud iam service-accounts create endorlabs-compute-service --description="Endor Labs Service account for keyless authentication" --display-name="Endor Labs Compute Instance SA"

Finally, we’ll assign endorlabs-compute-service permissions to impersonate the endorlabs-federation account to authenticate to Endor Labs through OIDC.

Step 3: Assign the serviceAccountOpenIdTokenCreator role to the endorlabs-compute-service service account:

gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com" --role="roles/iam.serviceAccountOpenIdTokenCreator"

Once we’ve created the necessary account permissions we will create an authorization policy in Endor Labs to allow the account endorlabs-federation to your Endor Labs tenant:

Use the following command to create an authorization policy in Endor Labs.

Note: Make sure to replace with your Endor Labs tenant name and with your GCP project name in the following command.**

endorctl api create -r AuthorizationPolicy -d '{
    "tenant_meta": { "namespace": "<your-tenant>" },
    "meta": {
        "name": "Keyless Auth",
        "kind": "AuthorizationPolicy",
        "tags": ["gcp"]
     },
     "spec": {
        "clause": ["email=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com", "gcp"],
        "target_namespaces": ["<your-tenant>"],
        "propagate": true,
        "permissions": {
            "rules": {},
            "roles": [
                "SYSTEM_ROLE_CODE_SCANNER"
            ]
        },
    }
}'

You’ve now set up the foundation of keyless authentication. You’ll now need to provision your compute resources with the appropriate GCP scopes and service account.

See Provisioning and Testing Keyless Authentication for GKE workloads for instructions on setting up GKE for keyless authentication.

See Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances for instructions on setting up a virtual machine instance for keyless authentication.

Provisioning and Testing Keyless Authentication for GKE workloads

Prerequisites

The following pre-requisites are required to setup keyless authentication on GKE workloads:

• Workload identity is enabled on the target GKE cluster. See the GCP documentation on using workload identity for instructions on migrating existing cluster node pools or creating new clusters to use GCP workload identity.
• The gcloud auth plugin is installed and operational on your machine. See the GCP instructions on enabling the gcloud auth plugin for more details.
• The kubectl CLI is installed. See the Kubernetes documentation for instructions.

Procedure:

1. (Optional) Create a GKE cluster with workload identity enabled if you do not already have a GKE cluster
2. Authenticate to the GKE cluster
3. (Optional) Create a namespace for Endor Labs scans
4. Create a Kubernetes service account to impersonate your GKE compute service account
5. Bind your Kubernetes service account to your GCP compute service account
6. Annotate your Kubernetes service account with your GCP service account to complete your binding
7. Test a scanning workload using keyless authentication

Setting Up and Testing Keyless authentication in GKE

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT
• The GKE cluster as CLUSTER_NAME

export PROJECT=<Insert_GCP_Project>
export CLUSTER_NAME=<GKE_CLUSTER>

Optional Step 1: To create a GKE cluster with workload identity enabled if you do not already have a GKE cluster with workload identity enabled run the following command:

gcloud container clusters create keyless-test --workload-pool=endor-github.svc.id.goog --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To authenticate to your GKE cluster run the following command:

gcloud container clusters get-credentials $CLUSTER_NAME

Optional Step 3: To create a namespace for Endor Labs scans run the following command:

kubectl create namespace endorlabs

Step 4: To create a Kubernetes service account to impersonate your GKE compute service account run the following command:

kubectl create serviceaccount endorlabs-compute-service -n endorlabs

Step 5: To bind your Kubernetes service account to your GCP compute service account run the following command:

Note: Make sure to replace in the following command with your GCP project name.

gcloud iam service-accounts add-iam-policy-binding endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:<insert-your-project>.svc.id.goog[endorlabs/endorlabs-compute-service]"

Step 6: To annotate your Kubernetes service account with your GCP service account to complete your binding run the following command:

Note: If you have created a different service account name replace endorlabs-compute-service with the appropriate service account name.

kubectl annotate serviceaccount endorlabs-compute-service -n endorlabs iam.gke.io/gcp-service-account=endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com

Step 7: Test scan your project with keyless authentication

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication.

Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances

The following high-level procedure describes the required steps to use keyless authentication with a GCP virtual machine instance:

Procedure:

1. Create a virtual machine instance with the appropriate scopes
2. Download and install endorctl on the virtual machine instance
3. Launch a test scan with keyless authentication

Setting up and Testing Keyless authentication on a GCP virtual machine instance

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT

export PROJECT=<Insert_GCP_Project>

To successfully test keyless authentication first you’ll need to provision a compute resource with the service account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com and the scope https://www.googleapis.com/auth/cloud-platform:

Step 1: To create a virtual machine instance with the appropriate scopes run the following command:

gcloud compute instances create test-keyless --service-account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To download and install endorctl on the virtual machine instance run the following series of commands:

First, SSH to the virtual machine instance you’ve created to test:

gcloud compute ssh --zone "us-west1-b" "test-keyless"  --project $PROJECT

Then download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com for federated access to Endor Labs such as in the below example:

endorctl api list --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com -r Project -n <insert-your-tenant> --count

If this scan runs successfully you’ve tested and scanned a project with keyless authentication to Endor Labs.

Enabling Keyless Authentication in GitHub

To enable Keyless Authentication for GitHub Actions, you’ll need to perform the following steps:

1. Ensure you are using the Endor Labs GitHub Action in your GitHub workflow.
2. Edit your GitHub Action workflow to add permission settings for the GitHub id-token
3. Create an authorization policy for GitHub Action OIDC
4. Test that you can successfully scan a project using Github Action OIDC

Add a GitHub Action OIDC authorization policy

To ensure that the GitHub action OIDC identity can successfully login to Endor Labs you’ll need to create an authorization policy in Endor Labs.

To create an authorization policy:

1. Under Manage go to Access Control
2. Navigate to the “Auth Policy” tab
3. Click on the “Add Auth Policy” button
4. Select “GitHub Action OIDC” as your identity provider
5. Select the permission for the GitHub Action. This permission should be “Code Scanner”
6. For the claim use the key user and put in a matching value that maps to the organization of your GitHub repository.

Configuring your GitHub Action workflow

To configure your GitHub Action workflow with GitHub Action OIDC you can use the following example as a baseline.

The important items in this workflow are:

1. The Usage of the Endor Labs GitHub action.
2. Setting Job level permissions to allow writing to the GitHub id-token

name: Example Scan of OWASP Java
on: workflow_dispatch
jobs:
  create_project_owasp:
    permissions:
      id-token: write # This is required for requesting the JWT
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3
        with:
          repository: OWASP-Benchmark/BenchmarkJava
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Compile Package
        run: mvn clean install
      - name: Scan with Endor Labs
        uses: endorlabs/github-action@main # This workflow uses the Endor Labs GitHub action to scan.
        with:
          namespace: 'demo'
          scan_summary_output_type: 'json'
          pr: false
          scan_secrets: true
          scan_dependencies: true

Now that you’ve successfully configured your GitHub action workflow file you can use this workflow file or one of your own designs to run a test scan using Keyless authentication for GitHub actions.

Enabling Keyless Authentication in AWS

To enable Keyless Authentication in AWS you’ll first need permissions to create or modify two roles and an instance profile with the appropriate roles configured

The two roles you will need are:

1. An instance access role - The instance access role is assigned to the compute resource, which needs to access Endor Labs. Your instance access role may already exist and you must ensure this role provides the permissions to allow the role to assume the role of a dedicated federation role.
2. A dedicated federation role - The dedicated federation role should have no permissions in AWS. Endor Labs will authorize requests that come from this role.

Procedure:

1. Create or modify an existing Instance Profile that may be used to assign a role to your EC2 instance.
2. Create or modify a role an instance access role, which enables services to assume a dedicated federation role.
3. Assign this role to the Instance Profile
4. Create a dedicated federation role to provide access to Endor Labs, which will be assumed by the instance access role
5. Create an authorization policy in Endor Labs
6. Test Keyless Authentication

Create or select an Instance Profile

An instance profile allows users to attach a single role to an EC2 instance. If you do not already have a pre-defined role or instance profile used by your EC2 instances you should create an instance profile for Endor Labs access.

To create an instance profile using the AWS CLI

aws iam create-instance-profile --instance-profile-name EndorLabsAccessProfile

Create or modify an instance access role

To successfully authenticate to Endor Labs you will need to assign an instance access role to the instance profile you created above.

The instance access role must at a minimum allow the compute resources that require access to perform the action sts:AssumeRole. If you already have a role you intend to assign to your instance profile, ensure that it has permissions to allow your compute resources to perform this action.

If you do not have an existing role you intend to use, create the role named endorlabs-instance-access-role using the instructions below.

First, add the following json to a file called endorlabs-instance-access-role.json

cat > endorlabs-instance-access-role.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Use this file to create your instance access role using the following command:

aws iam create-role --role-name endorlabs-instance-access-role --assume-role-policy-document file://endorlabs-instance-access-role.json

Next, ensure that the instance access role is assigned to the instance profile using the following command:

aws iam add-role-to-instance-profile --instance-profile-name EndorLabsAccessProfile --role-name endorlabs-instance-access-role

Finally, create your EC2 instance and ensure that your instance profile is assigned to it.

Create a dedicated federation role

A dedicated federation role is leveraged to provide a least privileged role that enables access to Endor Labs. This role is designed to be assumed only by specific other roles and does not provide access to AWS resources.

To create your federation role you will need the name and AWS account number of the instance access role, which should look similar to the below policy json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$ACCOUNT:role/$ROLE_NAME"
            },
            "Action": "sts:AssumeRole"
        }
    ]
  }

First, get the account number of the role:

export ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')

Then define the name of the instance access role. For the following example, we will assume it is endorlabs-instance-access-role.

export ROLE_NAME=endorlabs-instance-access-role

Next, create the IAM policy document:

cat > endorlabs-federation-aws-role.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::${ACCOUNT}:role/${ROLE_NAME}"
          },
          "Action": "sts:AssumeRole"
      }
  ]
}
EOF

Next, apply the policy document as a role:

aws iam create-role --role-name endorlabs-federation --assume-role-policy-document file://endorlabs-federation-aws-role.json

Finally, fetch the ARN of the IAM role you’ve created using the following command and create an authorization policy for it in Endor Labs.

To fetch the ARN of the Endor Labs federation role use the following command:

aws iam list-roles | jq -r '.Roles[] | select(.Arn|contains("endorlabs-federation"))'.Arn

To add the authorization policy to Endor Labs:

1. Login to Endor Labs as an administrator.
2. Under Manage, navigate to Access Control > Auth Policy
3. Click Add Auth Policy
4. Under Identity Provider Select AWS role
5. Provide the appropriate permissions for your authorization policy.
6. Under claims use the Key User and the value of the ARN that you fetched in the previous command.
7. Click Save Auth Policy to finalize your keyless authentication setup.

Testing Keyless Authentication with AWS

On the EC2 instance you’ve configured for keyless authentication, download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --aws-role-arn=<insert-your-arn> for federated access to Endor Labs such as in the below example:

endorctl --aws-role-arn=<insert-your-arn> api list -r Project -n <insert-your-endorlabs-tenant> --page-size=1

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication with AWS.

6.2 - Scanning with GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can use GitHub Actions to seamlessly include Endor Labs into your CI pipeline.

Using this pipeline, developers can view and detect:

• Policy violations in the source code
• Secrets inadvertently included in the source code

The Endor Labs verifications are conducted as automated checks and help you discover violations before pushing code to the repository. Information about the violations can even be included as comments on the corresponding pull request (PR). This enables developers to easily identify issues and take remedial measures early in the development life cycle.

• For policy violations, the workflow is designed to either emit a warning or return an error based on your action policy configurations.
• For secrets discovered in the commits, developers can view the PR comments and take necessary remedial measures.

To start using Endor Labs with GitHub:

• Install Software Pre-requisites
• Example GitHub Action Workflow
• Authenticate with Endor Labs
  • Authentication Without Keyless Authentication for GitHub
  • Enable PR Comments
• Configure Endor Labs Action Policies
• View PR Comments
• Endor Labs GitHub Action Configuration Parameters
  • Common parameters
  • Scanning parameters

Install Software Pre-requisites

To ensure the successful execution of the Endor Labs GitHub action, the following pre-requisites must be met:

• The GitHub action must be able to authenticate with the Endor Labs API.
• You must have the value of the Endor Labs namespace handy for authentication.
• You must have access to the Endor Labs API.
• If you are using keyless authentication, you must set an authorization policy in Endor Labs. See Authorization policies for details.

Example GitHub Action Workflow

Here is an example Endor Labs scanning workflow using GitHub actions that accomplishes the following tasks in your CI environment:

• Tests PRs to the default branch and monitors the most recent push to the default branch.
• Builds a Java project and sets up the Java build tools. If your project is not on Java, then configure this workflow with your project-specific steps and build tools.
• Authenticates to Endor Labs with GitHub Actions keyless authentication.
• Scan with Endor Labs.
• Comments on PRs if any policy violations occur.
• Generates findings and uploads results to GitHub in SARIF format.

Below is an example workflow to scan with Endor Labs for a Java application using the recommended keyless authentication for GitHub actions:

name: Endor Labs Dependency and Secrets Scan
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload Sarif artifact to GitHub
      contents: read # Used to check out a private repository
      actions: read # Required for private repositories to upload Sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication with Endor Labs
      issues: write # Required to automatically comment on PRs for new policy violations
      pull-requests: write # Required to automatically comment on PRs for new policy violations
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: Endor Labs Scan Pull Request
      if: github.event_name == 'pull_request'
      uses: endorlabs/github-action@v1.1.2
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: true
        enable_pr_comments: ${{ env.ENDORCTL_PR }}
        github_token: ${{ secrets.GITHUB_TOKEN }} # Required for PR comments on new policy violations

  scan-main:
    permissions:
      id-token: write
      repository-projects: read
      pull-requests: read
      contents: read
    name: endorctl-scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: 'Endor Labs Scan Push to main'
      if: ${{ github.event_name == 'push' }}
      uses: endorlabs/github-action@v1.1.2
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: false
        scan_summary_output_type: 'table'
        sarif_file: 'findings.sarif'
    - name: Upload findings to github
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'findings.sarif'

Authenticate with Endor Labs

Endor Labs recommends using keyless authentication in CI environments. Keyless authentication is more secure and reduces the cost of secret rotation. To set up keyless authentication see Keyless Authentication.

If you choose not to use keyless authentication, you can configure an API key and secret in GitHub for authentication as outlined in Managing API keys.

Authentication Without Keyless Authentication for GitHub

If you are not using keyless authentication for GitHub Actions, you must not provide id-token: write permissions to your GitHub token unless specifically required by a step in this job. You must also set enable_github_action_token: false in your Endor Labs GitHub action configuration.

The following example configuration uses the Endor Labs API key for authentication:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.2
        with:
          namespace: 'example'
          api_key: ${{ secrets.ENDOR_API_CREDENTIALS_KEY }}
          api_secret: ${{ secrets.ENDOR_API_CREDENTIALS_SECRET }}
          enable_github_action_token: false

The following example configuration uses a GCP service account for keyless authentication to Endor Labs:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.2
        with:
          namespace: 'example'
          gcp_service_account: '<Insert_Your_Service_Account>@<Insert_Your_Project>.iam.gserviceaccount.com'
          enable_github_action_token: false

Enable PR Comments

Make sure that your GitHub action workflow includes the following:

• The workflow must have a with clause including: enable_pr_comments to true to publish new findings as review comments and github_token: ${{ secrets.GITHUB_TOKEN }}. This token is automatically provisioned by GitHub when using GitHub actions. See GitHub configuration parameters for more information.
• To grant Endor Labs the ability to comment on PRs you must include the permissions issues: write and pull-requests: write.

The following example configuration comments on PRs if a policy violation is detected.

      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.2
        with:
          namespace: 'example' # Update with your Endor Labs namespace
          scan_summary_output_type: 'table'
          scan_dependencies: true
          scan_secrets: true
          pr: true
          enable_pr_comments: ${{ env.ENDORCTL_PR }}
          github_token: ${{ secrets.GITHUB_TOKEN }}

Configure Endor Labs Action Policies

Configure an action policy in the Endor Labs UI to perform an action when a rule is triggered. See Action Policies for details on action policies.

• Set the Policy Template to Detected Secrets and select the Template Parameters as desired.
• Choose Enforce Policy and
  • Select Warn as the recommended action.
  • Select Break the Build to fail the build CI pipeline.

View PR Comments

You can now proceed to check out the code, install the build toolchain and build the code. When you raise a PR, Endor Labs will scan and detect any policy violations and add PR comments accordingly. The CI pipeline will either warn you or fail the build based on your action policy configuration. The PR comments will also include recommendations to help you take necessary remedial actions.

Endor Labs GitHub Action Configuration Parameters

The following input configuration parameters are supported for the Endor Labs GitHub Action:

Common parameters

The following input global parameters are supported for the Endor Labs GitHub action:

Scanning parameters

The following input parameters are also supported for the Endor Labs GitHub action when used for scanning:

6.3 - Scanning in GitLab Pipelines

GitLab CI/CD pipelines are a part of GitLab’s integrated continuous integration and deployment features. They allow you to define and automate the different stages and tasks in your software development workflow.

This documentation page provides an overview and example job to integrate Endor Labs into your GitLab CI pipeline.

High Level Usage Steps

1. Setup authentication to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authentication to Endor Labs

Endor Labs reccomends using keyless authentication in continuous integration environments.

Keyless Authentication is more secure and reduces the cost of secret rotation.

To setup keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in GitLab for authentication using the following steps. See managing API keys for more information on getting an API key for Endor Labs authentication

1. In your GitLab environment, select the project you want to scan.
2. Go to Settings > CI/CD.
3. Click Expand in the Variables section.
4. Click the Add variable button at the bottom of the section.
5. In the Key field, enter ENDOR_API_CREDENTIALS_SECRET.
6. In the Value field, enter your Endor Labs API secret.
7. Under Flags, make sure you select Mask variable.
8. Repeat the previous steps to add your API key as the variable ENDOR_API_CREDENTIALS_KEY.

Configuring your GitLab CI pipeline

1. Create a .gitlab-ci.yml file in the root directory of your project if you do not already have one.
2. In your .gitlab-ci.yml file customize the job configuration based on your project’s requirements using the example below.
3. Modify the image field to align with the build tools necessary for building your software packages.
4. Update the before_script section to include any additional steps required before executing the scan, such as installing dependencies or building your project.
5. Save and commit the .gitlab-ci.yml file to your GitLab repository.
6. GitLab will automatically detect the .gitlab-ci.yml file and trigger the defined job whenever there are changes pushed to the repository.
7. Monitor the progress and results of the CI pipeline in the GitLab CI/CD interface.

Below is an example job to get you started. Please ensure to customize this job with your specific build environment and build steps as needed.

# You can copy and paste this template into a new `.gitlab-ci.yml` file.
# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword.
#
stages:
  - Scan
Endor Labs Dependency Scan:
  stage: test
  image: node # Modify this image to align with the build tools nessesary to build your software packages
  dependencies: []
  variables:
   ## Scan scoping section
   #
   ## Use the following environment variables for custom paths, inclusions and exclusions.
   # ENDOR_SCAN_PATH: "Insert a custom path to your git repository. Defaults to your pwd"
   # ENDOR_SCAN_EXCLUDE: "Insert a regexp style pattern of paths to exclude in the scan. Generally used for monorepos."
   # ENDORCTL_SCAN_INCLUDE: "Insert a regexp style pattern of paths to include in the scan. Generally used for monorepos."
   #
   ## Authentication to Endor Labs
   #
   ## Use the following environment variables for keyless authentication with your cloud provider. For more information visit: https://docs.endorlabs.com/continuous-integration/keyless-authentication/
   #
   # ENDOR_GCP_CREDENTIALS_SERVICE_ACCOUNT: "endorlabs@<yourproject.iam.gserviceaccount.com"
   # ENDOR_AWS_CREDENTIALS_ROLE_ARN: "arn:aws:iam::123456789012:role/my-role"
   #
   ## Follow the below instructions to use an API key and secret instead of keyless authentication.
   ## In your GitLab environment, select the project you want to scan.
   ## Go to `Settings` > `CI/CD`.
   ## Click `Expand` in the `Variables` section.
   ## Click the `Add variable` button at the bottom of the section.
   ## In the `Key` field, enter ENDOR_API_CREDENTIALS_SECRET.
   ## In the `Value` field, enter your Endor Labs API secret.
   ## Under `Flags` make sure you select `Mask variable`.
   ## Repeat to add your API key as the variable ENDOR_API_CREDENTIALS_KEY
   #
    ENDOR_ENABLED: "true"
    ENDOR_ALLOW_FAILURE: "false"
    ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    ENDOR_PROJECT_DIR: "."
    ENDOR_ARGS: |
      --path=${ENDOR_PROJECT_DIR}
      --detached-ref-name=$CI_COMMIT_REF_NAME
      --output-type=summary
      --exit-on-policy-warning
      --dependencies --secrets --git-logs      
  before_script:
    - npm install yarn # Replace with the build steps for your Endor Labs job.
  script:
    - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl;
    - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
      if [ $? -ne 0 ]; then
       echo "Integrity check failed";
       exit 1;
      fi
    - chmod +x ./endorctl
    - if [ "$DEBUG" == "true" ]; then
        export ENDOR_LOG_VERBOSE=true;
        export ENDOR_LOG_LEVEL=debug;
      fi
    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi
    - ./endorctl scan ${ENDOR_ARGS}
  rules:
  - if: $ENDOR_ENABLED != "true"
    when: never
  - if: $ENDOR_ALLOW_FAILURE == "true"
    allow_failure: true
  - if: $ENDOR_ALLOW_FAILURE != "true"
    allow_failure: false

6.4 - Scanning with CircleCI

CircleCI CI/CD pipelines allow you to configure your pipeline as code. Your entire CI/CD process is orchestrated through a single file called config.yml. The config.yml file is located in a folder called .circleci at the root of your project which defines the entire pipeline.

To integrate Endor Labs into your CircleCI CI/CD processes:

1. Authenticate to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authenticate to Endor Labs

Endor Labs recommends using keyless authentication in continuous integration environments. Keyless Authentication is more secure and reduces the cost of secret rotation but is only available on self-hosted runners in CircleCI.

To configure keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in CircleCI for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.

1. In your CircleCI environment, navigate to Organizational Settings.
2. From Contexts and select Create Context.
3. Enter a context name for reference such as endorlabs or re-use an existing context.
4. Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
5. In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
6. Select Add Environment Variable.
7. Repeat the previous 3 steps to add your API key secret as the environment variable ENDOR_API_CREDENTIALS_SECRET. Have the name of the context handy to reference in the workflows later.

Configure your CircleCI pipeline

To create a CircleCI pipeline reference the following steps:

1. Create a .cirlceci/config.yml file in your repository if you do not already have one.
2. In your config.yml file customize the job configuration based on your project’s requirements using one of the examples, simple CircleCI configuration or advanced CircleCI configuration.
3. Create two workflows called build_and_watch_endorlabs and build_and_test_endorlabs.
4. Ensure that the context you created is part of the workflow if you are not using keyless authentication.
5. Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
6. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
7. Update your default branch from main if you do not use main as the default branch name.
8. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Examples

Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.

Simple CircleCI configuration

version: 2.1

jobs:
  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            
  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - watch-endorlabs-scan:
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - test-endorlabs-scan:
          context:
            - endorlabs

Advanced CircleCI configuration

The following example is an advanced implementation of Endor Labs in circleCI which includes several optional performance optimizations and job maintainability updates.

This includes:

1. Caching and restoring caches of jobs and artifacts to improve performance. Caches should be modified to reflect the build artifacts and dependencies of your project.
2. Segmenting jobs and scans.

# You can copy and paste portions of this `config.yml` file as an easy reference.
#
version: 2.1

jobs:
  build:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build steps
    steps:
      - checkout
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Build Your Project"
          command: |
            mvn clean install            
      - persist_to_workspace:
          root: .
          paths:
            - target/ # Persist artifact across job. Change this if you are creating your artifact in a location outside of the target directory.
      - save_cache:
          paths:
            - ~/.m2/repository
          key: maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}

  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            

  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example" #Replace with your namespace in Endor Labs
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - build
      - watch-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - build
      - test-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs

Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.

6.5 - Scanning with Jenkins

Jenkins is an open-source automation server widely used for building, testing, and deploying software. Specifically in the context of CI/CD pipelines, Jenkins serves as a powerful tool to automate various stages of the software development lifecycle.

To integrate Endor Labs into your Jenkins CI/CD processes:

1. Authenticate to Endor Labs
2. Install nodeJS plugin in Jenkins.
3. Install your build toolchain
4. Build your code
5. Scan with Endor Labs

Authenticate to Endor Labs

To configure keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in Jenkins for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.

1. In your Jenkins environment, navigate to Manage Jenkins.
2. Enter a credential name for reference such as endorlabs or re-use an existing context.
3. Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
4. In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
5. Select Add Environment Variable.

Install nodeJS plugin in Jenkins

See Jenkins documentation to install nodeJS plugin in Jenkins. You must have the nodeJS plugin to use npm and download endorctl.

Configure your Jenkins pipeline

To create a Jenkins pipeline:

1. Create a configuration pipeline file in your repository if you do not already have one using the pipeline project.
2. In your configuration pipeline file customize the job configuration based on your project’s requirements using one of the examples, simple Jenkins configuration or Jenkins pipeline using curl.
3. Ensure that the context you created is part of the workflow if you are not using keyless authentication.
4. Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
5. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
6. Update your default branch from main if you do not use main as the default branch name.
7. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Examples

Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.

Simple Jenkins configuration using npm

pipeline {
    agent any
    tools {nodejs "NodeJS"}
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY_1')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET_1')
    }
    stages {
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                script {
                    // Define the Node.js installation name configured in Jenkins
                    NODEJS_HOME = tool name: 'NodeJS', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
                    PATH = "$NODEJS_HOME/bin:${env.PATH}"
                }

                // Download and install endorctl.
                sh 'npm install -g endorctl'
                // Check endorctl version and installation.
                sh 'endorctl --version'
                // Run the scan.
                sh('endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET')

            }
        }

        stage('Results') {
            steps {
                // Publish or process the vulnerability scan results
                // Publish reports, fail the build on vulnerabilities, etc.
                echo 'Publish results'
            }
        }
    }

}

Jenkins pipe line for curl to download endorctl binary

The following example includes curl to download the endorctl binary.

pipeline {
    agent any

    // Endorctl scan uses following environment variables to the trigger endorctl scan
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET')
    }
    stages {
        // Not required if repository is allready cloned to trigger a endorctl scan
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            // Not required if project is already built
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                // Download and install endorctl.
                sh '''#!/bin/bash
                    echo "Downloading latest version of endorctl"
                    VERSION=$(curl $ENDOR_API/meta/version | jq -r '.Service.Version')
                    ENDORCTL_SHA=$(curl $ENDOR_API/meta/version | jq -r '.ClientChecksums.ARCH_TYPE_LINUX_AMD64')
                    curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
                    echo "$ENDORCTL_SHA  endorctl" | sha256sum -c
                    if [ $? -ne 0 ]; then
                        echo "Integrity check failed"
                        exit 1
                    fi
                    chmod +x ./endorctl
                    // Check endorctl version and installation.
                    ./endorctl --version
                    // Run the scan.
                    ./endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
                '''
            }
        }
    }
}