Microservice Discovery in Your CI/CD Pipeline

🚧

Early Adopter Release

This feature is currently in early adopter release and may not be available to all customers.

Overview

You can set up microservice discovery by adopting the configuration-as-code approach. This involves using a configuration file, often referred to as a manifest file, to define the characteristics and metadata of each microservice. The manifest file is stored in a version-controlled repository, typically alongside the source code of the microservice.

The process of discovering microservices through a manifest file ensures that any modifications made to this file in your repository are mirrored in SAP LeanIX, keeping your inventory up to date. The manifest file serves as a single source of truth, ensuring all microservices along with their details remain current.

The configuration-as-code approach for microservice discovery offers several key benefits:

• Alignment with established practices: This approach is in line with established practices within the software development field, making it intuitive for developers and reducing the learning curve.
• In-context contribution: Technical stakeholders, including developers, can contribute to the architectural design within their usual development environment.
• Collaboration: This approach fosters collaboration between enterprise architects and developers, leading to more efficient and cohesive development processes.
• Up-to-date inventory: Changes made to a manifest file are automatically reflected in SAP LeanIX, ensuring your microservice fact sheets are always up to date.

This guide provides detailed instructions on how to set up microservice discovery using a YAML manifest file and a Python script.

📘

Note

We're working on an out-of-the-box integration with GitHub Enterprise for microservice discovery. If your organization uses GitHub Enterprise for source code hosting, you can take advantage of this integration instead of setting up microservice discovery manually, as described in this guide.

Prerequisites

Do the following:

• Get access to your repository and ensure that you can modify its Continuous Integration and Continuous Deployment (CI/CD) pipeline.
• In SAP LeanIX, get an API token by creating a technical user with admin permissions. For details, see Create a Technical User.

This guide assumes you have a basic understanding of:

• CI/CD concepts and practices
• YAML
• Technology Discovery API
• SAP LeanIX Technology Risk and Compliance product, including its technology standards management capabilities

Step 1: Create a Manifest File

To start your microservice discovery, first generate a valid manifest file (leanix.yaml) in your repository. Within this file, specify the details of your microservice.

To view the manifest file and schema, refer to Manifest File and Schema.

📘

Note

A manifest file can only describe one microservice. Each repository can contain multiple manifest files.

For monorepos, we recommend storing one manifest file per microservice within the monorepo.

Step 2: Commit the Manifest File

Establish a link with your microservice fact sheet in SAP LeanIX by committing the manifest file to the default branch of your repository, such as main or master.

📘

Note

You can link multiple microservices to microservice fact sheets in SAP LeanIX from the same repository by placing the manifest files in separate subfolders within your repository structure. This approach is especially useful for monorepos.

Step 3: Generate a Software Bill of Materials (SBOM)

SBOM files provide a comprehensive inventory of all software components, libraries, and modules used by microservices. They help ensure license compliance, identify potential security risks, and offer a deeper understanding of your software supply chain. For more information, see Software Bill of Materials (SBOM).

SAP LeanIX does not offer capabilities for automatic SBOM generation. For more information on SBOM requirements and generation methods, see Generating SBOMs.

Step 4: Upload the Manifest File

Before you proceed with the configuration of your CI/CD workflow, it's crucial to create a script that will upload the manifest file. This script extracts and gathers the necessary information needed to formulate the API request. This is an essential step in the process as it ensures that all relevant data from your microservices is accurately captured and ready for the next phase.

To assist you in this process, we provide the following example script that shows how to upload the manifest file. This script serves as a reference point, helping you understand the process and showcasing the kind of functionality your script should include.

📘

Note

While the SBOM ingestion can technically be executed as a standalone step, it is intrinsically linked to the microservice discovery. Therefore, we highly advise integrating these two processes for optimal results.

To ensure the successful execution of our script, it's essential to configure certain secrets and variables on your repository provider. These configurations are crucial for authenticating and directing the script to the correct SAP LeanIX workspace. For detailed instructions on how to configure variables, please refer to the documentation of your repository provider. The following variables are used in the example script:

• LEANIX_API_TOKEN (Secret): Your SAP LeanIX API token, which serves as your authentication key. To learn how to get an API token, see Create a Technical User.

• LEANIX_SUBDOMAIN (Variable): Your SAP LeanIX subdomain, which is used to direct the script to your specific SAP LeanIX workspace. You can copy your subdomain value from the workspace URL. For more information, see Base URL.

Example script:

import logging
from pathlib import Path
import yaml
import requests
import json
import os

logging.basicConfig(level=logging.INFO)

# Request timeout
TIMEOUT = 20

# API token and Subdomain are set as env variables.
# It is adviced not to hard code sensitive information in your code.
LEANIX_API_TOKEN = os.getenv("LEANIX_API_TOKEN")
LEANIX_SUBDOMAIN = os.getenv("LEANIX_SUBDOMAIN")
LEANIX_FQDN = f"https://{LEANIX_SUBDOMAIN}.leanix.net/services"

# OAuth2 URL to request the access token.
LEANIX_OAUTH2_URL = f"{LEANIX_FQDN}/mtm/v1/oauth2/token"

# Manifest API
MANIFEST_API = f"{LEANIX_FQDN}/technology-discovery/v1/manifests"

# Github Related
GITHUB_SERVER_URL = os.getenv("GITHUB_SERVER_URL")
GITHUB_REPOSITORY = os.getenv("GITHUB_REPOSITORY")

# Manifest file and SBOM file
LEANIX_MANIFEST_FILE = os.getenv("LEANIX_MANIFEST_FILE", "leanix.yaml")
SBOM_FILE = os.getenv("SBOM_FILE", "sbom.json")


def _ensure_file(file: Path):
    """Ensures that the provided file exists and is a file.

    Args:
        file (Path): The path to the file.

    Raises:
        FileNotFoundError: If the file does not exist or is not a file.
    """
    if not (file.exists() and file.is_file()):
        raise FileNotFoundError(f"File {file} not found")


def _obtain_access_token() -> str:
    """Obtains a LeanIX Access token using the Technical User generated
    API secret.

    Returns:
        str: The LeanIX Access Token
    """
    if not LEANIX_API_TOKEN:
        raise Exception("A valid token is required")
    response = requests.post(
        LEANIX_OAUTH2_URL,
        auth=("apitoken", LEANIX_API_TOKEN),
        data={"grant_type": "client_credentials"},
    )
    response.raise_for_status()
    return response.json().get("access_token")


def _prepare_auth() -> dict:
    """
    Prepares the headers for a GraphQL request to the LeanIX API.

    This function fetches the LeanIX access token from the environment variables and constructs
    the authorization header required for making authenticated requests to the LeanIX API.

        dict: A dictionary containing the authorization header with the access token.
    """
    # Fetch the access token and set the Authorization Header
    auth_header = f'Bearer {os.environ.get("LEANIX_ACCESS_TOKEN")}'
    # Provide the headers
    headers = {
        "Authorization": auth_header,
    }
    return headers


def create_or_update_micro_services(manifest_file: Path):
    """
    Creates or updates the LeanIX Microservice Fact Sheet based on the provided manifest file.

    This function checks if a microservice with the given external ID exists. If it does, the microservice is updated.
    If it does not exist, a new microservice is created. After the microservice is created or updated,
    the function triggers the registration of the relevant SBOM file with LeanIX.

    Args:
        manifest_file (Path): The SAP LeanIX manifest file.
    """
    logging.info(
        f"Processing manifest file: {manifest_file.name}"
    )

    # NOTE: application/yaml here does not mean the content type, but the type of the file.
    request_payload = {
        "file": (
            manifest_file.name,
            manifest_file.open("rb"),
            "application/yaml",
        )
    }
    auth = _prepare_auth()
    resp = requests.put(
        url=MANIFEST_API,
        headers=auth,
        files=request_payload,
        timeout=TIMEOUT,
    )
    resp.raise_for_status()
    logging.info(f"Successfully uploaded manifest file: {manifest_file.name}")
    factsheet_id = resp.json().get("data").get("factSheetId")
    if not factsheet_id:
        raise Exception("Service did not return a fact sheet ID")
    register_sboms(factsheet_id)


def register_sboms(factsheet_id: str):
    """
    Registers the Software Bill of Materials (SBOM) file with LeanIX.

    This function enables improved understanding of the dependency landscape of your microservices.
    The SBOM provides comprehensive details about software components, their relationships, and
    attributes, which are crucial for managing, securing, and licensing your open-source software.
    By registering the SBOM with LeanIX, these details can be effectively managed and tracked.

    Args:
        factsheet_id (str): The unique identifier of the microservice fact sheet. This ID is used
        to associate the SBOM with the corresponding microservice in LeanIX.

    Returns:
        None
    """
    sbom_path = Path(SBOM_FILE)
    # NOTE: If SBOMs are mandatory for your organization, modify this to raise an exception.
    try:
        _ensure_file(sbom_path)
    except FileNotFoundError:
        logging.warning("No sbom file found")
        return

    sbom_endpoint = f"{LEANIX_FQDN}/technology-discovery/v1/factSheets/{factsheet_id}/sboms"
    sbom_contents = dict()
    logging.info(
        f"Processing sbom file: {sbom_path.name} for Fact Sheet: {factsheet_id}"
    )
    with sbom_path.open("rb") as f:
        sbom_contents = f.read()

    # NOTE: application/json here does not mean the content type, but the type of the file.
    request_payload = {
        "sbom": (
            sbom_path.name,
            sbom_contents,
            "application/json",
        )
    }
    logging.debug(f"Populated payload for SBOM: {sbom_path.name}")
    # Fetch the access token and set the Authorization Header
    auth_header = _prepare_auth()
    # NOTE: Don't set the content type, `requests` should handle this.
    logging.info(f"Sending SBOM ingestion request for Fact Sheet: {factsheet_id}")
    response = requests.post(
        sbom_endpoint, headers=auth_header, files=request_payload, timeout=TIMEOUT
    )
    response.raise_for_status()
    logging.info(f"Successfully submited SBOM request for Fact Sheet: {factsheet_id}")


def main():
    """LeanIX helper to parse the manifest file create or update a microservice
    and register the relevant dependencies.
    """
    manifest_file = Path(LEANIX_MANIFEST_FILE)
    _ensure_file(manifest_file)
    create_or_update_micro_services(manifest_file)


if __name__ == "__main__":
    # Set the access token as an environment variable
    os.environ["LEANIX_ACCESS_TOKEN"] = _obtain_access_token()
    main()

Step 5: Configure Your CI/CD Workflow

To complete the discovery and update process of your microservices, you need to establish a CI/CD workflow within your repository. This workflow will ensure that the system automatically detects and incorporates any changes made to your microservices, significantly reducing manual intervention and potential errors.

While it's possible to register microservices through API requests in any environment without a CI/CD pipeline, we strongly recommend using the CI/CD approach. The primary reason for this is to ensure that your data is always up to date. With this approach, your manifest file serves as the source of truth, and any changes made to it are automatically reflected in your microservices.

The CI/CD workflow setup may differ depending on your repository provider. The following table provides examples of how to establish a CI/CD workflow with various repository providers:

📘

Note

The CI/CD approach is designed with flexibility in mind. If you decide to migrate to a Git integration down the line, the current setup effortlessly transitions. Your existing manifest file will seamlessly synchronize with the microservice fact sheet, eliminating the need for additional configuration or rework.

Best Practices

Software deployment typically involves multiple environments, such as testing, staging, and production. Since SAP LeanIX focuses on enterprise architecture management, we recommend cataloging only services deployed to the production environment. This approach aligns with our goal of providing a comprehensive view of the software infrastructure to support strategic decision-making.

For technical use cases that need visibility across all environments, specialized tools are available in the market.