Microservice Discovery Through a Manifest File

Register your microservices using the SAP LeanIX YAML manifest file.

🚧

Early Adopter Release

This feature is currently in early adopter release and may not be available to all users.

Overview

The Configuration-as-Code feature in SAP LeanIX provides a robust solution that allows you to register microservices directly from a YAML manifest file that is committed to the default branch of your Git repository. This approach forms a bridge between enterprise architecture and software development, promoting collaboration.

The process of discovering microservices through a manifest file ensures that any modifications made to the YAML file in your repository are mirrored in SAP LeanIX, keeping your enterprise architecture up to date. This method also enables technical stakeholders, including developers, to contribute to the architecture without stepping out of their development context.

Prerequisites

Before you start, do the following:

  • Ensure that you have access to your repository and can modify its Continuous Integration and Continuous Deployment (CI/CD) pipeline.
  • Get admin access to your SAP LeanIX workspace.
  • Obtain an API token by creating a Technical User with admin permissions. For more information, see Create a Technical User.

This guide assumes you have basic knowledge of:

Step 1: Create a Manifest File

To initiate your microservice discovery through Configuration-as-Code, the first step is to generate a valid manifest file (leanix.yml) in your repository. Within this file, you need to specify the details of your microservice.

The following code snippet provides an example SAP LeanIX manifest file.

version: 1
metadata:
  name: disputes-service-v1
  externalId: disputes-service-v1
  description: |
    A microservice responsible for payment disputes.
    This service handles payment transaction disputes and is an integral part of our payment ecosystem.
  type: Backend  
  repository:
    url: https://example.com
    path: /path/to/repo
    status: active
    visibility: private

  applications:
    - factSheetId: fa787383-7233-4896-8fad-c1f1bef30dd2

  tags:
    - tagGroupName: Domain
      tagNames:
        - Payments

    - tagGroupName: Location
      tagNames:
        - AWS-EU1
        - AWS-EU2

  teams:
    - factSheetId: 63eda74c-57f7-4768-b1c9-3b2813b11504
    - factSheetId: afd1ee0f-095b-4c68-88f6-d3628070ce18

  resources:
    - name: Disputes Process Flow
      type: documentation
      url: https://myorg.atlassian.net/wiki/spaces/disputes
      description: Disputes process flow and diagrams

📘

Note

A manifest file can only describe one microservice. Each repository can contain multiple manifest files.

For monorepos, we recommend to store one manifest file per microservice within the monorepo.

Manifest Schema

The following sections provide details on the attributes supported by the manifest file.

Version

The version of the manifest file schema. Currently, only version 1 is supported.

Metadata

The metadata section of the manifest file is dedicated to defining the characteristics of your microservice. The following table lists attributes to be provided in this section:

AttributeRequiredDescription
nameYesThe name of the microservice.
externalIdYesThe unique identifier of the microservice in the external system. It must be globally unique, ensuring that multiple microservices don't share the same external ID.
descriptionNoThe description of the microservice.
typeNoThe type of the microservice. It must be one of the values of the lxMicroserviceType field predefined in the meta model configuration, such as backend, ui, or data. If no type is passed, the default value backend is used.
repositoryNoThe details of the repository where the microservice is hosted, including url, path, status, and visibility.

We have a set of recommendations for formatting service names, especially for popular vendors. This uniform formatting helps in maintaining consistency and makes it easier to manage and locate services. The out-of-the-box Git integrations will follow this external ID schema. If you follow this convention, we don't expect any migration effort once we support that Git system.

ProviderFormatExample
GitHub{organization}/{repo-name}/{servicename}acme/banking-portal/payment-engine
GitLab{group}/{repo-name}/{servicename}acme/banking-portal/payment-engine
Bitbucket{workspace}/{repo-name}/{servicename}acme/banking-portal/payment-engine
Azure{organization}/{repo-name}/{servicename}acme/banking-portal/payment-engine

Applications

Microservices play a pivotal role in supporting business applications. They are essentially small, independent services that work together to run a complex application. Each microservice is a self-contained unit that performs a unique function within the broader application ecosystem.

The Applications section is necessary to establish the correct association between the microservice and the corresponding business applications.
Microservices can be linked to multiple business applications, as such you can provide multiple different business application fact sheets.

AttributeRequiredDescription
factSheetIdYesA list of business application fact sheet ids
nameYesA list of the business application fact sheet names (corresponds to the display name of the fact sheet)

📘

Note

If you supply both factSheetId and name attributes, factSheetId will take precedence.

Tags

The Tags section allows you to assign tag groups and tags to your microservice.

AttributeRequiredDescription
tagGroupNameNoThe name of the tag group to assign to the microservice
tagNamesNoA list of tags from the tag group to assign to the microservice

Teams

The Teams section is where you can define the teams that own the microservice.

AttributeRequiredDescription
factSheetIdYesA list of the team fact sheet ids of the owning teams
nameYesA list of the team fact sheet names (corresponds to the display name of the fact sheet)

📘

Note

If you supply both factSheetId and name attributes, factSheetId will take precedence.

Resources

In the resources section of the manifest file, you can define the resources associated with your microservice. For more information, see Store Resources on Fact Sheets. The following table lists attributes to be provided in this section:

AttributeRequiredDescription
nameYesThe name of the resource
typeYesThe type of resource. You can find a list of the supported resource types in the API documentation
urlNoA URL for the resource
descriptionNoA brief description of the specific resource

Repository

The repository section within the metadata part of the manifest file provides details about the repository where the microservice is hosted. The following table lists the attributes to be provided in this section:

AttributeRequiredDescription
urlYesThe URL of the repository where the microservice code is hosted.
pathNoThe path within the repository where the microservice code is located.
statusNoThe status of the repository, indicating whether it's ACTIVE or ARCHIVED.
visibilityNoThe visibility of the repository, specifying whether it's INTERNAL, PRIVATE, or PUBLIC.

📘

Note

Ensure that the tag groups, applications, and teams that you want to associate with your microservice exist within your SAP LeanIX workspace.

Step 2: Commit the Manifest File

Establish a link between your microservice fact sheet by committing the manifest file to the default branch of your repository, such as main or master.

📘

Note

You can link multiple microservices to SAP LeanIX microservice fact sheets from the same repository by placing the manifest files in separate subfolders within your repository structure. This is particularly useful if you have a monorepo.

Step 3: Create a Software Bill of Materials (SBOM)

SBOM files offer a detailed inventory of all software components, libraries, and modules used by a microservice. Using the Software Package Data Exchange (SPDX) format, you can efficiently track and manage these components. This enables you to ensure license compliance, identify potential security risks, and gain a deeper understanding of your software supply chain.

Incorporating the creation of a SBOM file into your CI/CD process can further automate the identification of software components within your microservice ecosystem. To learn how to integrate the generation of SBOM into your pipeline, see Generating a Software Bill of Materials (SBOM).

Step 4: Parse the Manifest File

Before you proceed with the configuration of your CI/CD workflow, it's crucial to create a script that will parse the manifest file. This script extracts and gathers the necessary information needed to formulate the API request. This is an essential step in the process as it ensures that all relevant data from your microservices is accurately captured and ready for the next phase.

To assist you in this process, we provide the following example script that shows how to parse the manifest file. This script serves as a reference point, helping you understand the process and showcasing the kind of functionality your script should include.

import logging
from pathlib import Path
import yaml
import requests
import json
import os

logging.basicConfig(level=logging.INFO)

# Request timeout
TIMEOUT = 20

# API token and Subdomain are set as env variables.
# It is adviced not to hard code sensitive information in your code.
LEANIX_API_TOKEN = os.getenv("LEANIX_API_TOKEN")
LEANIX_SUBDOMAIN = os.getenv("LEANIX_SUBDOMAIN")
LEANIX_FQDN = f"https://{LEANIX_SUBDOMAIN}.leanix.net/services"
LEANIX_MANIFEST_FILE = os.getenv("LEANIX_MANIFEST_FILE", "leanix.yaml")

# OAuth2 URL to request the access token.
LEANIX_OAUTH2_URL = f"{LEANIX_FQDN}/mtm/v1/oauth2/token"

# GraphQL API
LEANIX_GRAPHQL_API = f"{LEANIX_FQDN}/pathfinder/v1/graphql"

# SBOM API endpoint
LEANIX_SBOM_API = f"{LEANIX_FQDN}/technology-discovery/v1/microservices"

# Github Related
GITHUB_SERVER_URL = os.getenv("GITHUB_SERVER_URL")
GITHUB_REPOSITORY = os.getenv("GITHUB_REPOSITORY")


def obtain_access_token() -> str:
    """Obtains a LeanIX Access token using the Technical User generated
    API secret.

    Returns:
        str: The LeanIX Access Token
    """
    if not LEANIX_API_TOKEN:
        raise Exception("A valid token is required")
    response = requests.post(
        LEANIX_OAUTH2_URL,
        auth=("apitoken", LEANIX_API_TOKEN),
        data={"grant_type": "client_credentials"},
    )
    response.raise_for_status()
    return response.json().get("access_token")


def _parse_manifest_file() -> dict:
    """Parses the Manifest file and generates the payload for the
    API request for the LeanIX Microservices API.

    Returns:
        dict: The payload for the API request
    """
    manifest_data = dict()
    with open(LEANIX_MANIFEST_FILE, "r") as file:
        try:
            logging.info(f"Parsing manifest file: {file.name}")
            manifest_data = yaml.safe_load(file)
        except yaml.YAMLError as exc:
            logging.error(f"Failed to load Manifest file: {exc}")

    if not manifest_data:
        logging.info("No manifest entries found")
        return manifest_data
    manifest_microservice = manifest_data.get("metadata", {})
    return manifest_microservice


def microservice_exists(external_id: str) -> dict:
    """
    This function sends a GraphQL query to check if a microservice with the given external ID exists.
    It makes the request, and returns the microservice's data if it exists, or an empty dictionary if it does not.

    Args:
        external_id (str): The external ID of the microservice to check.

    Returns:
        dict: A dictionary containing the microservice's data if it exists. If the microservice does not exist,
        an empty dictionary is returned.
    """
    query = """
    query CheckMicroservice($externalIds: [String!]) {
        factSheets(externalIds: $externalIds) {
            edges {
                node {
                    ... on Application {
                        id
                        name
                        relApplicationToOwningOrganization {
                            edges {
                                node {
                                    factSheet {
                                        id
                                        name
                                    }
                                }
                            }
                        }
                        tags {
                            id
                            name
                        }
                    }
                }
            }
        }
    }
    """
    leanix_external_id = f"technologyDiscoveryId/{external_id}"
    variables = {"externalIds": [leanix_external_id]}
    resp = make_request(query, variables)
    response_data = resp.json().get("data", {}).get("factSheets", {}).get("edges", [])
    fact_sheet = dict()
    if response_data:
        fact_sheet = response_data[0].get("node", {})
    return fact_sheet


def generate_owning_teams(microservice_manifest: dict) -> list[dict] | list:
    """
    This function generates a list of dictionaries representing the owning teams of a microservice.
    Each dictionary represents an operation to add a new relationship between the microservice and
    its owning team. The operation is defined with fields 'op', 'path', and 'value'. The 'value'
    field contains the fact sheet ID of the owning team in a JSON format.

    Args:
        microservice_manifest (dict): A dictionary containing the microservice manifest data.
        It includes a 'teams' key which is a list of dictionaries, each containing details
        about a team including its 'factSheetId'.

    Returns:
        list[dict]: A list of dictionaries, each representing an operation to add a new relationship
        between the microservice and its owning team. If no teams are found in the manifest data,
        an empty list is returned.
    """
    teams = list()
    for id, team in enumerate(microservice_manifest.get("teams", []), start=1):
        fact_sheet_id = team.get("factSheetId")
        if fact_sheet_id:
            teams.append(
                {
                    "op": "add",
                    "path": f"/relApplicationToOwningOrganization/new_{id}",
                    "value": json.dumps({"factSheetId": fact_sheet_id}),
                }
            )
    return teams


def generate_graphql_payload(microservice_manifest: dict) -> dict:
    """
    This function generates a payload dictionary for a microservice based on the provided manifest data.
    The payload includes key information about the microservice such as its type and external ID,
    which are extracted from the manifest dictionary. This payload can be used to register or update
    the microservice in a system.

    Args:
        microservice_manifest (dict): A dictionary containing the manifest data for the microservice.

    Returns:
        dict: A dictionary containing the payload data for the microservice.
    """
    tags = list()
    for tagGroup in microservice_manifest.get("tags", []):
        for tagName in tagGroup.get("tagNames", []):
            tags.append({"tagName": tagName})

    variables = {
        "patches": [
            {"op": "replace", "path": "/category", "value": "microservice"},
            {
                "op": "replace",
                "path": "/description",
                "value": microservice_manifest.get("description"),
            },
            {
                "op": "replace",
                "path": "/technologyDiscoveryId",
                "value": json.dumps(
                    {"externalId": microservice_manifest.get("externalId")}
                ),
            },
            {
                "op": "replace",
                "path": "/lxRepositoryUrl",
                "value": f"{GITHUB_SERVER_URL}/{GITHUB_REPOSITORY}",
            },
            {"op": "replace", "path": "/lxRepositoryStatus", "value": "active"},
            {"op": "replace", "path": "/lxRepositoryVisibility", "value": "public"},
        ],
    }
    if len(tags):
        variables["patches"].append(
            {"op": "replace", "path": "/tags", "value": json.dumps(tags)}
        )
    return variables


def make_request(query: str, variables: dict | None) -> requests.Response:
    """
    Sends a GraphQL request to the LeanIX API.

    This function sends a POST request to the LeanIX API with the given GraphQL query and variables.
    It uses the LEANIX_ACCESS_TOKEN environment variable for authentication.

    Args:
        query (str): The GraphQL query to send in the request.
        variables (dict, optional): A dictionary containing the variables for the GraphQL query.

    Returns:
        requests.Response: The response from the LeanIX API. If the request fails, an exception is raised.
    """
    url = f"{LEANIX_GRAPHQL_API}"
    # Fetch the access token and set the Authorization Header
    auth_header = f'Bearer {os.environ.get("LEANIX_ACCESS_TOKEN")}'
    # Provide the headers
    headers = {
        "Authorization": auth_header,
    }
    response = requests.post(
        headers=headers, url=url, json={"query": query, "variables": variables}
    )
    response.raise_for_status()
    return response


def create_or_update_micro_services(manifest_payload: dict):
    """
    Creates or updates the LeanIX Microservice Fact Sheet based on the provided manifest file.

    This function checks if a microservice with the given external ID exists. If it does, the microservice is updated.
    If it does not exist, a new microservice is created. After the microservice is created or updated,
    the function triggers the registration of the relevant SBOM file with LeanIX.

    Args:
        manifest_payload (dict): A dictionary containing the manifest data for the microservice.
        This data includes the microservices name and external ID.
    """
    external_id = manifest_payload.get("externalId")
    microservice_name = manifest_payload.get("name")
    fact_sheet_id = None
    fact_sheet = dict()
    if not external_id:
        logging.error("Could not determine external id for microservice")
        return
    fact_sheet = microservice_exists(external_id)
    fact_sheet_id = fact_sheet.get("id")
    if not fact_sheet_id:
        logging.info(f"Microservice {microservice_name} does not exist")
        fact_sheet = create_microservice(manifest_payload)
        if not fact_sheet:
            logging.error(f"Failed to create microservice {microservice_name}")
            return
        fact_sheet_id = fact_sheet.get("id")
        if fact_sheet_id:
            logging.info(
                f"Created microservice {microservice_name}, fact sheet id: {fact_sheet_id}"
            )
    else:
        logging.info(
            f"Microservice {microservice_name} exists, fact sheet id: {fact_sheet_id}"
        )
        fact_sheet = update_microservice(manifest_payload, fact_sheet)
        if not fact_sheet:
            logging.error(f"Failed to update microservice {microservice_name}")
            return
        if fact_sheet:
            logging.info(f"Successfully updated microservice: {microservice_name}")
    if fact_sheet_id:
        register_sboms(fact_sheet_id)


def create_microservice(manifest_data: dict) -> dict:
    """
    This function sends a GraphQL mutation to create a new microservice with the given manifest data.
    It generates the necessary payload, makes the request, and returns the created microservice's data.
    If any errors occur during the creation process, an empty dictionary is returned.

    Args:
        manifest_data (dict): A dictionary containing the manifest data for the microservice.
        This data includes the microservice's name, type, and owning teams' fact sheet IDs.

    Returns:
        dict: A dictionary containing the created microservice's data. If any errors occur during
        the creation process, an empty dictionary is returned.
    """
    query = """
    mutation CreateMicroservice($input: BaseFactSheetInput!, $patches: [Patch]) {
        createFactSheet(input: $input, patches: $patches) {
            factSheet {
                id
                name
                type
                category
                ... on Application {
                    technologyDiscoveryId {
                        externalId
                        externalUrl
                        externalVersion
                    }
                    lxRepositoryUrl,
                    lxRepositoryStatus,
                    lxRepositoryVisibility
                    relApplicationToOwningOrganization {
                        edges {
                            node {
                                factSheet {
                                    name
                                    id
                                }
                            }
                        }
                    }
                    tags {
                        id
                        name
                    }
                }
            }
        }
    }
    """
    variables = generate_graphql_payload(manifest_data)
    variables["input"] = {"name": manifest_data.get("name"), "type": "Application"}
    teams = generate_owning_teams(manifest_data)
    variables["patches"].extend(teams)
    resp = make_request(query, variables)
    errors = resp.json().get("errors", [])
    if errors:
        logging.error(
            f"Errors encountered while creating microservice, errrors: {errors}"
        )
        return {}
    return resp.json().get("data", {}).get("createFactSheet", {}).get("factSheet", {})


def update_microservice(manifest_data: dict, fact_sheet: dict) -> dict:
    """
    This function sends a GraphQL mutation to update a microservice with the given manifest data.
    It generates the necessary payload, makes the request, and returns the updated microservice's data.
    If any errors occur during the update process, an error message is logged and an empty dictionary is returned.

    Args:
        manifest_data (dict): A dictionary containing the manifest data for the microservice.
        This data includes the microservice's name, type, and owning teams' fact sheet IDs.

        fact_sheet (dict): A dictionary containing the current fact sheet data of the microservice to be updated.
        It includes the microservice's ID.

    Returns:
        dict: A dictionary containing the updated microservice's data. If any errors occur during
        the update process, an empty dictionary is returned.
    """
    query = """
    mutation UpdateMicroservice($ID: ID!, $patches: [Patch]!) {
        updateFactSheet(id: $ID, patches: $patches) {
            factSheet {
                id
                name
                type
                category
                ... on Application {
                    technologyDiscoveryId {
                        externalId
                        externalUrl
                        externalVersion
                    }
                    lxRepositoryUrl,
                    lxRepositoryStatus,
                    lxRepositoryVisibility
                    relApplicationToOwningOrganization {
                        edges {
                            node {
                                factSheet {
                                    name
                                    id
                                }
                            }
                        }
                    }
                    tags {
                        id
                        name
                    }
                }
            }
        }
    }
    """
    variables = generate_graphql_payload(manifest_data)
    variables["ID"] = fact_sheet.get("id")
    resp = make_request(query, variables)
    errors = resp.json().get("errors", [])
    if errors:
        logging.error(
            f"Errors encountered while updating microservice: {manifest_data.get('name')}, errors: {errors}"
        )
        return {}
    return resp.json().get("data", {}).get("updateFactSheet", {}).get("factSheet", {})


def register_sboms(factsheet_id: str):
    """
    Registers the Software Bill of Materials (SBOM) file with LeanIX.

    This function enables improved understanding of the dependency landscape of your microservices.
    The SBOM provides comprehensive details about software components, their relationships, and
    attributes, which are crucial for managing, securing, and licensing your open-source software.
    By registering the SBOM with LeanIX, these details can be effectively managed and tracked.

    Args:
        factsheet_id (str): The unique identifier of the microservice fact sheet. This ID is used
        to associate the SBOM with the corresponding microservice in LeanIX.

    Returns:
        None
    """
    sbom_path = Path("sbom.json")
    if not sbom_path.exists():
        logging.warning("No sbom file found")
        return

    url = f"{LEANIX_SBOM_API}/{factsheet_id}/sboms"
    sbom_contents = dict()
    logging.info(
        f"Processing sbom file: {sbom_path.name} for Fact Sheet: {factsheet_id}"
    )
    with sbom_path.open("rb") as f:
        sbom_contents = f.read()

    request_payload = {
        "sbom": (
            sbom_path.name,
            sbom_contents,
            "application/json",
        )
    }
    logging.debug(f"Populated payload for SBOM: {sbom_path.name}")
    # Fetch the access token and set the Authorization Header
    auth_header = f'Bearer {os.environ.get("LEANIX_ACCESS_TOKEN")}'
    # Provide the headers
    # NOTE: Don't set the content type, `requests` should handle this.
    headers = {
        "Authorization": auth_header,
    }
    logging.info(f"Sending sbom ingestion request for Fact Sheet: {factsheet_id}")
    response = requests.post(
        url, headers=headers, files=request_payload, timeout=TIMEOUT
    )
    response.raise_for_status()
    logging.info(f"Successfully submited sbom request for Fact Sheet: {factsheet_id}")


def main():
    """LeanIX helper to parse the manifest file create or update a microservice
    and register the relevant dependencies.
    """
    manifest_data = _parse_manifest_file()
    create_or_update_micro_services(manifest_data)


if __name__ == "__main__":
    # Set the access token as an environment variable
    os.environ["LEANIX_ACCESS_TOKEN"] = obtain_access_token()
    main()

📘

Note

While the SBOM ingestion can technically be executed as a standalone step, it is intrinsically linked to the microservice discovery. Therefore, we highly advise integrating these two processes for optimal results.

Step 5: Set Up Your Secrets and Variables

To ensure the successful execution of our script, it's essential to configure certain secrets and variables on your repository provider. These configurations are crucial for authenticating and directing the script to the correct SAP LeanIX workspace.

  • LEANIX_API_TOKEN (Secret): Your SAP LeanIX API token, which serves as your authentication key. To learn how to get an API token, see Create a Technical User.

  • LEANIX_SUBDOMAIN (Variable): Your SAP LeanIX subdomain, which is used to direct the script to your specific SAP LeanIX workspace. You can copy your subdomain value from the workspace URL. For more information, see Base URL.

📘

Note

For detailed instructions on how to configure variables, please refer to the documentation of your repository provider.

Step 6: Configure Your Continuous Integration and Continuous Deployment

To complete the discovery and update process of your microservices, you need to establish a CI/CD workflow within your repository. This workflow will ensure that the system automatically detects and incorporates any changes made to your microservices, significantly reducing manual intervention and potential errors.

While it's possible to register microservices through API requests in any environment without a CI/CD pipeline, we strongly recommend using the CI/CD approach. The primary reason for this is to ensure that your data is always up to date. With this approach, your manifest file serves as the source of truth, and any changes made to it are automatically reflected in your microservices.

The CI/CD workflow setup may differ depending on your repository provider. The following table provides examples of how to establish a CI/CD workflow with various repository providers:

Repository ProviderAutomation Workflow
Azure DevOpsAzure Pipelines Example
BitbucketBitbucket Pipelines Example
GitHubGitHub Actions Example
GitHub Enterprise ServerGitHub Actions Example
GitLabGitLab Pipelines Example

📘

Note

The CI/CD approach is designed with flexibility in mind. If you decide to migrate to a Git integration down the line, the current setup effortlessly transitions. Your existing manifest file will seamlessly synchronize with the microservice fact sheet, eliminating the need for additional configuration or rework.

Best Practices

Software deployment often spans across several environments, such as testing, staging, and production. However, given that SAP LeanIX is a tool for enterprise architecture management, we advise cataloging only those services that are deployed to the production environment. This is because SAP LeanIX is designed to provide an overarching view of the software infrastructure in support of strategic decision-making.

For more technical use cases requiring visibility across all environments, there are specialized tools available in the market.