Microservice Discovery Through a Manifest File

Register your microservices using the LeanIX YAML manifest file.

🚧

This feature is currently in early adopter release and may not be available to all users.

Overview

The Configuration-as-Code feature in LeanIX provides a robust solution that allows you to register microservices directly from a YAML manifest file that is committed to the default branch of your Git repository. This approach forms a bridge between enterprise architecture and software development, promoting collaboration.

The process of discovering microservices through a manifest file ensures that any modifications made to the YAML file in your repository are mirrored in LeanIX, keeping your enterprise architecture up to date. This method also enables technical stakeholders, including developers, to contribute to the architecture without stepping out of their development context.

Prerequisites

Before you start, do the following:

  • Ensure that you have access to your repository and can modify its Continuous Integration and Continuous Deployment (CI/CD) pipeline.
  • Get admin access to your LeanIX workspace.
  • Obtain an API token by creating a Technical User with admin permissions. For more information, see Create a Technical User.

This guide assumes you have basic knowledge of:

Step 1: Create a Manifest File

To initiate your microservice discovery through Configuration-as-Code, the first step is to generate a valid manifest file (leanix.yml) in your repository. Within this file, you need to specify the details of your microservice.

The following code snippet provides an example LeanIX manifest file.

version: 1
services:
  - name: disputes-service-v1
    externalId: disputes-service-v1
    description: |
      A microservice responsible for payment disputes.
      This service handles payment transaction disputes and is an integral part of our payment ecosystem.

    applications:
      - factSheetId: fa787383-7233-4896-8fad-c1f1bef30dd2
      - factSheetId: ec3cdf45-4678-4df1-acab-866b7cb7c9cf

    tags:
      - tagGroupName: Domain
        tagNames: 
          - Payments

      - tagGroupName: Location
        tagNames:
          - AWS-EU1
          - AWS-EU2
    
    teams:
      - factSheetId: 63eda74c-57f7-4768-b1c9-3b2813b11504
      - factSheetId: afd1ee0f-095b-4c68-88f6-d3628070ce18

    resources:
      - name: Disputes Process Flow
        type: documentation
        url: https://myorg.atlassian.net/wiki/spaces/disputes
        description: Disputes process flow and diagrams

📘

Note

Each repository should contain only one manifest file. A manifest file has the capacity to describe multiple microservices.

Manifest Schema

The following sections provide details on the attributes supported by the manifest file.

Services

The services section of the manifest file is dedicated to defining the characteristics of your microservice. The following table lists attributes to be provided in this section:

AttributeRequiredDescription
nameYesThe name of the microservice
externalIdYesThe externalId is a unique identifier for this microservice within your system. It must be globally unique, ensuring no two microservices share the same external id.
descriptionNoThe description of the microservice

We have a set of recommendations for formatting service names, especially for popular vendors. This uniform formatting helps in maintaining consistency and makes it easier to manage and locate services.

ProviderFormatExample
GitHub{organization}/{repo-name}/{servicename}acme/banking-portal/payment-engine
GitLab{group}/{repo-name}/{servicename}acme/banking-portal/payment-engine
Bitbucket{workspace}/{repo-name}/{servicename}acme/banking-portal/payment-engine
Azure{organization}/{repo-name}/{servicename}acme/banking-portal/payment-engine

Applications

Microservices play a pivotal role in supporting business applications. They are essentially small, independent services that work together to run a complex application. Each microservice is a self-contained unit that performs a unique function within the broader application ecosystem.

The Applications section is necessary to establish the correct association between the microservice and the corresponding business applications.
Microservices can be linked to multiple business applications, as such you can provide multiple different business application fact sheets.

AttributeRequiredDescription
factSheetIdYesA list of business application fact sheet ids
nameYesA list of the business application fact sheet names (corresponds to the display name of the fact sheet)

📘

Note

If you supply both factSheetId and name attributes, factSheetId will take precedence.

Tags

The Tags section allows you to assign tag groups and tags to your microservice.

AttributeRequiredDescription
tagGroupNameNoThe name of the tag group to assign to the microservice
tagNamesNoA list of tags from the tag group to assign to the microservice

Teams

The Teams section is where you can define the teams that own the microservice.

AttributeRequiredDescription
factSheetIdYesA list of the team fact sheet ids of the owning teams
nameYesA list of the team fact sheet names (corresponds to the display name of the fact sheet)

📘

Note

If you supply both factSheetId and name attributes, factSheetId will take precedence.

Resources

In the resources section of the manifest file, you can define the resources associated with your microservice. For more information, see Store Resources on Fact Sheets. The following table lists attributes to be provided in this section:

AttributeRequiredDescription
nameYesThe name of the resource
typeYesThe type of resource. You can find a list of the supported resource types in the API documentation
urlNoA URL for the resource
descriptionNoA brief description of the specific resource

📘

Note

Ensure that the tag groups, applications, and teams that you want to associate with your microservice exist within your LeanIX workspace.

Step 2: Commit the Manifest File

Establish a link between your microservice fact sheet by committing the manifest file to the default branch of your repository, such as main or master.

📘

Note

You can link multiple microservices to LeanIX microservice fact sheets from the same repository by placing the manifest files in separate subfolders within your repository structure. This is particularly useful if you have a monorepo.

Step 3: Create a Software Bill of Materials (SBOM)

SBOM files offer a detailed inventory of all software components, libraries, and modules used by a microservice. Using the Software Package Data Exchange (SPDX) format, you can efficiently track and manage these components. This enables you to ensure license compliance, identify potential security risks, and gain a deeper understanding of your software supply chain.

Incorporating the creation of a SBOM file into your CI/CD process can further automate the identification of software components within your microservice ecosystem. To learn how to integrate the generation of SBOM into your pipeline, see Generating a Software Bill of Materials (SBOM).

Step 4: Parse the Manifest File

Before you proceed with the configuration of your CI/CD workflow, it's crucial to create a script that will parse the manifest file. This script extracts and gathers the necessary information needed to formulate the API request. This is an essential step in the process as it ensures that all relevant data from your microservices is accurately captured and ready for the next phase.

To assist you in this process, we provide the following example script that shows how to parse the manifest file. This script serves as a reference point, helping you understand the process and showcasing the kind of functionality your script should include.

import logging
from pathlib import Path
import yaml
import requests
import json
import os

logging.basicConfig(level=logging.INFO)

# Request timeout
TIMEOUT = 20

# API token and Subdomain are set as env variables.
# It is adviced not to hard code sensitive information in your code.
LEANIX_API_TOKEN = os.getenv('LEANIX_API_TOKEN')
LEANIX_SUBDOMAIN = os.getenv('LEANIX_SUBDOMAIN')
LEANIX_FQDN = f'https://{LEANIX_SUBDOMAIN}.leanix.net/services'
LEANIX_MANIFEST_FILE = os.getenv('LEANIX_MANIFEST_FILE', 'leanix.yaml')

# OAuth2 URL to request the access token.
LEANIX_OAUTH2_URL = f'{LEANIX_FQDN}/mtm/v1/oauth2/token'

# Microservices APIs
LEANIX_MICROSERVICES = f'{LEANIX_FQDN}/technology-discovery/v1/microservices'

# Github Related
GITHUB_SERVER_URL = os.getenv('GITHUB_SERVER_URL')
GITHUB_REPOSITORY = os.getenv('GITHUB_REPOSITORY')

def obtain_access_token() -> str:
    """Obtains a LeanIX Access token using the Technical User generated
    API secret.

    Returns:
        str: The LeanIX Access Token
    """
    if not LEANIX_API_TOKEN:
        raise Exception('A valid token is required')
    response = requests.post(
        LEANIX_OAUTH2_URL,
        auth=('apitoken', LEANIX_API_TOKEN),
        data={'grant_type': 'client_credentials'},
    )
    response.raise_for_status()
    return response.json().get('access_token')


def _parse_manifest_file() -> dict:
    """Parses the Manifest file and generates the payload for the
    API request for the LeanIX Microservices API.

    Returns:
        dict: The payload for the API request
    """
    with open(LEANIX_MANIFEST_FILE, 'r') as file:
        try:
            logging.info(f'Parsing manifest file: {file.name}')
            manifest_data = yaml.safe_load(file)
        except yaml.YAMLError as exc:
            logging.error(f'Failed to load Manifest file: {exc}')
    
    if not manifest_data:
        logging.info('No manifest entries found')
        return
    manifest_microservices = manifest_data.get('services', [])
    micro_services = []
    for micro_service in manifest_microservices:
        api_data = {
            'externalId': micro_service.get('externalId', GITHUB_REPOSITORY),
            'name': micro_service.get('name'),
            'description': micro_service.get('description'),
            'applications': [
                {
                    'factSheetId': application.get('factSheetId')
                }
                for application in micro_service.get('applications', [])
            ],
            'repository': {
                'url': f'{GITHUB_SERVER_URL}/{GITHUB_REPOSITORY}',
                'status': 'ACTIVE',
                'visibility': 'PUBLIC'
            },
            'tags': [
                {
                    'tagGroupName': tag.get('tagGroupName'),
                    'tagNames': [
                        tag for tag in tag.get('tagNames')
                    ]
                }
                for tag in micro_service.get('tags', [])
            ],
            'teams': [
                {
                    'factSheetId': team.get('factSheetId')
                }
                for team in micro_service.get('teams', [])
            ],
            'resources': [
                {
                    'name': resource.get('name'),
                    'type': resource.get('type'),
                    'url': resource.get('url'),
                    'description': resource.get('description')
                }
                for resource in micro_service.get('resources', [])
            ]
        }
        micro_services.append(api_data)
    return micro_services

def _create_or_update_micro_services(microservice: dict, factsheet_id:str, create: bool=False) -> requests.Response:
    """Creates or Updates the LeanIX Microservice Fact Sheet

    Args:
        microservice (dict): The LeanIX matching API request payload
        create (bool, optional): Indicates wether to `create` or `update` the Fact Sheet. Defaults to False.

    Returns:
        requests.Response: The response of the request for further processing.
    """
    url = f'{LEANIX_MICROSERVICES}'
    if not create:
        url = f'{url}/{factsheet_id}'
    method = 'POST' if create is True else 'PUT'
    # Fetch the access token and set the Authorization Header
    auth_header = f'Bearer {os.environ.get('LEANIX_ACCESS_TOKEN')}'
    # Provide the headers
    headers = {
        'Authorization': auth_header,
    }
    response = requests.request(method=method, headers=headers, url=url, json=microservice)
    response.raise_for_status()
    return response
    
    
def create_or_update_micro_services(microservices: list):
    """Creates or updates the LeanIX Microservice Fact Sheet based on the provided manifest file.

    This function either updates an existing microservice or registers a new one based on the HTTP response status.
    If the response is `200`, an update is performed while a `404` status leads to registration of the microservice.
    Once the relevant operation is complete and the corresponding fact sheet ID has been received, 
    the function triggers the sbom ingestion request to register the relevant SBOM file with LeanIX.

    Args:
        microservices (List[dict]): A list of dictionaries, each representing a microservice.

    Returns:
        None
    """
    for microservice in microservices:
        factsheet_id = None
        params = {'externalId': microservice.get('externalId')}
        url = f'{LEANIX_MICROSERVICES}'
        # Fetch the access token and set the Authorization Header
        auth_header = f'Bearer {os.environ.get('LEANIX_ACCESS_TOKEN')}'
        # Provide the headers
        headers = {
            'Authorization': auth_header,
        }
        response = requests.get(
            url,
            params=params,
            headers=headers,
            timeout=TIMEOUT
        )
        if response.status_code == 200:
            # Micro Service exists, update
            logging.info(f'Microservice {microservice.get('externalId')} exists, updating')
            # Get the fact sheet id in order to perform the update.
            factsheet_id = response.json().get('data').get('factSheetId')
            crud_response = _create_or_update_micro_services(microservice, factsheet_id)
            logging.info(f'Updated Microservice: {microservice.get('externalId')}')
            logging.debug(f'Response: {json.dumps(crud_response.json())}')
        elif response.status_code == 404:
            # Microservice does not exist, create it
            crud_response = _create_or_update_micro_services(microservice, factsheet_id=None, create=True)
            logging.info(f'Microservice {microservice.get('externalId')} does not exist, creating')
            factsheet_id = crud_response.json().get('data').get('factSheetId')
            logging.info(f'Created Microservice: {microservice.get('externalId')}')
            logging.debug(f'Created Microservice: {json.dumps(crud_response.json())}')
        else:
            logging.error(f'Microservice check failed with: {response.status_code}, {response.content}')
            response.raise_for_status()
        if factsheet_id:
            register_sboms(factsheet_id)
            
    
def register_sboms(factsheet_id: str):
    """
    Registers the Software Bill of Materials (SBOM) file with LeanIX.

    This function enables improved understanding of the dependency landscape of your microservices.
    The SBOM provides comprehensive details about software components, their relationships, and 
    attributes, which are crucial for managing, securing, and licensing your open-source software.
    By registering the SBOM with LeanIX, these details can be effectively managed and tracked.

    Args:
        factsheet_id (str): The unique identifier of the microservice fact sheet. This ID is used 
        to associate the SBOM with the corresponding microservice in LeanIX.

    Returns:
        None
    """
    sbom_path = Path('sbom.json')
    if not sbom_path.exists():
        logging.warning('No sbom file found')
        return
    
    url = f'{LEANIX_MICROSERVICES}/{factsheet_id}/sboms'
    sbom_contents = dict()
    logging.info(f'Processing sbom file: {sbom_path.name} for Fact Sheet: {factsheet_id}')
    with sbom_path.open('rb') as f:
        sbom_contents = f.read()
        
    request_payload = {
        'sbom': (
            sbom_path.name,
            sbom_contents
        )
    }
    logging.debug(f'Populated payload for SBOM: {sbom_path.name}')
    # Fetch the access token and set the Authorization Header
    auth_header = f'Bearer {os.environ.get('LEANIX_ACCESS_TOKEN')}'
    # Provide the headers
    # NOTE: Don't set the content type, `requests` should handle this.
    headers = {
        'Authorization': auth_header,
    }
    logging.info(f'Sending sbom ingestion request for Fact Sheet: {factsheet_id}')
    response = requests.post(
        url, 
        headers=headers,
        files=request_payload,
        timeout=TIMEOUT
    )
    response.raise_for_status()
    logging.info(f'Successfully submited sbom request for Fact Sheet: {factsheet_id}')
    
def main():
    """LeanIX helper to parse the manifest file create or update a microservice
    and register the relevant dependencies.
    """
    manifest_data = _parse_manifest_file()
    create_or_update_micro_services(manifest_data)


if __name__ == "__main__":
    # Set the access token as an environment variable
    os.environ['LEANIX_ACCESS_TOKEN'] = obtain_access_token()
    main()

📘

Note

While the SBOM ingestion can technically be executed as a standalone step, it is intrinsically linked to the microservice discovery. Therefore, we highly advise integrating these two processes for optimal results.

Step 5: Set Up Your Secrets and Variables

To ensure the successful execution of our script, it's essential to configure certain secrets and variables on your repository provider. These configurations are crucial for authenticating and directing the script to the correct LeanIX workspace.

  • LEANIX_API_TOKEN (Secret): Your LeanIX API token, which serves as your authentication key. To learn how to get an API token, see Create a Technical User.

  • LEANIX_SUBDOMAIN (Variable): Your LeanIX subdomain, which is used to direct the script to your specific LeanIX workspace. You can copy your subdomain value from the workspace URL. For more information, see Base URL.

📘

Note

For detailed instructions on how to configure variables, please refer to the documentation of your repository provider.

Step 6: Configure Your Continuous Integration and Continuous Deployment

To complete the discovery and update process of your microservices, you need to establish a CI/CD workflow within your repository. This workflow will ensure that the system automatically detects and incorporates any changes made to your microservices, significantly reducing manual intervention and potential errors.

While it's possible to register microservices through API requests in any environment without a CI/CD pipeline, we strongly recommend using the CI/CD approach. The primary reason for this is to ensure that your data is always up to date. With this approach, your manifest file serves as the source of truth, and any changes made to it are automatically reflected in your microservices.

The CI/CD workflow setup may differ depending on your repository provider. The following table provides examples of how to establish a CI/CD workflow with various repository providers:

Repository ProviderAutomation Workflow
Azure DevOpsAzure Pipelines Example
BitbucketBitbucket Pipelines Example
GitHubGitHub Actions Example
GitHub Enterprise ServerGitHub Actions Example
GitLabGitLab Pipelines Example

📘

Note

The CI/CD approach is designed with flexibility in mind. If you decide to migrate to a Git integration down the line, the current setup effortlessly transitions. Your existing manifest file will seamlessly synchronize with the microservice fact sheet, eliminating the need for additional configuration or rework.

Best Practices

Software deployment often spans across several environments, such as testing, staging, and production. However, given that LeanIX is a tool for enterprise architecture management, we advise cataloging only those services that are deployed to the production environment. This is because LeanIX is designed to provide an overarching view of the software infrastructure in support of strategic decision-making.

For more technical use cases requiring visibility across all environments, there are specialized tools available in the market.