Advanced Use Cases for the Integration API

Field and Value Processing

Data Processors allow to map the incoming data found in the LDIF to fields of the specific entity in SAP LeanIX (e.g. a Fact Sheet) or the other way round in case of "Outbound Data Processors". The configuration allows to map incoming values to different types of SAP LeanIX fields (single value, float, multi value, life cycle...).

Applying Field and Value mappings may result in errors in case of source fields are not existing. Reason may be low input data quality, optional data in the source system. This types of "error" is expected and would only be noted in "test mode". Processing for next configuration lines of the data processor and next data processor always continues.

Type conversions to proper output type (variable for SAP LeanIX, String for LDIF) happen transparently. JUEL provides implicit type conversion and allows even calculations on Strings that contain numbers.

JUEL and RegEx

To provide high flexibility, predictability and at the same time guarantee to easily understand the configuration, all relevant configuration options for the Data Processors always support a combination of JUEL (http://juel.sourceforge.net/) and regEx configuration executed after each other.

JUEL allows to access and combine all input fields and values of the incoming data and data of the target entity (e.g. Fact Sheet).

RegEx allows final string mapping on the JUEL result.

All conversion of data types happens transparently.

While a most simple JUEL is always required to define the value to be used as an output, the RegEx replace may be empty if no value conversion is supposed to happen. Both methods provide in parts overlapping functionality. This is wanted and allows the user to focus on a potential solution based on technical knowledge.

Fields and value mapping in the Data Processors are configured as a list of single field configurations. Each configuration allows a JUEL/RegEx for the Key and a list of JUEL/RegEx Match/RegEx Replace for the value(s). This allows multi value field support.

Each JUEL Expression returns data.

This logic allows for configurations that fit many types of scenarios.
Processing of each field configuration works following this specification:

RegEx and JUEL

All RegEx filters allow negation and case insensitivity. The Java RegEx syntax can be applied: To match all but "notMe", "^((?!notMe).)*$" would be used. To ensure matching in a case insensitive manner, you'd add "(?i)" to the beginning of the regular expression.

Each inbound Data Processor JUEL expression contain the following references to the data object the is in scope for processing:

Each of them allow to access all data elements in the same or in subsections. It allows to e.g. access the id of the connector creating the LDIF. "${header.connectorId}" would result in an evaluated string "Kub Dev-001".

Using the "header" section, there is as well access to the global custom data section. Using "header.customFields.myGlodaldata1" the value of "myGlobaldata1" would be useable in any expression, given such a global value is provided in the JUEL. If not present (no customFields section or no defined key), this will always evaluate to an empty string.

Users can use any type of operation that can be executed on String objects in Java. Documentation of all the Java String methods is not in scope of this documentation. Methods for Java 8 can be found here: https://docs.oracle.com/javase/8/docs/api/java/lang/String.html

Advanced JUEL

JUEL Use Cases

Best Name for Fact Sheet

There are situations where it is not easy to find the best potential name for a Fact Sheet based on incoming data. The best name may not be available as it is not unique. Another use case might be that the source may provide different candidates for a name where we want to select from best possible option to lower ranked options automatically based on information availability for each data object.
On the other hand we want to ensure that we do not change names of already created Fact Sheets all of a sudden just because a better name option became available during an update.
All the above use cases can be covered simply by providing a list of potential name candidates. Every candidate that results in a null (evaluated in the 'values' section) or is already taken by another Fact Sheet will be skipped. In case I want to keep a name once set and not changed after creation, admins configure to read current Fact Sheet content and use the existing name as a first option. This will automatically be skipped if the Fact Sheet is not yet existing.

Example

Please see the example of a processor and a sample LDIF. You may test play around with matching against existing Fact Sheet names and remove/rename some of the keys from the source data and do test runs:

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Application",
   "filter": {
    "exactType": "Deployment"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 0,
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${lx.factsheet.name}"
      },
      {
       "expr": "${data.app}"
      },
      {
       "expr": "${data.app2}"
      },
      {
       "expr": "${data.app3}"
      }
     ]
    }
   ],
   "read": {
    "fields": [
     "name"
    ]
   },
   "logLevel": "debug"
  }
 ]
}

{
 "connectorType": "ee",
 "connectorId": "Kub Dev-001",
 "connectorVersion": "1.2.0",
 "lxVersion": "1.0.0",
 "description": "Imports kubernetes data into LeanIX",
 "processingDirection": "inbound",
 "processingMode": "partial",
 "customFields": {},
 "content": [
  {
   "type": "Deployment",
   "id": "634c16bf-198c-1129-9d08-92630b573fbf",
   "data": {
    "app3": "veryLongAndUnhandyNameIDoNotWantToSeeIfPossible",
    "app2": "littleBitBetterNameButStillNotGood",
    "app": "Best Name"
   }
  }
 ]
}

Using "Integration" Object

Example

To test the below example, please change the id in "ids" to an existing internal id of a Fact Sheet in your workspace. You may just open a Fact Sheet and copy the id from the browser URL.
In real world scenarios, you may not want to export the whole object, but iterate over tag groups or export a subset of the information.

{
 "scope": {
  "ids": [
   "869ee28b-c60a-4e88-8d18-f9e4ff466456"
  ],
  "facetFilters": []
 },
 "processors": [
  {
   "processorType": "outboundFactSheet",
   "processorName": "Export tag groups and tags",
   "processorDescription": "Sample how to export all available tag groups and all tags as part of one fact sheet export",
   "fields": [
    "name"
   ],
   "output": [
    {
     "key": {
      "expr": "content.id"
     },
     "values": [
      {
       "expr": "${lx.factsheet.id}"
      }
     ]
    },
    {
     "key": {
      "expr": "content.type"
     },
     "values": [
      {
       "expr": "${lx.factsheet.type}"
      }
     ]
    },
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${lx.factsheet.name}"
      }
     ]
    },
    {
     "key": {
      "expr": "tagGroupsAndTags"
     },
     "values": [
      {
       "object": "${integration.tags.getAllTagGroups()}"
      }
     ]
    }
   ]
  }
 ]
}

A similar configuration will help gathering all tag groups where a specific Fact Sheet has tags set. The lx.tagGroups list will be filled with all tag groups where the Fact Sheet has at least one tag set. And inside each tag group element there will be a list of the found tags for the Fact Sheet. The below example filters and returns only a sub set (default tag group). Just adding "${true}" in the filter will ensure to return all tag groups and included tags the Fact Sheet has set:

{
 "scope": {
  "ids": [
   "bb8b0b74-f737-4f1b-a937-a06bddf3fe47"
  ],
  "facetFilters": [
   {
    "keys": [
     "Application"
    ],
    "facetKey": "FactSheetTypes",
    "operator": "OR"
   }
  ]
 },
 "processors": [
  {
   "processorType": "outboundFactSheet",
   "processorName": "Export to LDIF",
   "processorDescription": "This is an example how to use the processor",
   "enabled": true,
   "fields": [
    "lifecycle",
    "location",
    "createdAt",
    "technicalSuitabilityDescription",
    "description"
   ],
   "relations": {
    "filter": [
     "relToParent",
     "relApplicationToITComponent"
    ],
    "fields": [
     "description"
    ],
    "targetFields": [
     "displayName",
     "externalId"
    ],
    "constrainingRelations": false
   },
   "tags": {
    "groups": [
     "SomeTagGroupName"
    ],
    "multipleGroups": "${dm.tagGroup.name =='Other tags'}"
   },
   "subscriptions": {
    "types": [
     "RESPONSIBLE"
    ]
   },
   "documents": {
    "filter": ".*"
   },
   "output": [
    {
     "key": {
      "expr": "content.id"
     },
     "values": [
      {
       "expr": "${lx.factsheet.id}"
      }
     ]
    },
    {
     "key": {
      "expr": "content.type"
     },
     "values": [
      {
       "expr": "${lx.factsheet.type}"
      }
     ]
    },
    {
     "key": {
      "expr": "Description"
     },
     "values": [
      {
       "expr": "${integration.toJson(lx.tagGroups).toString()}"
      }
     ],
     "optional": true
    }
   ]
  }
 ]
}

Calculating with single-select fields

This processor enables calculations with single-select fields

Example

{
 "connectorId": "id-92476445-10b3-40f7-9386-6f13c61e4b89",
 "connectorType": "ee",
 "connectorVersion": "1.2.0",
 "processingDirection": "inbound",
 "processingMode": "partial",
 "processors": [
  {
   "enabled": true,
   "filter": {
    "type": "DataObject"
   },
   "identifier": {
    "internal": "${content.id}"
   },
   "logLevel": "debug",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "processorName": "Apps from Deployments",
   "processorType": "inboundFactSheet",
   "read": {
    "fields": [
     "businessValue",
     "projectRisk",
     "dataClassification"
    ],
    "noNullForOrdinal": true
   },
   "run": 0,
   "type": "DataObject",
   "updates": [
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "expr": "No null - ${lx.toOrdinal('dataClassification')}"
      }
     ]
    }
   ],
   "variables": [
    {
     "key": "deploymentMaturity",
     "value": "${data.maturity}"
    }
   ]
  }
 ]
}

Updating multi_select fields from an array

This processor supports updating multi_select fields in SAP LeanIX.

Example

Pre-requisite:- Have the multi-select field myMultiSelect available in the workspace with options: FOO and BAR.
Update myMultiSelect field with values from myField in input.

{
	"processors": [
		{
			"processorType": "inboundFactSheet",
			"processorName": "Apps from Deployments",
			"processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
			"type": "Application",
			"filter": {
				"exactType": "Deployment"
			},
			"identifier": {
				"external": {
					"id": {
						"expr": "${content.id}"
					},
					"type": {
						"expr": "externalId"
					}
				}
			},
			"updates": [
				{
					"key": {
						"expr": "name"
					},
					"values": [
						{
							"expr": "${data.app}"
						}
					]
				},
				{
					"key": {
						"expr": "myMultiSelect"
					},
					"values": [
						{
							"expr": "${integration.output.valueOfForEach}",
							"forEach": {
								"elementOf": "${data['myField']}"
							}
						}
					]
				}
			]
		}
	]
}

{
	"connectorType": "Kubernetes",
	"connectorId": "Kub Dev-001",
	"connectorVersion": "1.2.0",
	"lxVersion": "1.0.0",
	"description": "Imports Kubernetes data into LeanIX",
	"processingDirection": "inbound",
	"processingMode": "partial",
	"customFields": {},
	"content": [
		{
			"type": "Deployment",
			"id": "634c16bf-198c-1129-9d08-92630b573fbf",
			"data": {
				"app": "HR Service",
				"tags": [],
				"myField": [
					"FOO"
				],
				"version": "1.8.4",
				"maturity": "3",
				"clusterName": "westeurope"
			}
		},
		{
			"type": "Deployment",
			"id": "784616bf-198c-11f9-9da8-9263b0573fbe",
			"data": {
				"app": "Finance Service",
				"tags": [
					"Important"
				],
				"myField": [
					"FOO",
					"BAR"
				],
				"version": "10.5",
				"maturity": "5",
				"clusterName": "westeurope"
			}
		}
	]
}

Remove content from fields

The integration API can be used to remove content. In case Values array contains "null" values after evaluating all configured elements in the Values Array, the Integration API will try to reset the configured field to an initial "not filled" state. This is specifically helpful for Single or multi select fields. String fields can simply be cleaned by passing an empty String. Numbers may rather be set to 0.
To avoid a warning, that no value could be found, ensure the "optional" field is used.

📘

variableProcessor

Is used to only write values to internal variables. This will be used for aggregation use cases where the LDIF content needs to be used to only collect values without directly writing anything to SAP LeanIX.

Write-back to Fact Sheet

inboundFactsheet processor, the inboundRelation and the writeToLdif processor allow to read information from the Fact Sheet (currently supported: fields, relations, subscriptions, tags, documents and metrics) and use the information when writing back to the Fact Sheet. In case you need to work with read information in other processors, please write results to a variable first. The below example shows two use case examples, where a cost field is increased by the incoming value and an update of the risk section will only be done if the description is not starting with a key word "manually".
The example as well contains information how to use this feature.
In case you define the read section for the inboundRelation processor, the fields will be read for the Fact Sheet defined in the "from" section. You can still read the fields from the target Fact Sheet using the "relations/targetFields" as shown below.

Example

For the example to work, the workspace needs to contain a Project Fact Sheet with external ID "12345". Or change the LDIF data to an external ID of a Project Fact Sheet existing in the workspace:

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Project",
   "filter": {
    "exactType": "prj"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 0,
   "updates": [
    {
     "key": {
      "expr": "budgetOpEx"
     },
     "values": [
      {
       "expr": "${lx.factsheet.budgetOpEx+data.monthlyOpEx}"
      }
     ]
    },
    {
     "key": {
      "expr": "projectRisk"
     },
     "values": [
      {
       "expr": "${(lx.tags.toString().contains('\"name\":\"MANUAL_INPUT'))?null:data.risk}",
       "regexMatch": ".+"
      }
     ],
     "optional": true
    },
    {
     "key": {
      "expr": "projectRiskDescription"
     },
     "values": [
      {
       "expr": "${(lx.tags.toString().contains('\"name\":\"MANUAL_INPUT'))?null:data.riskDescription}",
       "regexMatch": ".+"
      }
     ],
     "optional": true
    },
    {
     "key": {
      "expr": "metrics"
     },
     "values": [
      {
       "expr": "${integration.toJson(lx.metrics.variableName.values)}"
      }
     ]
    }
   ],
   "logLevel": "debug",
   "read": {
    "fields": [
     "budgetOpEx"
    ],
    "tags": {
     "groups": [
      "Other tags"
     ]
    },
    "relations": {
     "filter": [
      "relToParent",
      "relApplicationToITComponent"
     ],
     "fields": [
      "description"
     ],
     "targetFields": [
      "displayName",
      "externalId",
      "location"
     ]
    },
    "subscriptions": {
     "types": [
      "RESPONSIBLE"
     ]
    },
    "metrics": [
     {
      "name": "variableName",
      "measurement": "money",
      "fieldName": "dollars_per_day",
      "aggregationFunction": "MEAN",
      "groupBy": "1h",
      "start": "2020-01-20T00:00:00Z",
      "duration": "P0DT24H30M",
      "rules": {
       "key": "factSheetId",
       "comparator": "=",
       "compareWith": "${lx.factsheet.id}"
      }
     }
    ],
    "impacts": {
     "readAll": true
    }
   }
  }
 ]
}

{
  "connectorType": "showcaseUpdate",
  "connectorId": "showcaseUpdate",
  "connectorVersion": "1.0.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "prj",
      "id": "12345",
      "data": {
        "monthlyOpEx": 50000,
        "risk": "lowProjectRisk",
        "riskDescription": "The risk is considered to be low."
      }
    }
  ]
}

Example to access fields on relations and on the target Factsheet of a relation:

{
 "lx.relationsElement": {
  "id": "9316291b-361a-4050-ac79-bf9f96811fb1",
  "type": "relApplicationToITComponent",
  "target": {
   "id": "161abc0d-7bed-4440-b756-5c14a741e1ad",
   "name": "Application Development",
   "type": "ITComponent"
  },
  "activeFrom": "2021-01-18",
  "description": ""
 }
}

📘

Access to fields on relations and relation target fields

By defining the fields on the relations and on the target Fact Sheets of a relation, admins can use the values in JUEL expressions in the output section. The found relations need to be iterated using "forEach". Each element will then contain the standard information about e.g. name and type of a relation plus the requested fields. They can be accesses following the structure shown in the below example

Dynamic definition of Factsheet fields to read

Sometimes it is helpful to decide at run time which fields from a Fact Sheet to read and not hard code the names of the fields in the configuration.
For this purpose, Integration API allows to define a key "multipleFields" and a value that is a JUEL expression and will be resolved to boolean true and false. The Integration API will iterate over all available fields taken from the data model and allow the expression to do any filter logic required. As input value, the currently iterated field can be used with "dm.factSheetField.name". The type of the field can be identified with "dm.factSheetField.type"
In following JUEL expressions like forEach or update section, the list of read fields can be used with dm.factSheetFields, which is an object with keys: name, type and factsheetType

Example

The below example would read all fields of type "STRING" from a Fact Sheet.

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Application",
   "filter": {
    "exactType": "Deployment"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "read": {
    "multipleFields": "${dm.factSheetField.type=='STRING'}"
   },
   "updates": [
    {
     "key": {
      "expr": "${integration.valueOfForEach.name}"
     },
     "values": [
      {
       "expr": "${lx[integration.valueOfForEach.name]} - added by Integration API"
      }
     ]
    }
   ],
   "forEach": "${dm.factSheetFields}",
   "logLevel": "debug"
  }
 ]
}

In the sample, all String fields of the Fact Sheet get a string " - added by Integration API" appended.
multipleFields can be used for relations as well.
After evaluation of dynamic relation reading, fields dm.relationFilters[], lx.relationTargetFields[] and lx.relationFields[] are available

"multipleFields": "${data.fieldsToRead.contains(dm.relationField.name) && dm.relationField.type=='relToParent'}"

"multipleTargetFields": "${data.fieldsToRead.contains(dm.factSheetField.name) && dm.relationType=='relToParent'}"

"multipleFilters": "${data.myDynamicRelationList.contains(dm.relationType)}",

Same way, dynamic definition of Tags can happen: (after execution, the collected tag groups are available in a "lx.tagGroups" list.
In the filter "multipleGroups", the object of the currently iterated tag group "dm.tagGroup" can be used

{
 "read": {
  "tags": {
   "multipleGroups": "${true}",
   "groups": [
    "Cloud Transformation"
   ]
  }
 }
}

🚧

Availability of information read from the Fact Sheet

Information read from the Fact Sheet is available in the output section. The information is not available in the outer forEach, in the identifier and the filter section. The reason for this is, that at the time when the content in these sections is evaluated, the target Fact Sheet is not yet identified.

Auto deletion with inboundProcessors

Integration API supports the processing mode "full" mode when creating the configuration. Only in case, the configuration is set to mode "full", a section "deletionScope" is read from the processor configuration. The following operations are supported:

• Deletion of Fact Sheets If that section does contain a key "factSheets", all Fact Sheets matching the scope query inside will be removed if they are not found in the processed LDIF
• All Fact Sheets that match the deletion scope but are not touched by an inbound Data Processor during processing, will be removed (set to "Archived")
• Relations can be automatically removed as well. The structure to define relations to be deleted is similar. See an example configuration below. The example removes all relations but by narrowing the scope to fewer Fact Sheets, only for these Fact Sheets relations will be removed
• Documents can be deleted by defining a scope of Fact Sheets and adding a regular expression pattern to match the documents by name that may be removed if they are no longer referenced by the incoming LDIF data

How does this work? Do I need to delete any data manually or first delete in a processor?

The concept of deletion is unique with the Integration API in that there is no active deletion needed by the user. All the deletion is done by the iAPI automatically through the configuration of a 'deletion scope'.

The logic of deletion works as follows:

1. In an inbound 'Run', factsheets and other data like tags or relations are created and updated. These actions of creating or updating mark artifacts as 'touched'.
2. Upon finishing a the 'Run', the API determines which artifacts within the defined deletion scope have not been touched at all in the course of the run and these untouched artifacts are deleted.
3. This prevents constant “delete-create-delete-create” cycles that would be visible in the audit log of factsheets and avoids any manual work deletion any data

📘

Sending data for an archived Fact Sheet in the SyncRun

In case an archived Fact Sheet exists in the Workspace with the same externalId as sent in one of the input LDIFs, Integration API will create a new FactSheet rather than recovering the deleted Fact Sheet, this is done in order to avoid unexpected data on the archived Fact Sheet reflected on the Active Fact Sheet.
PS: This scenario can only happen when the "uniqueFactSheet" attribute for a Fact Sheet is set to "false" when set to "true" externalIds are already cleared while archiving a Fact Sheet.

📘

Multiple deletion scopes

Please note that you can define multiple sets of deletion scopes for every type (e.g. 2 Fact Sheet deletion scopes and 3 relation deletion scopes). Processed items during synchronization runs will be compared against each set separately. Any item in each deletion scope definition will be removed if not touched during processing. It is allowed to even define overlapping scopes. Each item will be handled once only.

Example

The example processor in this section removes the following data from a workspace:

• All project fact sheets that are no longer part of the incoming LDIF data
• All relations from applications to IT components
• All documents prefixed with "MyDocs_"

🚧

Caution

Before running this example processor, consider implications and always proceed with caution. The processor is configured to delete all project fact sheets present in the workspace. To limit the deletion scope, modify the code so that a tag such as "TEST_PRJ" is assigned to test projects. You can add this tag as a facetFilter to the deletion scope definition as shown below:

JSON

{
     "facetKey": "${integration.tags.getTagGroupId('Testing')}",
     "operator": "OR",
     "keys": [
      "${integration.tags.getTagId('Testing','TEST_PRJ')}"
     ]
}

Run this example processor multiple times to first create all projects, then remove one item and try again.

{
 "deletionScope": {
  "maximumDeletionRatio": {
   "relations": 40,
   "factSheets": 30
  },
  "factSheets": [
   {
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Project"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ],
     "ids": []
    }
   }
  ],
  "relations": [
   {
    "relationTypes": [
     "relApplicationToITComponent"
    ],
    "scope": {
     "facetFilters": [],
     "ids": []
    }
   }
  ],
  "documents": [
   {
    "documentMatches": [
     "^MyDocs_.*"
    ],
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Project"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ],
     "ids": []
    }
   }
  ]
 },
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Deployments",
   "type": "Project",
   "filter": {
    "exactType": "prj"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 0,
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${data.name}"
      }
     ]
    }
   ],
   "enabled": true,
   "logLevel": "debug"
  }
 ]
}

{
 "connectorType": "prjFull",
 "connectorId": "prjFull",
 "connectorVersion": "1.0.0",
 "lxVersion": "1.0.0",
 "content": [
  {
   "type": "Project",
   "id": "prj-42",
   "data": {
    "name": "Project 42"
   }
  },
  {
   "type": "Project",
   "id": "prj-43",
   "data": {
    "name": "Project 43"
   }
  },
  {
   "type": "Project",
   "id": "prj-44",
   "data": {
    "name": "Project 44"
   }
  }
 ]
}

Remove tags

Deletion of Tags works similar. One or more deletion scope sections of type "tags" needs to be configured. The scope defines the set of Fact Sheets to be looked at when removing tags and allows to configure a tag group and a tag name to be deleted. Tag names support regular expression matching to allow removal of tags based on name patterns. Please note that tags will be removed from the Fact Sheets where no longer referenced by processors adding them but tags themselves will not be deleted.

Example

To remove tags and subscriptions from Fact Sheets

{
 "deletionScope": {
  "maximumDeletionRatio": {
   "tags": 40
  },
  "tags": [
   {
    "tagScopes": [
     {
      "group": "myGroup",
      "tag": "Prefix_.*"
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Project"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ],
     "ids": []
    },
    "advanced": "${lx.tag.tagGroup.name==null}"
   }
  ],
  "subscriptions": [
   {
    "subscriptionScopes": [
     {
      "type": "RESPONSIBLE",
      "roles": [
       "My Role"
      ]
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Project"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ],
     "ids": []
    }
   }
  ]
 },
 "processors": []
}

Preventing removal of items

The Integration API can be configured to not delete any configured item if the ratio of to be deleted items in any deletion scope exceeds a defined ratio compared to the total items in scope for the deletion scope. This can be used to protect the existing data in case of erroneously sending incomplete data into a full scope synchronisation with deletion defined. by defining "maximumDeletionRatio" for each type of deletion scope, this mechanism can be used. A format example can be found in the examples above. A threshold of 30 means that all deletion will be stopped if 30% or more items would be deleted for a given deletion scope.

📘

Valid content for the deletion scope

To create valid JSON content to define the scope of Fact Sheets to be deleted if they no longer exist in the incoming LDIF, admins may want to use an outbound configuration. Using this configuration, a Button "Scope" is available that opens the facet filter UI. Once confirmed, the scope is automatically pasted to the processor configuration. Admins may copy and paste it into the inbound configuration where they need to use automatic deletion.

Advanced deletion

The functionality to delete content for elements we do no longer see references in the LDIF, can be used in an even more advanced was if required. The deletion scope for Relations, Tags and Documents may contain an optional key "advanced". If configured in the specific deletion scope, the value is evaluated as a JUEL expression resulting in "true" or "false". Only such elements will be added to the scope of elements potentially to be deleted if not touched where the JUEL expression evaluates to "true".

Multiple deletion scopes of same or different types can be defined. Please note that deletions of Fact Sheets will always happen last. This allows to use the Fact Sheet meta data lx.factsheet.* even for Fact Sheets that will be deleted. If allows to even use the owner field of the Fact Sheet deletion and evaluate in a relation, tag, document or subscription deletion scope in the state before the owner of the current run will be removed from the field (see ownership concept of advanced Fact Sheet deletion)

Please note, that currently no content from fields of type projectStatus is available for advanced deletion.

The information about the current Fact Sheet (for relations always the source Fact Sheet of a relation) is available using "lx.factsheet.*". All fields of the Fact Sheet can be used. In addition, all meta data fields of relations, documents and tags can be used in the JUEL for the related type of deletion scope.

{
 "deletionScope": {
  "factSheets": [
   {
    "scope":
     {
      "facetFilters": [
       {
        "keys": [
         "Project"
        ],
        "facetKey": "FactSheetTypes",
        "operator": "OR"
       }
      ]
     }
   }
  ],
  "relations": [
   {
    "relationTypes": [
     "relProjectToITComponent"
    ],
    "scope": {
     "ids": [],
     "facetFilters": []
    },
    "advanced": "${lx.relation.description.contains('from hr service')}"
   }
  ],
  "documents": [
   {
    "documentMatches": [
     ".*"
    ],
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Project"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ]
    },
    "advanced": "${lx.document.documentType.equals('jira') || lx.document.name.equals('someName') }"
   }
  ]
 },
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Read Projects",
   "processorDescription": "Creates LeanIX Projects from Project Management Solution",
   "type": "Project",
   "filter": {
    "exactType": "prj"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   }
  }
 ]
}

Deletion for multiple external sources

Advanced deletion is as well available for Fact Sheets but works slightly different. Advanced Fact Sheet deletion supports multiple external sources for one SAP LeanIX Fact Sheet. Deletion would not happen unless the last referenced source of a Fact Sheet does not longer contain information about the Fact Sheet.
This functionality is helpful in cases where a Fact Sheet might be created and again removed by potentially more than one foreign system providing separate LDIF to update the SAP LeanIX side.
In cases, where the Fact Sheet was no longer referenced by one of the sources, a deletion would not be a valid solution unless all sources no longer contain the information.
For such cases, the advanced deletion allows every sources to set a unique id as a marker. If this marker, called "owner", was found in the configuration, the Integration API first checks the field with all markers and only removes (archives) the Face Sheet if the list of markers found in the field is empty. The field to store the markers needs to be created in the data model as a standard String type. Integration API will read the content ant treat as a JSON list.
The below example shows example usage including the way to add the marker for a specific owner as part of the output section.

Example

Please note that the example uses the "alias" field to store the owner information. This is for testing and demonstration as it allows easy inspection and required no data model work. For production usage, this needs to be written to a new created field not visible in the UI.

{
 "deletionScope": {
  "factSheets": [
   {
    "scope": {
     "ids": [],
     "facetFilters": [
      {
       "keys": [
        "Process"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ]
    },
    "owner": {
     "fieldName": "alias",
     "ownerId": "myOwner"
    }
   }
  ]
 },
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Process",
   "identifier": {
    "external": {
     "id": {
      "expr": "fullSyncOwnerTest ${integration.valueOfForEach}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 0,
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "Full Sync Owner Test ${integration.valueOfForEach}"
      }
     ]
    },
    {
     "key": {
      "expr": "alias"
     },
     "values": [
      {
       "expr": "${helper:addIfNotExisting(lx.factsheet.alias, 'myOwner')}"
      }
     ]
    }
   ],
   "enabled": false,
   "forEach": "${data.tags}",
   "logLevel": "debug",
   "read": {
    "fields": [
     "alias"
    ]
   }
  }
 ]
}

Delete tags of multiple groups

Use case: Remove all tags of two tag groups from Fact Sheets of a given type can be performed as easy as shown in the below example. The deletion scope marks all Fact Sheets with tags in the tag groups and the given Fact Sheet type to be deleted if not touched. Then the processor configuration stays empty. No Fact Sheet will be touched when processing. At the end of the run, all tags will be removed.

Example

Please do not forget to increase the deletion limit set to 50% by default to 101 to allow removing even 100% of the tags.

{
 "deletionScope": {
  "maximumDeletionRatio": {
   "tags": 101
  },
  "tags": [
   {
    "tagScopes": [
     {
      "group": "Cloud:Region",
      "tag": ".*"
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "facetKey": "FactSheetTypes",
       "operator": "OR",
       "keys": [
        "CloudComponent"
       ]
      }
     ],
     "ids": []
    }
   },
   {
    "tagScopes": [
     {
      "group": "Cloud:Cloud Service",
      "tag": ".*"
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "facetKey": "FactSheetTypes",
       "operator": "OR",
       "keys": [
        "CloudComponent"
       ]
      }
     ],
     "ids": []
    }
   },
   {
    "tagScopes": [
     {
      "group": "Cloud:Tech Category",
      "tag": ".*"
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "facetKey": "FactSheetTypes",
       "operator": "OR",
       "keys": [
        "CloudComponent"
       ]
      }
     ],
     "ids": []
    }
   }
  ]
 },
 "processors": []
}

Deletion of Subscriptions

Example configuration to use advanced deletion on subscriptions. Here, to remove all subscriptions on application Fact Sheets that have an anonymised user as a subscriber for a subscription of type "RESPONSIBLE".

Please note, that the below configuration does not configure any processors, thus can work with an empty LDIF as input to trigger Integration API.

Example

{
 "deletionScope": {
  "subscriptions": [
   {
    "subscriptionScopes": [
     {
      "type": "RESPONSIBLE",
      "roles": []
     }
    ],
    "scope": {
     "facetFilters": [
      {
       "keys": [
        "Application"
       ],
       "facetKey": "FactSheetTypes",
       "operator": "OR"
      }
     ]
    },
    "advanced": "${lx.subscription.user.userName=='AnonymizedUser'}"
   }
  ]
 },
 "processors": []
}

{
 "connectorId": "subscription",
 "connectorType": "subscription",
 "connectorVersion": "1.0.0",
 "content": [],
 "lxVersion": "1.0.0"
}

📘

Using External IDs in pathfinder search scopes

When working with integrations, specifically with deletion scopes it is handy to know that pathfinder is capable filtering documents by their external id. and not only by internal id which is most of the times not known to foreign systems.
To filter by external ids, just use the field "externalIds" instead of "ids" in the search scope definition.
Please note that Pathfinder required a special syntax when defining external ids using the name of the externalID field, a slash ("/") then value of the externalId.
Example for the default externalId field:
"externalIds": ["${'externalId/'.concat(header.customFields.myExternalId)}"]

Data exchange and Aggregation between Objects

In some situations it may be required to use information from multiple Data Objects and store a joint result in another entity like a Fact Sheet or Relation. Even Creating specific relations if we find certain value combinations in different Data Objects is possible.

In order to perform such operations, a "variables" section is available to write and add to, while iterating over Data Objects. Data Processors in the following Runs (!! Not in the same Run) can then read the values and perform defined operations on them.

This works in the following steps:

Writing Variables using Expressions

{
 "variables": [
  {
   "key": "prefix_§{dataMyNameFromDataObjectValue}",
   "value": "${data.myValueFromDataObject}"
  }
 ]
}

Example

Processor Writing Variables
Below is an example of a processor with a matching LDIF that shows how variables work. In the first run which is marked by the processor with "run": 0, the variable section can be called on with the key aggregatedCosts and gathers together all the costs in the data section of each entry of an LDIF that is gathered by the filter in place. In this case, that filter is a Fact Sheet of type ITComponent. In the next run marked with "run": 1, the processor is calling the sum function on the variable aggregatedCosts and writing the sum to the description field of the Fact Sheets that fall under the specified filter, which in this case is all the Applications in the LDIF.

The example's result is three Fact Sheets created, two IT Components, and one Application with a description of 11. Note that the costs of the IT Components were not written in the IT Component's Fact Sheets.

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Create IT Components",
   "processorDescription": "One Processor for IT Components",
   "enabled": true,
   "type": "ITComponent",
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 0,
   "filter": {
    "exactType": "ITComponent"
   },
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${data.name}"
      }
     ]
    }
   ],
   "variables": [
    {
     "key": "aggregatedCosts",
     "value": "${data.cost}"
    }
   ]
  },
  {
   "processorType": "inboundFactSheet",
   "processorName": "Create Applications",
   "processorDescription": "Aggregated IT Costs in Application's Description",
   "enabled": true,
   "type": "Application",
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "run": 1,
   "filter": {
    "exactType": "Application"
   },
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${data.name}"
      }
     ]
    },
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "expr": "${variables.aggregatedCosts.sum()}"
      }
     ]
    }
   ]
  }
 ],
 "variables": {}
}

{
  "connectorType": "ee",
  "connectorId": "Kub Dev-001",
  "connectorVersion": "1.2.0",
  "lxVersion": "1.0.0",
  "content": [
    {
      "type": "ITComponent",
      "id": "itc1",
      "data": {
        "name": "IT1",
        "cost": 5
      }
    },
    {
      "type": "ITComponent",
      "id": "itc2",
      "data": {
        "name": "IT2",
        "cost": 6
      }
    },
    {
      "type": "Application",
      "id": "app",
      "data": {
        "name": "My App"
      }
    }
  ]
}

Dynamic Variable Handling

${variables[integration.valueOfForEach.concat('_cost')].sum()}
(which is same as: variables['12345_cost'].sum() in case valueOfForEach is "12345")

Supported Dynamic Variable Operations

Supported operations are listed below. Each invalid entry will be counted as "0" when calculating.

forEach Logic

Each data processor provides additional capabilities to handle values that are lists. Using the standard functionality, every data processor will be executed exactly one time for each data object sent to the Integration API.

Sometimes however, there is a need to update multiple Fact Sheets or multiple fields in a Fact Sheet for each value we find in a list of values found in the LDIF.

{
 "data": {
  "attachedDocuments": [
   {
    "extension": "vsdx",
    "name": "thediagram.vsdx",
    "displayName": "Diagram",
    "url": "sotrage.azure.com/123/thediagram.vsdx",
    "content": null
   },
   {
    "extension": "docx",
    "name": "thedoc.docx",
    "displayName": "Documentation",
    "url": "sotrage.azure.com/123/thedoc.docx",
    "content": null
   },
   {
    "extension": "html",
    "name": "webpage.html",
    "displayName": "Web Page",
    "url": null,
    "content": "<body>the vm 789 ...</body>"
   }
  ],
  "version": "1.8.4",
  "myForEachField": "attachedDocuments",
  "maturity": "3",
  "note": "I did the first comment here",
  "Home Country": "D",
  "Other Country": "UK",
  "clusterName": "leanix-westeurope-int-aks"
 }
}

{
 "processorType": "inboundFactSheet",
 "processorName": "Deployment To Application",
 "processorDescription": "The processor creates or updates an Application from every data object of type 'Deployment'",
 "type": "Application",
 "name": "My Awesome App",
 "run": 0,
 "enabled": true,
 "identifier": {
  "external": {
   "id": {
    "expr": "${content.id}"
   },
   "type": {
    "expr": "externalId"
   }
  }
 },
 "filter": {
  "exactType": "Deployment"
 },
 "forEach": "${data.myForEachField}",
 "updates": [
  {
   "key": {
    "expr": "name"
   },
   "values": [
    {
     "expr": "${data.attachedDocuments[integration.indexOfForEach].name}",  // or in short: ${integration.valueOfForEach.name}, remove this comment before trying
     "regexReplace": {
      "match": "",
      "replace": ""
     }
    },
    {
     "expr": "${data.value}"
    }
   ]
  }
 ]
}

Using the "forEach" section in each data processor as in the example above. Will result in executing the data processor "Deployment To Application" four times for the given data object and each run will allow the user to use the index of the current iteration in all expressions (integration.indexOfForEach).

To fill some output field of the data processor with the specific url (see example above), the configuration would look like this: ${data.attachedDocuments[integration.indexOfForEach].name}. This will generate the three different names of the attached documents in each run of the data processor. This could be used to create separate Fact Sheets and relations from the source data.

There is another way to access the value of the element referenced by the current index:

${integration.valueOfForEach}

Which is the same as:

${data.attachedDocuments[integration.indexOfForEach].name}

The index variable however can be used to reference the same index of another list element e.g. Important note: The admin can configure a "regexReplace" section in the forEach section. This will allow to manipulate the JSON representation of the value object resulting from the expression. In case such a manipulation is configured, it will have impact on the "integration.valueOfForEach" and not alter the original data one may reference using the indexOfForEach variable in the original data and reference manually.

Of course, the logic could be used to always execute a data processor n times. Just add '[1,2,3]' as configuration and the data processor will execute three times with the index variable integration.indexOfForEach set to 0-2 for reference.

In case the field 'attachedDocuments' is not available or contains an empty list, the data processor will not execute (operate on an empty list). In case the url is a single value and no list, the data processor will execute once.

The Integration API allows to iterate over list values and map values. In case of iterating over a map, indexOfForEach will always return -1 as maps are not sorted. For maps there is an additional variable "keyOfForEach" available providing access to the name of the key. The value will be accessed with "valueOfForEach"

Example: forEach

{
 "connectorType": "ee",
 "connectorId": "Kub Dev-001",
 "connectorVersion": "1.2.0",
 "lxVersion": "1.0.0",
 "content": [
  {
   "type": "Deployment",
   "id": "634c16bf-198c-1129-9d08-92630b573fbf",
   "data": {
    "app": "HR Service",
    "version": "1.8.4",
    "myList": [
     "lValue1",
     "lValue2"
    ],
    "myMap": {
     "key1": "value1",
     "key2": "value2"
    }
   }
  }
 ]
}

{
  "processors": [
    {
      "processorType": "inboundFactSheet",
      "processorName": "Apps from Deployments",
      "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
      "type": "Application",
      "filter": {
        "exactType": "Deployment"
      },
      "identifier": {
        "external": {
          "id": {
            "expr": "${content.id}"
          },
          "type": {
            "expr": "externalId"
          }
        }
      },
      "run": 0,
      "updates": [
        {
          "key": {
            "expr": "name"
          },
          "values": [
            {
              "expr": "${data.app}"
            }
          ]
        },
        {
          "key": {
            "expr": "description"
          },
          "values": [
            {
              "expr": "${integration.keyOfForEach}: ${integration.valueOfForEach}"
            }
          ]
        }
      ],
      "forEach": "${data.myMap}",
      "logLevel": "debug"
    }
  ]
}

For each logic can as well be applied inside the value section for and update key. The list or map will be iterated and the values result will contain n entires that are then mapped to the defined key. Please ensure to set the mode to "list" if not only the first value is to be used as a result (default mode for every key in the update section of a processor is "selectFirst" to only take the first non null result from what was defined in the values array. The "inner forEach" is behaving exactly as if the admin defined a fix number of elements in the "values" section.

Example: Nested forEach

Using all three options to iterate with "forEach" functionality, Integration API now allows to ingest data in LDIF where data structures are nested up to three levels.

{
 "key": {
  "expr": "targetITComponents"
 },
 "mode": "list",
 "values": [
  {
   "forEach": {
    "elementOf": "${lx.relations}",
    "filter": "${true}"
   },
   "map": [
    {
     "key": "id",
     "value": "${integration.output.valueOfForEach.target.id}"
    },
    {
     "key": "type",
     "value": "${integration.output.valueOfForEach.target.type}"
    },
    {
     "key": "name",
     "value": "${integration.output.valueOfForEach.target.displayName}"
    }
   ]
  }
 ]
}

Given the processor was configured to read relations (read section) and put the results into a list "lx-relations", the above example of an outboundFactsheet processor, will output all relation results into an array as value of a key named "targetITCmponents".
Please note that admins may configure the filter JUEL expression evaluating to boolean in order to not have some of the input list elements in the output. The JUEL may contain references to "integration.output.valueOfForEach" and filter on any content.

A third option to iterate using "forEach" is to add the key at the level of "updates". It allows to create a dynamic set of field updates to be pushed to e.g. a Fact Sheet. In the below example the fields to be updated will be read from the incoming LDIF. In order to execute the example, the referenced Fact Sheet needs to already exist. Most easy way is to one time execute the "starter example" configuration on the workspace.

Example: forEach : dynamic fields

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Application",
   "filter": {
    "exactType": "Deployment"
   },
   "logLevel": "debug",
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "updates": [
    {
     "key": {
      "expr": "${integration.updates.keyOfForEach}"
     },
     "values": [
      {
       "expr": "${integration.updates.valueOfForEach}"
      }
     ],
     "forEach": {
      "elementOf": "${data}",
      "filter": "${integration.updates.valueOfForEach!='toBeFiltered'}"
     }
    }
   ]
  }
 ]
}

{
 "connectorType": "Kubernetes",
 "connectorId": "Kub Dev-001",
 "connectorVersion": "1.2.0",
 "lxVersion": "1.0.0",
 "description": "Imports Kubernetes data into LeanIX",
 "processingDirection": "inbound",
 "processingMode": "partial",
 "customFields": {},
 "content": [
  {
   "type": "Deployment",
   "id": "634c16bf-198c-1129-9d08-92630b573fbf",
   "data": {
    "name": "HR Service",
    "version": "toBeFiltered",
    "description": "test description"
   }
  }
 ]
}

Use "object" key to output all objects

The "object" key might be used instead of "map" or "expr" to retrieve a representation of any potential input object as a defined value. This allows to easily export all information without need to know about the details inside the object.

{
 "scope": {
  "facetFilters": [
   {
    "facetKey": "FactSheetTypes",
    "operator": "OR",
    "keys": [
     "Application"
    ]
   }
  ],
  "ids": [
   "90a8296c-92fe-4009-a4cf-21db710719ec"
  ]
 },
 "processors": [
  {
   "processorType": "outboundFactSheet",
   "logLevel": "debug",
   "fields": [
    "lifecycle"
   ],
   "output": [
    {
     "key": {
      "expr": "content.id"
     },
     "values": [
      {
       "expr": "${lx.factsheet.id}"
      }
     ]
    },
    {
     "key": {
      "expr": "content.type"
     },
     "values": [
      {
       "expr": "${lx.factsheet.type}"
      }
     ]
    },
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "object": "${lx.factsheet.lifecycle}"
      }
     ]
    }
   ]
  }
 ]
}

Filter Processor execution based on current Fact Sheet content

Using the onRead filter in an inbound processor allows to execute a processor based on currently existing Fact Sheets. The processor can be configured to only execute if a Fact sheet already exists or if the Fact sheet has defined values in some fields.

Or exactly the other way round: The Processor may only be executed if the Fact Sheet does not yet exist.

Example

The following Example shows a Processor that will execute if a Fact Sheet exists that has a defined name and is flagged with a certain Tag.
It adds " (Cloud)" to the name of a Fact Sheet if current the name is exactly as defined in the data object and a tag "Public Cloud" in a tag group "Cloud Transformation" is set on the Fact Sheet.

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Application",
   "filter": {
    "exactType": "Deployment",
    "onRead": "${lx.factsheet.name==data.name && lx.tags.size()>0 && lx.tags[0].name=='Public Cloud'}"
   },
   "identifier": {
    "external": {
     "id": {
      "expr": "${content.id}"
     },
     "type": {
      "expr": "externalId"
     }
    }
   },
   "updates": [
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${lx.factsheet.name} (Cloud)"
      }
     ]
    }
   ],
   "logLevel": "debug",
   "read": {
    "fields": [
     "name"
    ],
    "tags": {
     "groups": [
      "Cloud Transformation"
     ]
    }
   }
  }
 ]
}

❗️

onRead is available for inbound processors only

The onRead filter is available for inbound processors. The outbound processor will ignore setting this filter configuration.
Functionality is available same way outbound however as it is possible to use read content in the "advanced" filter like e.g. "filter": {"advanced": "${lx.relations.size()>0}"}, for cases where you only want to export if any requested relation was found

📘

Order of RegEx execution

Using the replace regEx will allow to modify the output after applying the match regEx

Load large LDIFs

Depending on the source of the incoming data, LDIF files can be very large. The Integration API may not accept LDIF larger than 50 MB. If your file is bigger, it needs to be provided as a URL reference on an Azure Blob storage.
The configuration can be added to the Processor configuration part:

{
  "dataProvider": {
    "url": "${header.customFields.url}"
  }, 
  "processors": [
  ]
}

The value can be configured fix or as in the example passed in as part of the custom fields information in the LDIF. Please ensure to not send the "content" section in case you want to read from Azure. If content is part of the API call, this content will be used instead of the content in the Azure storage.
The URL needs to contain the path to the blob storage entry plus the Azure SAS token. See Azure documentation for details. https://docs.microsoft.com/bs-latn-ba/azure/storage/blobs/storage-blob-user-delegation-sas-create-cli

📘

No support for IP whitelisting

Please note that we process from our Azure infrastructure where IP addresses can dynamically change. Reading data from a URL only works if no IP whitelisting is set. Instead SaS tokens with limited ttl can be used.

Long running API calls

The default behaviour of Integration API is to execute all changes with the user that is provided by the API client when logging in. E.g. the history of factsheets will show this user as the one executing the changes as if the user logged into the SAP LeanIX UI and did the changes manually. This is easy to understand and communicate. In some situations however, it may not be sufficient.

Use Cases may be:

• The process takes more than 60 minutes.
• All changes the API does are caused by different login users, but should rather be shown to the user as a change the Integration API did instead of showing all the different API users.

In order to support long running inbound or outbound processes, an API Token may be provided in the configuration. This will be used instead of the access token that comes with the call from the API (as it will expire after one hour and does not contain any refresh token to grant access to the workspace data for more than 60 minutes)

{
  "credentials": { 
    "apiToken": "..." 
  }, 
  "processors": [
  ]
}

As an alternative, the Integration API can be configured to use a "Technical User" for accessing the SAP LeanIX Pathfinder backend. This user called INTEGRATION_API will be automatically created in the workspace by the API if it is not existing.
To use the Technical Users, please add the following section

{
  "credentials": { 
    "useTechnicalUser": true 
  }, 
  "processors": [
  ]
}

Executing Processor Configurations with Custom Technical Users

Besides using the default technical user that is created by the integration, it is possible to use technical users that are already set up in the workspace. To execute a processor configuration with a custom technical user, the user ID of the technical user has to be added to the credentials section of the configuration. In the following example, the CUSTOM_TECHNICAL_USER_ID placeholder would need to be replaced by the ID of the technical user that will be used when reading data from or writing data to your workspace.

{
  "credentials": { 
    "technicalUserId": "CUSTOM_TECHNICAL_USER_ID" 
  }, 
  "processors": [
  ]
}

Restricting the Execution of Processor Configurations

To restrict the execution of processor configurations to specific users, you can add the executionRestrictions object next to the processors of the configuration.

{
  "executionRestrictions": { 
    "defaultTechnicalUser": true,
    "userIds": ["USER_ID"]
  }, 
  "processors": [
  ]
}

The defaultTechnicalUser parameter specifies whether the default technical user should be allowed to execute the configuration, while the userIds parameter specifies the user IDs of the users that should be allowed to execute the configuration. To restrict execution to multiple users, simply add their user IDs to the userIds array. Both parameters are optional and when the executionRestrictionsobject is specified and keept empty no user will be able to execute the processor.

Search Based Matching of Fact Sheets

When using the "search" based identification of the Fact Sheet that are supposed to be updated by the incoming data object, then the section may contain a section to limit the scope of searched Fact Sheets and an expression filtering the Fact Sheets that should be updated.

In case Integration API iterates over a search result, two variables can be used in all JUEL expressions: search.resultSize (indicating the total number of items we iterate over in the processor) and search.resultIndex (number of current item being iterated)

When configuring an inboundFactsheet processor, a key "search" is now allowed. the value of this key is an object as defined in the example below. One or more Fact Sheets may be identified by the search and be updated based on the same data object in the LDIF.

The search works in two steps:

1. The "scope" defines a search against the pathfinder backend and limits the number of Fact Sheets to be matched. A valid scope can e.g. be created by using an outbound Integration API configuration and click on "set scope". Then the Scope can be copied from there.

2. After reducing the scope of Fact Sheets potentially in scope as good as possible with pathfinder filtering options, an additional JUEL expression is being executed to further narrow down the scope (key: "filter"). This is the far more costly process but allows much more flexibility identifying the right Fact Sheets. Admins should always try to limit the scope in phase 1 as far as possible to avoid long processing times. In the JUEL expression all fields defined in the "read" section can be used for filtering (e.g. ${lx.factsheet.description.startsWith('Autoupdate: ')}

As the Outcome, all identified Fact Sheets will be processed as if they had been found by the processor one by one.

The "search" key can be used in conjunction with the "external" key. In this scenario, the Integration API first tries to find based on the external ID. If that fails, the search will be executed. If that fails as well, the integration API will try to create the Fact Sheet based on information in the "external" value. The last step can be avoided if creation is not allowed for the use case by adding an onRead filter and check for "lx.factsheet" not being null.
From a use case perspective, it allows to search for a Fact Sheet with a specific external id, if not existing then search for a Fact Sheet with e.g. a specific name and add the external id in case it is existing. OR create a new Fact Sheet in case a Fact Sheet with the name was not found.

The key "multipleMatchesAllowed" allows to define the API behaviour in case multiple Fact Sheets are matching the search criteria. Some use cases may only want to update if exactly one Fact Sheet was found (then the value will be set to "false"). By allowing multiple matches, bulk updates on multiple Fact Sheets are possible. Default if not existing is "true"

Example

The Below Processor will update all descriptions of Application Fact Sheets that have a tag "AsiaPacific" in the tag group "Region".

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Update all Cloud Apps",
   "processorDescription": "Updates all Apps with tag 'Cloud'",
   "type": "Application",
   "filter": {
    "exactType": "AppUpdate"
   },
   "identifier": {
    "search": {
     "scope": {
      "facetFilters": [
       {
        "facetKey": "FactSheetTypes",
        "operator": "OR",
        "keys": [
         "Application"
        ]
       },
       {
        "facetKey": "${integration.tags.getTagGroupId('Region')}",
        "operator": "OR",
        "keys": [
         "${integration.tags.getTagId('Region','AsiaPacific')}"
        ]
       }
      ],
      "ids": []
     },
     "filter": "${true}",
     "multipleMatchesAllowed": true
    }
   },
   "logLevel": "debug",
   "updates": [
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "expr": "External sync executed ${data.dateTime}"
      }
     ]
    }
   ]
  }
 ]
}

{
 "connectorType": "searchBasedScope",
 "connectorId": "searchBasedScope",
 "connectorVersion": "1.0.0",
 "lxVersion": "1.0.0",
 "description": "Updates external sync date",
 "processingDirection": "inbound",
 "processingMode": "partial",
 "customFields": {},
 "content": [
  {
   "type": "AppUpdate",
   "id": "apps",
   "data": {
    "dateTime": "06/08/2019"
   }
  }
 ]
}

Extending the example is easy to e.g. only match update the Fact Sheets where the description already starts with a specific text, indicating that an automatic update was allowed:

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Update all Cloud Apps",
   "processorDescription": "Updates all Apps with tag 'Cloud'",
   "type": "Application",
   "filter": {
    "exactType": "AppUpdate"
   },
   "identifier": {
    "search": {
     "scope": {
      "facetFilters": [
       {
        "facetKey": "FactSheetTypes",
        "operator": "OR",
        "keys": [
         "Application"
        ]
       },
       {
        "facetKey": "${integration.tags.getTagGroupId('Region')}",
        "operator": "OR",
        "keys": [
         "${integration.tags.getTagId('Region','AsiaPacific')}"
        ]
       }
      ],
      "ids": []
     },
     "filter": "${lx.factsheet.description.startsWith('External sync')}",
     "multipleMatchesAllowed": true
    }
   },
   "logLevel": "debug",
   "read": {
    "fields": [
     "description"
    ]
   },
   "updates": [
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "expr": "External sync executed ${data.dateTime}"
      }
     ]
    }
   ]
  }
 ]
}

Include Archived Fact Sheets in scope result

It is even possible to tell iAPI to include archived Fact Sheets in the result by enabling a specific flag. Just set "omitArchivedFactSheets" to false in the scope:

{
	"scope": {
	    "omitArchivedFactSheets": false,
		"ids": [],
		"facetFilters": [
			{
				"keys": [
					"BusinessCapability"
				],
				"facetKey": "FactSheetTypes",
				"operator": "OR"
			},
			{
				"keys": [
					"archived"
				],
				"facetKey": "TrashBin",
				"operator": "OR"
			}
		]
	},
	"processors": [
		....
	]
}

📘

Using ExternalIDs in search based scope (and in deletion scope)

Scope filters may not only filter for items by their internal ids (key "ids") but by external ids as well. To use this, the key "externalIds needs to be defined and contain an array of searched external ids. Each need to be prefixed with the name of the external id field and a slash. See example below

The Below example can even be dynamic and inject content from the LDIF custom fields:
"externalIds": ["${'externalId/'.concat(header.customFields.myExternalId)}"]

{
 "scope": {
  "facetFilters": [],
  "externalIds": [
   "externalId/Ext-ID-0m0NiY6Z"
  ]
 }
}

Bookmarks to define search scope

It is possible to not hardcode the search scope into the Integration API configuration but allow users with access to specific bookmarks to dynamically change the scope of Integration API runs by modifying the Bookmark in the Frontend of the application. In Integration API the ID or name of a Bookmark can be configured to be used to set the scope for search based scoping as shown in the below example:

Example

In case, a bookmark is used, the whole bookmark object is available to be used in JUEL expressions in the processor. The below example shows this by accessing the bookmark name in the updates section.

{
 "processors": [
  {
   "processorType": "inboundFactSheet",
   "processorName": "Apps from Deployments",
   "processorDescription": "Creates LeanIX Applications from Kubernetes Deployments",
   "type": "Application",
   "filter": {
    "exactType": "Deployment"
   },
   "identifier": {
    "search": {
     "filter": "${true}",
     "multipleMatchesAllowed": true,
     "scopeFromBookmark": "${integration.bookmarks.getBookmarkId('book-1')}"
    }
   },
   "updates": [
    {
     "key": {
      "expr": "description"
     },
     "values": [
      {
       "expr": "bookmark name: '${bookmark.name}' AND id=${integration.bookmarks.getBookmarkId('book-1')}"
      }
     ]
    }
   ]
  }
 ]
}

📘

Find out details about Bookmarks

You may output the bookmark object into the description field to inspect the structure and available information in a Bookmark. Use the "test run" mode to not alter any Fact Sheets.

Read-Only FactSheet Processor

In case you want to collect information from the resulting FactSheets into variables, you can enable the "readonly" mode on the inboundFactsheet processor. In the example below, all entries of the "releases" field of each application are collected into a variable releases. Results could then be used in a next run.

Using the "filter": "${myExpression}", the set of Fact Sheet values collected can be even more narrowed down according to the use case. This feature comes handy to save processing time as the processor does not need to prepare any write operation.

Example

{
 "processorType": "inboundFactSheet",
 "processorName": "Process variables with Search Scope",
 "processorDescription": "Collect deploymentMaturity ",
 "type": "Application",
 "filter": {
  "exactType": "Deployment"
 },
 "identifier": {
  "search": {
   "scope": {
    "ids": [],
    "facetFilters": [
     {
      "keys": [
       "Application"
      ],
      "facetKey": "FactSheetTypes",
      "operator": "OR"
     }
    ]
   }
  }
 },
 "run": 0,
 "enabled": true,
 "variables": [
  {
   "key": "releases",
   "value": "${lx.release}"
  }
 ],
 "logLevel": "debug",
 "readOnly": true
}

Accessing the "Hierarchy Level" of a Fact Sheet

It is easily possible to access the hierarchy level of a Fact Sheet just by reading the field "level" provided by the pathfinder backend. This information can be used to filter for certain hierarchy levels or do calculations.
In the below example, it is used to filter for Level 2 Project Fact Sheets for export.

This could of-course have been done by just applying the restriction to the pathfinder scope query. The below is just to show case and allow extending for more advanced filtering.

Example

{
 "scope": {
  "facetFilters": [
   {
    "facetKey": "FactSheetTypes",
    "operator": "OR",
    "keys": [
     "Project"
    ]
   }
  ],
  "ids": []
 },
 "processors": [
  {
   "processorType": "outboundFactSheet",
   "processorName": "Export Projects L2",
   "processorDescription": "Exports only Level 2",
   "enabled": true,
   "filter": {
    "advanced": "${lx.factsheet.level==2}"
   },
   "fields": [
    "name",
    "level"
   ],
   "output": [
    {
     "key": {
      "expr": "content.id"
     },
     "values": [
      {
       "expr": "${lx.factsheet.id}"
      }
     ]
    },
    {
     "key": {
      "expr": "content.type"
     },
     "values": [
      {
       "expr": "project}"
      }
     ]
    },
    {
     "key": {
      "expr": "name"
     },
     "values": [
      {
       "expr": "${lx.factsheet.name}"
      }
     ]
    },
    {
     "key": {
      "expr": "level"
     },
     "values": [
      {
       "expr": "${lx.factsheet.level}"
      }
     ]
    }
   ]
  }
 ]
}