Demystifying CI/CD Pipelines for SAP Cloud Integration Projects

[SurajitNeogi]

Introduction

CI/CD is a hot topic in the world of software development. It stands for Continuous Integration and Continuous Deployment (or Continuous Delivery), and it plays a pivotal role in modern software development processes. CI/CD is a set of practices, tools, and principles aimed at automating and streamlining the software development lifecycle.

Most modern developer tools, such as VS Code, even SAP BAS, etc. now come equipped with native Git integration. However, SAP Cloud Integration differs significantly from other development tools, and as a result, implementing a CI/CD solution for it presents unique challenges.

Getting started with this process can be relatively straightforward, but it does necessitate a certain level of familiarity with Git, pipelines, and some knowledge on any CI/CD tools such as Jenkins, GitHub Actions, Bitbucket, Azure Pipelines or GitLab, among others. Moreover, you'll need to install additional tools or programs on your local computer to kickstart the CI/CD process.

However, it's important to note that while constructing a CI/CD pipeline using CI/CD tools can be relatively straightforward, resolving issues or errors in the pipeline and maintaining it demands additional skills, time, and effort. SAP customers may need to train their workforce to effectively use CI/CD tools.

 

So far, solutions available in the context of SAP Cloud Integration include:

• Project Piper Jenkins template by SAP. Details can be found on the blog post by axel.albrecht.


• FlashPipe by engswee.yeoh. Details can be found on the blog post.


• SAP CPI: CI/CD from from zero to hero blog series by nuno.pereira3


• SAP BTP CI/CD service by SAP. Details can be found on the blog post.


• Figaf DevOps Suite by daniel.graversen

 

All the above solutions have been crafted by experts and possess their own merits. However, it's important to note that all the above solutions have not been developed using the native style of SAP Cloud Integration development, which typically involves the use of the building blocks of the integration artifacts available. With this in mind, my goal was to leverage the inherent capabilities of SAP Cloud Integration, alongside the Power of Groovy and APIs, to create a solution that aligns more closely with the platform's core development style.

Allow me to introduce another possible approach to implement the Continuous Integration (CI) & Continuous Delivery/Deployment (CD) pipelines for SAP Cloud Integration, that enables anyone to setup CI/CD pipelines easily without even relying on native Git, a prerequisite for most CI/CD and DevOps tools. Designed with ease-of-use in mind, making it simple and straightforward to begin without the need for any additional external tools or specialized knowledge. It is designed entirely around the core building blocks and artifacts of SAP Cloud Integration.

Solution Overview

Fig.1 – General architecture

 

The diagram describes the general architecture of the solution, beginning with SAP Cloud Integration where integration artifacts/flows are executed as part of a CI/CD pipeline. GitHub is used as the Source Code Management (SCM) tool since it is the most popular choice in the community.

These integration artifacts use the standard public ODATA APIs of the SAP Integration Suite and the public GitHub REST API under the hood to create CI/CD pipelines.

Since different SAP customers have diverse landscapes and processes, CI/CD pipelines also vary for each SAP customer. Instead of supporting only one approach and potentially missing out on others, the design/architecture aims to provide a common framework. This framework enables anyone to easily select and adapt useful scenarios (as required), such as synchronizing artifacts from a tenant to GitHub, uploading artifacts from GitHub to a tenant, or copying/transporting artifacts from one tenant to other instance/tenant, and deploying artifacts to the runtime if necessary.

These integration artifacts are built in a way that requires you to replace only a few configuration parameters specific to your environment and organization/project, leaving the rest unchanged. Furthermore, you can use these integration artifacts as a starting point for further customization tailored to the needs of your organization or project.

This integration artifacts is published as an open source project.

What does this mean for you?

Firstly, it guarantees that you will always have access to the complete source code, which will be readily available on GitHub and included in every project release. Secondly, the open-source license, specifically the MIT License, provides you with the freedom to make modifications to the code. This is released under the MIT License, which is known for its permissiveness and simplicity among open-source licenses. This means that if you wish to make additions or alterations, you can do so without any restrictions.

You can download the complete integration package from GitHub at this link: Download. The full source code and all other necessary files are included in the download. Everything you need is also available in the GitHub repository ci-cd-sap-cloud-integration. For documentation and installation instructions, please refer to the user documentation. If you encounter any issues, please do let me know by either commenting below or creating an issue on GitHub.

Contributions from the community are welcome. If you would like to contribute code, provide fixes, or suggest improvements, please refer to our contribution guidelines page on GitHub.

Solution Details

Integration artifacts Details:

This integration package contains 4 integration flows. For more details, please refer to the table below.

Table 1

Reference to General Architecture	Integration Flow Name	Descriptions
1.	Sync Integration Contents to GitHub	Sync all or specific integration contents (only packages, iflows, value mappings) from the SAP Cloud Integration tenant to GitHub Repository to store and see the full history of changes
2.	Sync Integration Contents to GitHub - Delta	Check the SAP Cloud Integration tenant for a new version of your integration contents (only packages, iflows) and if a new version exists, sync the changed integration contents to GitHub Repository and keep track of the latest changes.
3.	Transport or Migrate Integration Contents	Export the latest version of all or specific integration artifacts (either packages or iflows or value mappings) from the Source tenant and either import or update the artifact on the Target tenant without using any external tool.
4.	Upload Integration Contents from GitHub	Download the latest version of either an integration flow artifact or resources of an integration flow artifact from GitHub repository and either update or create the artifacts on the tenant.

Some Key Points:

Recommendation:
Before going deep into the topic, I think it is very important to understand the underlying ideas, concepts, and design style of the above development by experimenting in your trial or non-production tenant with a smaller set of interfaces. Once you grasp the concept of each exchange property, Groovy, chaining of API requests, and HTTP Receiver configuration, you will easily able to change and adapt as per your project needs.

1. You can select each Integration Flow based on the artifact type (such as packages, iFlows, or value mappings). For more details, please refer to the configuration guide for each Integration Flow and Table 2.


2. In each Integration Flow, you will have the option to select all, multiple, or single IDs for each artifact type (such as packages, iFlows, or value mappings). Additionally, for the 'select all' option, you can restrict the amount of data retrieved through Odata API using the 'Top' and 'Skip' properties.


3. Integration Flow 1 and 2 have the capability to create the required Zip file or raw files under one of two folders: "Zip" or "UnZip," depending on the selected configuration parameter. If "Zip" is selected, the Zip files of the integration artifacts will be stored in GitHub under the "Zip" folder. If "UnZip" is selected, then all the raw files of the integration artifacts will be stored in GitHub under the "UnZip" folder.


4. The "Zip" folder will exclusively contain ZIP files that can be easily downloaded and re-imported to the tenant. The "UnZip" folder will contain raw files, which can be used to keep track of changes in each version. Refer to the GitHub repository sap-cloud-integration-artifacts to understand the folder structure and how files will be stored there.


5. For Integration Flow 4 to work properly, corresponding artifacts for Integration flow should be stored under "Zip" folder and the artifacts for Integration resources should be stored under "UnZip" folder in the GitHub repository.


6. All the integration flows generate two CSV files, namely HTTP LOGS and SUMMARY REPORT. 'HTTP LOGS' file will contain all the API call logs in sequential order, inspired by Postman Collection Runner. These logs provide details about each API call, aiding in troubleshooting and pinpointing the exact API call where any errors occurred. 'SUMMARY REPORT' file will provide a high-level overview of the overall execution.


7. Lastly, it's important to note that all the integration flows share a similar and consistent design style, including the Groovy code. Once you grasp the design style of one iFlow, you'll find that the others follow a nearly identical design style and coding approach.

 

Each above integration flow provides the capability to filter artifacts by [Packages, Integration Flow, Value Mapping]. For more details, please refer to the table below. Additionally, please note that filtering options for other artifacts types will be added in upcoming releases.

Table 2

Integration Flow Name	Packages	Integration Flow	Value Mapping
Sync Integration Contents to GitHub	Yes	Yes	Yes
Sync Integration Contents to GitHub - Delta	Yes	Yes	No
Transport Integration Contents	Yes	Yes	Yes
Upload Integration Contents from GitHub *	No	Yes	No

* Additionally you can filter (single or multiple) specific resources of an integration flow artifact.


Additionally, each above integration flow provides optional parameters. For more details, please refer to the table below.

Table 3

Integration Flow Name	Deploy	Send Notification to MS Teams Channel
Sync Integration Contents to GitHub	No	Yes
Sync Integration Contents to GitHub - Delta	No	Yes
Transport Integration Contents	Yes	Yes
Upload Integration Contents from GitHub	No	No

How to use?

Prerequisite

To use this integration artifacts, you will need the following:

1. 
   
   
   
   1. Download the Integration Package ZIP archive from the GitHub Repo and import Integration Package into your tenant.
   
   
   2. Create an OAuth client (Client Credentials grant type) on the tenant.
      
      
      
      • Navigate to your SAP BTP cockpit of the Cloud Foundry subaccount, go to 'Services', then 'Service Marketplace' and select 'Process Integration'
      
      
      • Create a new instance with service plan 'api' and with the following configuration:
        
        
        
        • Grant-types: 'client_credentials'
        
        
        • Roles:
          
          
          
          • AuthGroup_IntegrationDeveloper
          
          
          • AuthGroup_Administrator
          
          
          • AuthGroup_BusinessExpert
          
          
          
          
        
        
        
        
      
      
      • Create a Service Key and copy the entire JSON text to your clipboard to use in the step 4 for 'Create OAuth2 Client Credentials'.
      
      
      • [Optional] Repeat the above steps for the Target tenant (required only for Transport Integration Contents iflow)
      
      
      • 
        

        For the Neo Environment, please refer to steps 1 and 2 in the Neo documentation. Assign the roles mentioned above to the user 'oauth_client_'. Copy the Token Endpoint (found in the branding tab), Client ID, and Client Secret to your clipboard for use in Step 4 when creating the OAuth2 Client Credentials.

        
        
      
      
      
      
   
   
   3. GitHub Repo Creation.
      
      
      
      • If you already have a GitHub account you can skip this step , otherwise Signup for the new Account.
      
      
      • Once account is created, login to GitHub and Create Personal Access Token (classic). Copy the new token to your clipboard to use in the step 4 for 'Create Secure Parameter'.
      
      
      • Create a new private repository with appropriate name in GitHub, example
        

        integration-content-repo or sap-cloud-integration-artifacts

        
        
      
      
      
      
   
   
   4. Create the User Credentials.
      
      
      
      • Manage Security Material -> Add Create OAuth2 Client Credentials. Deploy the credentials.
        
      
      
      • [Optional] Repeat the above steps for the Target tenant (required only for Transport Integration Contents iflow).
      
      
      • Manage Security Material -> Add Create Secure Parameter.
      
      
      
      
   
   
   5. [Optional] Microsoft Teams Channel Webhook configuration - Configure only in case you want to send Notification to your MS Teams Channel.
      
      
      
      • Create Incoming Webhooks for your MS Teams Channel. Copy and store the webhook address.
      
      
      
      
   
   
   6. [Optional] You can download the test packages from the  GitHub repository sap-cloud-integration-artifacts, which you can readily use for testing. You can explore various combinations within each of the test packages and integration flows and try out the overall solution in your trial or non-production tenant. While you can also use your own test packages and integration flows, these are provided for quick reference to get you started.
   
   
   
   

Configuration

If the prerequisites are fulfilled, you’ll be ready to configure the external parameters of the integration flow!

Follow the configuration guide for each interface in the table below, which contains links to their respective configuration guides.

Integration Flow Name	Config Guide Link
Sync Integration Contents to GitHub	[Link]
Sync Integration Contents to GitHub - Delta	[Link]
Transport Integration Contents	[Link]
Upload Integration Contents from GitHub	[Link]

Step-by-Step Walk-through

Integration Flow Name	Walk-through/Demo Link
Sync Integration Contents to GitHub	[Link]
Sync Integration Contents to GitHub - Delta	[Link]
Transport Integration Contents	[Link]
Upload Integration Contents from GitHub	[Link]

What Next?

In the upcoming release, I have planned to implement the following features:

1. Addition of Artifact types: Script Collections and Message Mappings.


2. Optional ServiceNow Add-on to update ticket/incident number associated with change pushed to GitHub or next tenant.

Based on feedback and responses from the community, I am open to considering other features as well.

Summary

This was a humble attempt to create an alternative approach for building CI/CD by leveraging the inherent capabilities of SAP Cloud Integration, harnessing the power of Groovy, chaining APIs, and seeing how far I could go with it. I would also like to invite you all to share your feedback or thoughts in the comments section. I'm certain there are still opportunities for improvement and ideas for new rules or features that can be added to make this solution even better and more robust.