Extensible Pipelines for SAP Continuous Integration and Delivery

[holger_partsch]

The SAP Continuous Integration and Delivery service just reached a new stage of its evolution: Up until now, the available pipelines were made of a fixed set of building blocks, such as: build, test, scan your code and deploy. You could choose which of those blocks you wanted for your pipeline, but you couldn't create your own.

That's about to change. Pipelines now have extension points that allow you to implement your own logic in the form of scripts.

In the next paragraph, I'll give some background on the why and the how. If you're only interested in how to use the new feature, feel free to skip it.

A Bit of History

When we productized the SAP Continuous Integration and Delivery service around 2020, the main quality we aimed for was simplicity, which has two main aspects for the service:

• enabling you to implement CI/CD (continuous integration and delivery) without owning any infrastructure, for example, build servers


• focusing on SAP development scenarios (for example, CAP, Fiori) to provide ready-made pipelines that you can use without any additional scripting

That made sense, because our target audience were customers at the beginning of their CI/CD journey. On the other end of spectrum were customers who had already invested in CI/CD. They were not our main target because we knew that CI/CD requirements are highly specific within each organization, and our service wouldn't have the flexibility to accommodate that.

Fast forward to the present. The service is well received by its target audience: Customers who only had manual processes before now use automated pipelines.

But we also received feedback that surprised us:

We like the Continuous Integration and Delivery service a lot, so we're going to migrate our complete development landscape (hundreds of projects) to the service. But could you please implement support for our Radar Scans?

As far as this story goes, Radar Scans are clearly a fictional example. However the point stands: for static code checking, we only support Sonar Scans. It's not likely that we'll support other code scanning tools in the near future, because to keep our configuration UIs simple, we decided to support only one mainstream tool for each area of functionality.

Now you can see the dilemma we are facing: How can we meet diverse customer requirements and keep the service simple at the same time?

Making Pipelines Extensible

The only scalable solution is to enable you to help yourself. To do so, we added extension points to the pipelines that allow you to run arbitrary commands and scripts. So it's scripts after all, you say? Yes, true. But the important thing is that you can get started without scripting and continue so as long as you stick to the mainstream stuff we support. Later, when you need that extra bit of functionality, scripts are the way to go.

Freestyle Configuration and Freestyle Steps

Now let's have a look at those extension points. Before adding freestyle steps, we introduced freestyle configuration. With this feature you can add arbitrary environment variables to all processes executed within a stage, and even inject secrets.

You can use freestyle configuration when configuring those stages which are scriptable by nature. The build stage is a typical example, because no matter which build tool you use (be it maven or npm), it's easy to integrate custom scripts into the build process. Freestyle configuration now unlocks lots of options. For an example on how powerful it is, see the blog post Accessing Private Registries.

But what about those stages that are not scriptable by nature? Recall the question: How can I do Radar if the service only provides Sonar?

To cover this, we added two extension points per stage, called runFirst and runLast.

The semantics are as expected:

• the runFirst command is executed before everything else in that stage


• the runLast command is executed after everything else in that stage

If you decide to enable the preconfigured steps in the stage (for example, Sonar scan), the runFirst and runLast commands can be used for preparation and post-processing.

If you decide to do something else instead, use one of the extension points to execute it.

Sure, a single command is often not enough, but you can just add a more complex script to your source repository and use the command to start it.

If this makes you curious, check out the official documentation.

Container Images

Being able to execute a script sounds great, but you might want to know in which environment it runs and which tools you can use. Like everything else in the Continuous Integration and Delivery service, custom commands are executed in a Linux container.

As the base image for that container, we decided for an image containing the standard Linux shell tooling plus a recent node runtime and npm.

Because JavaScript is the most popular implementation language on SAP Business Technology Platform, we decided to go for node as a scripting language. Using npm, you can easily bootstrap additional dependencies.

Chances are you're not happy with the choice we've made. Because, obviously, lua is the only acceptable scripting language. But don't despair. Just get in contact with us and tell us about your requirements, because the plan is to make the container image configurable. At first, it might be a fixed list of images maintained by us. But eventually, we want to enable you to "bring-your-own-image", despite some challenging compliance aspects that it might bring along.

Conclusion

Now you know how we make pipelines in SAP Continuous Integration and Delivery more flexible and our motivation behind it. If you have remarks or suggestions, please comment or contact me directly.