Customized SAP Data Intelligence Deployment cycle using Github Action Workflows

[patrobibeksap]

Introduction:

This is going to be my first blog on SAP Data Intelligence and quite excited about sharing with all my SAP colleagues/members working on this tool.

This document overviews about deployment life cycle management for SAP Data Intelligence using CI/CD methodology. Since SAP DI doesn’t provide any proprietary tools for deployments as of the current DI version 2303.15.28.

Business Pain Points:

• There is no SAP proprietary solution for deployment of SAP DI pipelines and operators using CI/CD approach


• Exporting and Importing files into Production systems are restricted in some of the client deployments


• Manual method of deploying DI objects is susceptible to human mistakes

Business Value Proposition:

• Implementation of CI/CD throughout Development, testing and Production phases of SAP DI development.


• Automatic deployment of DI pipelines and operators without any manual intervention


• Review of the DI objects when pushed for deployment to check its completeness and inform developers/object owners about it.


• Authentication based approach where the DI objects are reviewed and authenticated at every level to discard any unintended changes to get moved between different systems.

Deployment Workflow Path:

The following picture shows the path how the objects are moved starting from development till getting it deployed into Production.

The following points to be noted here:

• Develop, Test and Master branches are protected so that no one can able to push to it directly.


• The changes can be moved from Feature branches to develop, test and Master separately which helps us independently deploy DI objects in case required.


• Each workflows are event based triggered like based on pull or push requests


• Each workflow can identify and deploy only new DI pipelines or changes pipeline rather than picking all from the repository. This is the strength of this pipeline which avoids re-deployment of unchanged DI objects into test and production systems.

Design of deployment workflow

The workflows are designed using Git Action commands. These action commands are branch specific and execution is based on event (Push or Pull requests). This workflow can identify the DI pipelines/operator files which are modified or newly created and trigger deployment only for those objects.

The sample code here can be referenced for designing your deployment workflows.

name: Deployment of DI objects to DEVELOPMENT tenant



on: 

  push:

    branches: [<Github development branch name>]



jobs:

  setup:

    runs-on: ubuntu-latest

    outputs:

      matrix: ${{steps.list_file.outputs.value}}

      

    steps:

      - name: check out repo

        uses: actions/checkout@v2



      - name: Get Diff Action

        uses: technote-space/get-diff-action@v6

        with:

          format: JSON



      - name: Filter files for graphs and operators only  

        run: |

          res=$(echo '${{ env.GIT_DIFF }}' | jq '.[] | select(. | contains("graph.json") or contains("operator.json"))' | jq -Rsc '. / "\n" - [""]')

          echo "filtered_op=$res" >> $GITHUB_ENV

        shell: bash



      - name: Passing the final output to build

        id: list_file

        run: |

          res=$(echo '${{ env.filtered_op}}' | jq '.[]')

          if [ -z $res ]; then

            echo '::set-output name=value::["NO DEPLOYMENT OBJECTS FOUND"]'

          else 

            echo '::set-output name=value::${{ env.filtered_op }}'

          fi        

        

  build:

    needs: [setup]

    runs-on: windows-latest

    strategy: 

      max-parallel: 1

      matrix: 

        value: ${{fromJson(needs.setup.outputs.matrix)}}

   

    steps:

      - name: Checking out your current git repository

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        uses: actions/checkout@v2



      - name: Capturing the solution name to bundle

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: echo "obj_name=$(echo ${{ matrix.value }} | awk 'BEGIN{FS=OFS="/"} {print $--NF}')" >> $GITHUB_ENV

        shell: bash



      - name: Finding the Source directory

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          echo "source_dir=$(echo ${{matrix.value}} | awk 'BEGIN{FS=OFS="/"}NF--')" >> $GITHUB_ENV

        shell: bash

                

      - name: Defining the target directory for solution

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          echo "target_dir=$(echo ${{ env.obj_name }}/content/files/vflow/${{env.source_dir}})" >> $GITHUB_ENV

        shell: bash



      - name: Building the solution directory structure 

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: | 

          mkdir -p ${{env.target_dir}}

          cp -R ${{env.source_dir}}/* ${{env.target_dir}}/

          cp -R ${{env.source_dir}}/manifest.json ${{ env.obj_name }}/

        shell: bash



      - name: Log-in to DI through vctl

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: vctl\vctl.exe login ${{secrets.<DEV INSTANCE URL GIT SECRET>}} --user-cert ${{secrets.<bundle.pem GIT SECRET>}} ${{secrets.<key.pem GIT SECRET>}}



      - name: Bundle the solution into zip file in current working directory

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: vctl\vctl.exe solution bundle ${{env.obj_name}}



      - name: Capture the solution version post bundle

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          echo "zip_name=$(ls | sed -n 's/\.zip$//p')" >> $GITHUB_ENV

        shell: bash



      - name: Get list of solutions from the strategy 

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          vctl\vctl.exe strategy get ${{secrets.<default DI strategy name in GIT SECRET>} -o "json" | Out-File ./input.json

      

      - name: Look up for this solution version if available in the strategy

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          jarray=$(cat ./input.json | jq '.layers')

          obj_arr=$(echo $jarray | jq -c -r '.[]')

          echo "str_exist_flag=$(echo "NO")" >> $GITHUB_ENV

          flag=0

          for obj in ${obj_arr[*]}

          do

            val=$(echo $obj)

            if [ "$val" = "${{env.zip_name}}" ]; then

              echo "str_exist_flag=$(echo "YES")" >> $GITHUB_ENV

              flag=1

            fi

          done

          if [ $flag -eq 1 ]; then

            echo "This solution ${{env.zip_name}} version already exists in strategy."

          else 

            echo "This solution ${{env.zip_name}} version doesn't exists in strategy."

          fi

        shell: bash



      - name: Remove the solution version from strategy if already exists

        if: ${{ env.str_exist_flag == 'YES' }}

        run: vctl\vctl.exe strategy remove ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}



      - name: Get list of all solutions from the tenant 

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          vctl\vctl.exe solution list -o "json" | Out-File ./solution.json

      

      - name: Look up for this solution version if available in the tenant

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: |

          obj_arr=( $(jq -c -r '.[] | (.Manifest.name) + "-" + (.Manifest.version)' solution.json) )

          echo "sol_exist_flag=$(echo "NO")" >> $GITHUB_ENV

          flag=0

          for obj in ${obj_arr[*]}

          do

            val=$(echo $obj)

            if [ "$val" = "${{env.zip_name}}" ]; then

              echo "sol_exist_flag=$(echo "YES")" >> $GITHUB_ENV

              flag=1

            fi

          done

          if [ $flag -eq 1 ]; then

            echo "This solution ${{env.zip_name}} version already exists in solution list."

          else 

            echo "This solution ${{env.zip_name}} version doesn't exists in solutions."

          fi

        shell: bash



      - name: Delete the existing solution version from tenant if already exists

        if: ${{ env.sol_exist_flag == 'YES' }}

        run: vctl\vctl.exe solution delete $(echo ${{env.zip_name}} | cut -d "-" -f 1) $(echo ${{env.zip_name}} | cut -d "-" -f 2)



      - name: Uploading updated solution version to the tenant

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: vctl\vctl.exe solution upload ${{env.zip_name}}.zip



      - name: Adding updated solution version to the strategy

        if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}

        run: vctl\vctl.exe strategy add ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}

Expected learnings from this blog:

• This is a ready to use template which can be reused in DI project deployments with CI/CD approach.


• With the use of this approach, manual handling of deployments like import/Export can be avoided.


• We can track the history of changes for any object using git commit history.

I encourage all readers to provide your comment and happy to hear back any suggestions/ improvements in this approach. This will help me to improve this deployment workflow.