SAP Data Intelligence starting graphs by REST API

[rajarshi_muhuri]

Hi

I have a couple of questions

1. We are trying to delta loads using the Gen 2 operator "Read Data From SAP System". Unlike Gen 1 operators , stooping the graph makes it lose the pointer and all changes made to source SAP system (while the graph is in stopped state ) is lost when the graph is restarted . So we found that if the graph is paused and un-paused , then deltas work correctly . we don't want to use replicate option as we don't want the graphs to run all the time . So if we un-pause for say 10 min and then pause it again . Will it bring all the changes in source system ( what if there are millions of rows of data that needs to be bought in and the transfer would take 30 min ) .

2. We are planning to pause and un-pause the graphs using the REST API . We would have a graph with python3 operator that first checks a snowflake table , for the last successful run of DI and load of data into snowflake tables . if all is good then it would un-pause the graph by the rest api . But that needs us to get the restart ID

/app/pipeline-modeler/service/v1/runtime/restartgroups/42a9eb5cd21e47ae8af02896ce601d50?action=resume

But the restart ID can only be obtained , after the graph is run . So how would be do this in production . as then we would have run the graphs in PROD first and then get the RESTART ID and then add them in the python3 graphs in DEV and migrate from DEV to QA to PROD . But since the Restart ID is unique in PROD , we wont be able to test in DEV and QA .

Any suggestions ?

Rajarshi Muhuri

[SandeshK]

Hi Rajarshi,

I am also working on a similar scenario of yours, where we started using GEN2 and was wondering how can we stop and start an incremental read from SAP. Since, there is no concept of Subscription ID in GEN2, I was looking for a solution to it. Thanks to your discussion here, I got an idea how we can achieve this 🙂

A question, calling this API to pause and resume the graph, are you doing it from any other middleware tool or you are managing it on SAP DI itself. Do you have any reference to a blog about how this was implemented?

Regards,
Sandesh

[rajarshi_muhuri]

Hi Sandesh

I agree with your comment on subscription ID and I am finding GEN 2 a pain . Please do share your idea if you get something nice working . I am going to post this as a question too so someone else can answer if they have a solution .

To answer your question , I am not scheduling it via DI 's own scheduler , then I would not have needed the REST API calls anyway . My company has its own enterprise scheduler , that will call the REST API to start the graph on schedule . Inside the graph I will unpause the ETL graph and based on some logic will pause again .

I am still exploring the logic , but right now its something along the lines , of querying the ODQDATA table in S4HANA for delta counts ( exposed as AMDP --> CDS odata ) and then checking / tallying the target snowflake record counts , if record counts in target does not match , then run for another 10 min and then check .

.

[ansharm71]

Hi Rajarshi,

I have implemented one solution to stop the Gen1 graph automatically in delta mode after loading records from CDS views. Solution is based on trigger file concept. Basically , sending a stop signal to terminate the graph using trigger file once load is completed.

But this logic is implemented using a python operator .

Below is the Json file of graph for your reference

Thanks,

Ankit

[rajarshi_muhuri]

Hi Ankit

We had moved away from GEN 1 to GEN 2 because we had heard that GEN 1 will eventually be deprecated . But with GEN 2 being such a pain ( even their python3 gen 2 operator is very convoluted ) . and given that you made gen1 delta work , I might just revisit gen 1 .

Thanks a lot for the json file . I really appreciate your help .

rishi

[rajarshi_muhuri]

Hi ankit , can you contact me at rmuhuri at yahoo dot com . I want to share some info with you

[Arna]

Hi @rajarshi_muhuri , Please let me know the answer to your first question(Using the Gen 2 operator to load delta). I have the same requirement. Have you got any solution to this?

Thanks in advance

[rajarshi_muhuri]

I got the graph started , for some reason , normal authentication does not work . After I used Certificate based authentication it worked . I also had to alter the payload .

{ "src": "PY3_OP", "name": "x", "executionType": "", "configurationSubstitutions": {}, "async": true, "autoRestartConfig": { "maxRestartCount": 0, "restartDelaySeconds": 0 }, "snapshotConfig": { "enabled": true, "periodSeconds": 30 } }

But I see that it cant start multiple instances of the graph , so need to add another logic to query if its already running.

For cert based authentication , please see this blog

https://blogs.sap.com/2022/03/03/certificate-authentication-to-call-an-openapi-servlow-pipeline-on-s...

[rajeshps]

rajarshimuhuri Could you please provide your inputs here. Thanks!

HANA DB TO REST API SYNC CALL

[rajarshi_muhuri]

Hi Sachinandan

I was able to get the other ones working ( like pause , unpause ) , so I had not checked the start and stop . After your question , I just did but am getting "Forbidden Cross site Request " . Please update me if you can get it working at your end

so my body was :

{
   "src": "PY3_0P",
   "name": "x",
   "executionType": "flow",
   "configurationSubstitutions": {},
   "async": true,
   "autoRestartConfig": {
      "maxRestartCount": 2,
      "restartDelaySeconds": 180,
      "resetTimeThreshold": "string"
   },
   "snapshotConfig": {
      "enabled": true,
      "periodSeconds": 30
   }
}

[rajarshi_muhuri]

Thanks Jim

sorry for the confusion about dev and QA

I had meant that I get the restart ID = 42a9eb5cd21e47ae8af02896ce601d50 when the graph is running in PROD . At the time of writing this question , I was only getting the ID from the modeler screen , But later I found that I can query any API and get the RESTART ID . that solves my problem , I can query the graph name and hence get the ID and then use that to pause and unpause the graph

[former_member863078]

Hi Rajarshi,

I have a usecase where i want to push data from Snowflake to Hana. I am using Data Intelligence graph to pull data from Snow and push to Hana.

I wanted to trigger this graph using SAP DI Rest API, I was able to list graphs, and get details of graph through rest api, but could you send me an example on how to run a graph from rest api. The SAP documentation has a vague example which i could not test.

[jimgiffin]

Rajarshi,

You can see the restartID from the /v1/runtime/graphs API call (or if you have the graph handle, you can query that directly). In my limited testing, I believe the restart ID shows up in autoRestartInfo.id in the JSON payload returned for a graph. Obviously this only works for gen2 graphs.

I am unclear on what you would want to monitor in Prod from your Dev/QA stacks. I think leveraging the API to call Prod from Prod will insulate you from having to hardcode anything.