SAP Data Intelligence - Terminate a Graph after data load is complete in SAP HANA Table

[former_member704859]

Hi All,

Did you try graph terminator with HANA Client operator? I am having an issue in terminating the graph after the load is completed. My graph is as below:

Constant Generator -->> SAP HANA Client -->> Python Operator -->> SAP HANA Client -->> Graph Terminator

When I use graph terminator, it simply terminates the graph as soon as first set of records are loaded to hana table. Ideally, I would want all data to be loaded before graph is terminated.

Any inputs on how to achieve this?

[ThorstenHa]

Yes and it works perfectly fine. Tested it again:

SELECT * FROM TABLE. -> Write FILE -> Terminate and all datasets have been written (JSON including attributes)

Suppose there is a logic issue somewhere in your python code.

[former_member704859]

Thanks for your response thorsten.hapke . Python code is giving output in a dataframe all together. But data is getting loaded into HANA table in small chunks. After the 1st chunk is loaded in the table, graph terminator thinks all data is loaded and it terminates the graph.

It might work with small dataset as all data will get loaded in single attempt. But in case where data volume is high, data will be loaded in small batches and graph terminator should wait until all data is loaded.

Regards

Achin Kimtee

[ThorstenHa]

Good point.

I produced "batches" as output from HANA client and then of course after the first batch the graph terminates.

Have look to the attributes of HANA client. Example:

{"message.batchIndex":410,"message.batchSize":10,"message.batchSizeUnit":"rows","message.error":false,"message.lastBatch":false}

This you can use with the operator: "Message Filter" to check if "message.lastBatch":true. But you need to pass the attributes through your Python operators. Because you have 2 HANA Clients you might need to put some additional logic to your code.

[former_member704859]

Hi thorsten.hapke

I don't seem to find the attributes of HANA Client you mentioned as below.

{"message.batchIndex":410,"message.batchSize":10,"message.batchSizeUnit":"rows","message.error":false,"message.lastBatch":false}

I have done the proper configuration in HANA client but that does not help.

By "Message Filter", do you mean Message Key Store operator?

Regards

Achin Kimtee

[ThorstenHa]

Please do these steps:

1. Configure one HANA client with: "Output in batches": YES and "Output batch size (rows)": 1000 or whatever makes sense for the size of your table

2. Add a wiretab to the outport and check the output. There you should see attributes and data.

If the output is strange then you can send it to me via email: thorsten.hapke@sap.com

[cesarsan7]

Hi achinank ,

Hi Achin Kimtee , I hope you are well, exactly what you reported is happening to me, can you tell me how you could solve it, thank you very much.

My pipeline is the following SLT Operator V2 -> Wiretap -> Python Operator -> Write File -> Graph Terminator

(Write File use bach join = true)

Regards

César Sánchez Gutiérrez

[abhimanyu_sharma]

Hi Achin , have you got an solution for this? I am facing same issue .

[former_member688309]

Hello achinank,

As a troubleshooting, can you try to remove the last step of the graph - Graph Terminator? Then you can test it and if the Python code is executed successfully and all of the data sets are transferred, then you can try the steps, that Thorsten has proposed. If not, then you can check your code for any termination after the first record set. You can always stop the test graph from the Modeler, and then you can delete it (after it is completed). If you have issues, you can delete it using kubectl, first by finding it using

kubectl -n <NAMESPACE> get pods

and then

kubectl -n <NAMESPACE> delete pod <POD_NAME>

Kind regards,

MS

[former_member704859]

Hi mihail.sevdiev

My graph is working perfectly fine without graph terminator. But the only issue is I don't want to manually kill the graph every time especially it's not possible when the graphs are to be scheduled. I am looking to automate the complete process and thus need to find a way to terminate the graph automatically only after all the records are loaded to HANA table.

Regards

Achin Kimtee