The SAP Data Hub Pipeline Engine offers different ways to serve a RESTful API from within a pipeline. In general, this can be achieved by using one of the predefined SAP Data Hub Pipeline Operators or by implementing a custom operator that exposes a REStful server:
All three approaches have their advantages and disadvantages, however, one challenge that all three have in common is providing a stable endpoint for the server that can be accessed from outside of the Kubernetes cluster.
During runtime, pipeline operators are executed within Docker containers and without further adjustments, the ports exposed by each operator are only reachable within the Docker container or the Kubernetes Pod resp. Moreover, the Kubernetes Node on which a Pod is scheduled is not stable, i.e. the Kubernetes Pod IP address changes once the corresponding SAP Data Hub Pipeline is restarted. This has to be considered when the RESTful service running in a pipeline shall be accessible from outside the Kubernetes cluster.
In the following, I will explain three different ways of implementing a RESTful service using SAP Data Hub Pipelines and how these services can be accessed externally. The functionality described in this blog is based on SAP Data Hub Version 1.0 SPS03.
In SAP Data Hub Version < 2.5.
The predefined HTTP Server operator starts an HTTP server on a configurable port. It handles GET and POST requests received from a configurable handler path. The operator supports the following HTTP methods:
The following pipeline shows how the HTTP Server Operator can be used to handle GET- and POST-requests that are sent from outside the cluster:
The pipeline consists of the following operators:
This operator starts an HTTP server on port 7070 and serves GET- and POST-requests on handle path "/":
All data received from the operator ToBlob Converter is returned in the response of the GET-request and the data received in the body of a POST-request is forwarded to the operator Wiretap.
The Constant Generator sends (once) the JSON string {"some":"response"} to the input port of the HTTP Server operator:
This string is returned by the HTTP Server operator in the response body of all GET-requests.
The operator Reverse Proxy is basically a Terminal operator with a custom route of type rproxy that forwards all external requests on path /service to the HTTP Server listening on localhost:7070:
(All operators are running in the same default group and therefore within the same Pod/Container). Without the reverse proxy route specified above, the HTTP Server would not be reachable from outside the Pod or the Kubernetes cluster resp.:
The Wiretap operator is just used to display the payload that is sent via the body of the POST requests.
When running the pipeline, you can discover the URL to the Reverse Proxy by opening the UI of the Terminal operator:
The URL of the Terminal has the following structure:
https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal<id>/
The URL changes with every restart of the pipeline. Therefore, you have to lookup and change the URL in the client that interacts with the HTTP server everytime the pipeline is stopped and started.
By appending the Path that was set in the configuration of the Reverse Proxy operator above, you can obtain the URL of the HTTP Server:
https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal<id>/service
With that in place, you can test the HTTP Server, e.g. using Postman:
The GET-request returns the JSON-string that was specified in the Constant Generator operator above.
Postman example:
Corresponding Curl command:
The authentication token can be obtained via the following URL with basic authentication:
The POST-Request returns an empty body with HTTP status code 200. The JSON string in the body of request is {"send":"data"} .
Postman example:
Corresponding Curl command:
The authentication token can be obtained via the following URL with basic authentication:
Once the POST-Request is sent to the server, you should see the following output in the Wiretap Operator:
This example shows how the HTTP Operator can be used to receive data via a POST-request and how to return data to a GET-request. With the Terminal operator and custom reverse proxy route, you can route external requests to the HTTP Server. However, the disadvantage is that the URL is not stable and changes upon restart of the pipeline. In the next example, you will learn how to implement a static endpoint using the OpenAPI Servlow operator.
The OpenAPI Servlow Operator currently provides the most convenient way of implementing a RESTful service in an SAP Data Hub Pipeline. The operator is suitable for providing services described in a Swagger/OpenAPI document but can also be used for unspecified HTTP services. The component runs within the web container of the SAP Data Hub Pipeline Engine and inherits its security configuration. More concretely, this operator is configured to start its REST endpoint relative to the Pipeline Modeler's service path at /openapi/service/basePath/. The clear advantage is, that this URL stays stable, even after a pipeline restart.
There are two predefined example pipelines shipped with the SAP Data Hub that illustrate the usage of the OpenAPI Servlow Operator very nicely:
In the following, I explain the usage of the OpenAPI Servelow Operator without Swagger document (second example pipeline):
The pipeline consists of the following operators:
When a request matching the configured basePath /samples/plain_greeter arrives via HTTP, an output message is generated and sent to the output port to the connected Wiretap operator. The OpenAPI Servlow operator supports both request-response and one-way message exchange patterns (MEPs). When the operator is running in the request-response mode, it waits for the response up to the specified timeout of 300000 ms. If the response message is returned to this operator over the engine's response callback mechanism, it is returned to the HTTP caller. Otherwise, an error is returned to the HTTP caller:
The output message generated by OpenAPI Servlow operator depends on whether it is configured in the plain mode (i.e., no swagger document is specified) or the swagger-driven mode (i.e., a swagger document is specified). In the plain mode, the headers and body of the incoming HTTP request will be directly transferred to the output message. In contrast, in the swagger-driven mode, the matching operation to the incoming HTTP request will be determined and the corresponding parameters for that operation will be extracted. Subsequently, these operation specific parameters will be transferred to the output message.
The Wiretap operator is used to display the messages that are sent by the OpenAPI Servlow operator to the Greeter operator.
This is a Javascript Operator that receives the messages from the OpenAPI Servlow operator indirectly via the Wiretap operator. The code below shows how it is used to handle:
The API function $.sendResponse() is used to send a response to the OpenAPI Servlow operator via the engine's response callback mechanism.
Please note that in SAP Data Hub 1.0 SPS03, the Javascript operator is the only way to use the response callback mechanism. In future releases of SAP Data Hub, there will be dedicated operators for invoking the response callback also outside of the Javascript operator.
When running the pipeline, the service exposed by the OpenAPI operator can be reached via the following static URL:
https://<host>:<vsystem-port>/app/pipeline-modeler/openapi/service/<basePath>/
With that in place, you can test the OpenAPI Servlow based web service, e.g. using Postman:
Postman example:
Corresponding Curl command:
The authentication token can be obtained via following URL with basic authentication:
Postman example:
Corresponding Curl command:
The authentication token can be obtained via following URL with basic authentication:
Once the POST-Request is sent to the server, you should see the following output in the Wiretap Operator which is then handled by the Javascript operator accordingly.
As you can see, it is very easy to handle various kinds of HTTP requests with the OpenAPI Servlow operator. The second pre-shipped example graph com.sap.demo.openapi.server.plain_greeter demonstrates an even more convenient method for implementing the server-side component by providing a Swagger specification in the operator configuration.
Last but not least you can also embed your own RESTful service in an operator, e.g. using Flask in Python or Node.JS. Please stay tuned for an upcoming description.
- HTTP Server Operator: This operator starts an HTTP server on a configurable port.
- OpenAPI Servlow Operator: A convenient server-side component that is suitable for providing services described in Swagger/OpenAPI or unspecified HTTP services.
- Custom Operator: For example Python (using Flask).
All three approaches have their advantages and disadvantages, however, one challenge that all three have in common is providing a stable endpoint for the server that can be accessed from outside of the Kubernetes cluster.
During runtime, pipeline operators are executed within Docker containers and without further adjustments, the ports exposed by each operator are only reachable within the Docker container or the Kubernetes Pod resp. Moreover, the Kubernetes Node on which a Pod is scheduled is not stable, i.e. the Kubernetes Pod IP address changes once the corresponding SAP Data Hub Pipeline is restarted. This has to be considered when the RESTful service running in a pipeline shall be accessible from outside the Kubernetes cluster.
In the following, I will explain three different ways of implementing a RESTful service using SAP Data Hub Pipelines and how these services can be accessed externally. The functionality described in this blog is based on SAP Data Hub Version 1.0 SPS03.
HTTP Server Operator
In SAP Data Hub Version < 2.5.
The predefined HTTP Server operator starts an HTTP server on a configurable port. It handles GET and POST requests received from a configurable handler path. The operator supports the following HTTP methods:
- GET: Send JSON-formatted data as a response to a GET request. The data included in the response is all the data that was received via the input port of the operator. There is no way to reset the response body or to send a different response based on parameters in the GET request. For this reason, the GET-method is currently only suited for scenarios where a static response body shall be returned.
- POST: Allows to receive the JSON-formatted body of a POST request. The body is directly forwarded to the output port of the operator and upon success, an empty body with status code 200 is returned. It is not possible to send a different body in the response of the POST-request. Therefore, this method is only suitable for scenarios where data is sent to a pipeline whereas acknowledgment based on the content is not required.
Example Pipeline using HTTP Server Operator
The following pipeline shows how the HTTP Server Operator can be used to handle GET- and POST-requests that are sent from outside the cluster:
The pipeline consists of the following operators:
HTTP Server
This operator starts an HTTP server on port 7070 and serves GET- and POST-requests on handle path "/":
All data received from the operator ToBlob Converter is returned in the response of the GET-request and the data received in the body of a POST-request is forwarded to the operator Wiretap.
Constant Generator
The Constant Generator sends (once) the JSON string {"some":"response"} to the input port of the HTTP Server operator:
This string is returned by the HTTP Server operator in the response body of all GET-requests.
Reverse Proxy
The operator Reverse Proxy is basically a Terminal operator with a custom route of type rproxy that forwards all external requests on path /service to the HTTP Server listening on localhost:7070:
(All operators are running in the same default group and therefore within the same Pod/Container). Without the reverse proxy route specified above, the HTTP Server would not be reachable from outside the Pod or the Kubernetes cluster resp.:
Wiretap
The Wiretap operator is just used to display the payload that is sent via the body of the POST requests.
Interaction with the HTTP Server
When running the pipeline, you can discover the URL to the Reverse Proxy by opening the UI of the Terminal operator:
The URL of the Terminal has the following structure:
https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal<id>/
- host: The hostname of the System Management service.
- vsystem-port: The port of the System Management service.
- graph-id: Automatically generated unique if of the pipeline instance.
- id: Automatically generated number representing the Terminal instance.
The URL changes with every restart of the pipeline. Therefore, you have to lookup and change the URL in the client that interacts with the HTTP server everytime the pipeline is stopped and started.
By appending the Path that was set in the configuration of the Reverse Proxy operator above, you can obtain the URL of the HTTP Server:
https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal<id>/service
With that in place, you can test the HTTP Server, e.g. using Postman:
GET-Request:
The GET-request returns the JSON-string that was specified in the Constant Generator operator above.
Postman example:
Corresponding Curl command:
curl --request GET \
--url https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal1/s... \
--header 'cookie: Authorization="Bearer <token>"' \
--insecure
The authentication token can be obtained via the following URL with basic authentication:
curl -vk -u "tenant\\user:password" https://<host>:<vsystem-port>/auth/loginPOST-Request:
The POST-Request returns an empty body with HTTP status code 200. The JSON string in the body of request is {"send":"data"} .
Postman example:
Corresponding Curl command:
curl --request POST \
--url https://<host>:<vsystem-port>/app/pipeline-modeler/service/v1/graphs/<graph-id>/operator/terminal1/s... \
--header 'referer: https://<host>:<vsystem>/app/pipeline-modeler/' \
--header 'cookie: Authorization="Bearer <token>"' \
--header 'x-requested-with: XMLHttpRequest' \
--data '{"send":"data"}' \
--insecureThe authentication token can be obtained via the following URL with basic authentication:
curl -vk -u "tenant\\user:password" https://<host>:<vsystem-port>/auth/loginOnce the POST-Request is sent to the server, you should see the following output in the Wiretap Operator:
This example shows how the HTTP Operator can be used to receive data via a POST-request and how to return data to a GET-request. With the Terminal operator and custom reverse proxy route, you can route external requests to the HTTP Server. However, the disadvantage is that the URL is not stable and changes upon restart of the pipeline. In the next example, you will learn how to implement a static endpoint using the OpenAPI Servlow operator.
OpenAPI Servlow Operator
The OpenAPI Servlow Operator currently provides the most convenient way of implementing a RESTful service in an SAP Data Hub Pipeline. The operator is suitable for providing services described in a Swagger/OpenAPI document but can also be used for unspecified HTTP services. The component runs within the web container of the SAP Data Hub Pipeline Engine and inherits its security configuration. More concretely, this operator is configured to start its REST endpoint relative to the Pipeline Modeler's service path at /openapi/service/basePath/. The clear advantage is, that this URL stays stable, even after a pipeline restart.
Example Pipeline using the OpenAPI Servlow Operator
There are two predefined example pipelines shipped with the SAP Data Hub that illustrate the usage of the OpenAPI Servlow Operator very nicely:
- com.sap.demo.openapi.server.greeter: Example providing a demo greeter service with its swagger document.
- com.sap.demo.openapi.server.plain_greeter: Example providing a demo greeter service with no swagger document.
In the following, I explain the usage of the OpenAPI Servelow Operator without Swagger document (second example pipeline):
The pipeline consists of the following operators:
OpenAPI Servlow
When a request matching the configured basePath /samples/plain_greeter arrives via HTTP, an output message is generated and sent to the output port to the connected Wiretap operator. The OpenAPI Servlow operator supports both request-response and one-way message exchange patterns (MEPs). When the operator is running in the request-response mode, it waits for the response up to the specified timeout of 300000 ms. If the response message is returned to this operator over the engine's response callback mechanism, it is returned to the HTTP caller. Otherwise, an error is returned to the HTTP caller:
The output message generated by OpenAPI Servlow operator depends on whether it is configured in the plain mode (i.e., no swagger document is specified) or the swagger-driven mode (i.e., a swagger document is specified). In the plain mode, the headers and body of the incoming HTTP request will be directly transferred to the output message. In contrast, in the swagger-driven mode, the matching operation to the incoming HTTP request will be determined and the corresponding parameters for that operation will be extracted. Subsequently, these operation specific parameters will be transferred to the output message.
Wiretap
The Wiretap operator is used to display the messages that are sent by the OpenAPI Servlow operator to the Greeter operator.
Greeter
This is a Javascript Operator that receives the messages from the OpenAPI Servlow operator indirectly via the Wiretap operator. The code below shows how it is used to handle:
- GET-Requests sent to the relative path /samples/plain_greeter/v1/ping
- POST-Requests sent to the relative path /samples/plain_greeter/v1/echo
$.setPortCallback("input",onInput);
// for simplicity, no
var count = 0;
var totalgreeted = 0;
var greeted = {};
function isByteArray(data) {
return (typeof data === 'object' && Array.isArray(data)
&& data.length > 0 && typeof data[0] === 'number')
}
function onInput(ctx,s) {
var msg = {};
var inbody = s.Body;
var inattributes = s.Attributes;
// convert the body into string if it is bytes
if (isByteArray(inbody)) {
inbody = String.fromCharCode.apply(null, inbody);
}
// just send a copy of the request to the output for someone else (e.g., if port output is connected)
msg.Attributes = {};
for (var key in inattributes) {
msg.Attributes[key] = inattributes[key];
}
msg.Body = inbody;
// pass a copy to the output if connected
if ($.output != null) {
$.output(msg);
}
// send the response
var reqmethod = inattributes["openapi.method"];
var reqpath = inattributes["openapi.request_uri"];
var resp = {};
resp.Attributes = {};
// as there is no swagger spec configured to specify the responese content-type, set it here
resp.Attributes["openapi.header.content-type"] = "application/json";
switch (reqpath) {
case "/samples/plain_greeter/v1/ping":
resp.Body = {"pong": count++};
$.sendResponse(s, resp, null);
break
case "/samples/plain_greeter/v1/echo":
resp.Body = {"echo": inbody};
$.sendResponse(s, resp, null);
break;
default:
$.sendResponse(s, null, Error("Unexpected operation at " + reqpath))
break;
}
}
The API function $.sendResponse() is used to send a response to the OpenAPI Servlow operator via the engine's response callback mechanism.
Please note that in SAP Data Hub 1.0 SPS03, the Javascript operator is the only way to use the response callback mechanism. In future releases of SAP Data Hub, there will be dedicated operators for invoking the response callback also outside of the Javascript operator.
Interaction with the OpenAPI Servlow Operator
When running the pipeline, the service exposed by the OpenAPI operator can be reached via the following static URL:
https://<host>:<vsystem-port>/app/pipeline-modeler/openapi/service/<basePath>/
- host: The hostname of the System Management service.
- vsystem-port: The port of the System Management service.
- basePath: The relative base path configured in the OpenAPI Servlow operator.
With that in place, you can test the OpenAPI Servlow based web service, e.g. using Postman:
GET-Request:
Postman example:
Corresponding Curl command:
curl --request GET \
--url https://<host>:<vsystem-port>/app/pipeline-modeler/openapi/service/samples/plain_greeter/v1/ping
\
--header 'cookie: Authorization="Bearer <token>"' \
--insecure
The authentication token can be obtained via following URL with basic authentication:
curl -vk -u "tenant\\user:password" https://<host>:<vsystem-port>/auth/loginPOST-Request:
Postman example:
Corresponding Curl command:
curl --request POST \
--url https://<host>:<vsystem-port>/app/pipeline-modeler/openapi/service/samples/plain_greeter/v1/echo \
--header 'referer: https://<host>:<vsystem>' \
--header 'cookie: Authorization="Bearer <token>"' \
--header 'x-requested-with: XMLHttpRequest' \
--data 'testme' \
--insecureThe authentication token can be obtained via following URL with basic authentication:
curl -vk -u "tenant\\user:password" https://<host>:<vsystem-port>/auth/loginOnce the POST-Request is sent to the server, you should see the following output in the Wiretap Operator which is then handled by the Javascript operator accordingly.
As you can see, it is very easy to handle various kinds of HTTP requests with the OpenAPI Servlow operator. The second pre-shipped example graph com.sap.demo.openapi.server.plain_greeter demonstrates an even more convenient method for implementing the server-side component by providing a Swagger specification in the operator configuration.
Custom Operator
Last but not least you can also embed your own RESTful service in an operator, e.g. using Flask in Python or Node.JS. Please stay tuned for an upcoming description.
- SAP Managed Tags:
15 Comments
