In this blog post, I want to describe the scope and usage of the new generation 2 ABAP operator “Read Data from SAP System” as part of ABAP Integration starting with SAP Data Intelligence 3.2 and SAP Data Intelligence Cloud 2110 . To demonstrate the functionality of the generation 2 operator, I will create a sample graph that is based on a real-world example that will be published in a separate blog post later.
First of all, however, I will briefly recap on the existing generation 1 operator scope and compare the different handling when you want to switch from an existing generation 1 graph to a new generation 2 graph in the context of CDS View extraction from SAP S/4HANA, ODP based extraction from SAP Business suite, SAP S/4HANA on Premise and SAP Business Warehouse systems as well as table-based extraction from SAP Business suite systems.
Before we get started with the generation 2 operator, I want to provide a brief summary of the existing Generation 1 operator scope of the ABAP integration with SAP Data Intelligence. Below you can find an overview of the different integration use cases using generation 1 operators:
*Minimum DMIS 2011 SP17 / DMIS 2018 SP02 depending on NetWeaver release, but it is highly recommended to use latest SP-level.
**Considering the SLT version dependency in the SLT installation guide: Link
For more information, see the following blog post from Karine Farias: https://blogs.sap.com/2021/08/19/beginners-guide-how-to-start-with-abap-integration-within-data-inte...
The main scope of Generation 1 ABAP Integration looks as follows:
In contrast, the figure below describes the current scope of the generation 2 “Read Data from SAP System” operator: It lists the systems that can be integrated with this operator and provides a summary of the high-level prerequisites in terms of minimum versions of the connected SAP ABAP source systems that I will further describe in the next paragraph below.
* Minimum DMIS 2018 SP06 with TCI or DMIS 2020 SP03 or newer
** Considering the SLT version dependency in the SLT installation guide: Link
Out of these SAP Notes, we can derive the following prerequisites for using generation 2:
Note: Please always check the general SAP Release Note for each system as mentioned in the previous paragraph to learn about all latest SAP Notes that might be required in addition.
For extracting CDS Views with the new generation 2 operator, the CDS Views need to be extraction enabled as well as enabled for change data capturing via the respective analytics annotations similar to the Generation 1 CDS Reader operator.
If you plan to use an SAP S/4HANA system (on-premise), you can not only use standard CDS Views that are extraction enabled, but you can also create custom CDS Views with the required analytics annotations and consume them with this operator as well. More information about the required analytics annotations is available in the following documentation and blog posts: https://help.sap.com/viewer/3a65df0ce7cd40d3a61225b7d3c86703/Cloud/en-US/55b2a17f987744cba62903e97dd... & the following block posts from Simon Kranig and Martin Boeckling : https://blogs.sap.com/2019/12/16/cds-based-data-extraction-part-ii-delta-handling/ & https://blogs.sap.com/2021/01/21/abap-cds-replication-in-data-intelligence/ .
If you are integrating CDS Views from SAP S/4HANA Cloud, you can only integrate standard CDS Views that are extraction enabled and C1 released. More information can be found here: https://help.sap.com/viewer/0f69f8fb28ac4bf48d2b57b9637e81fa/2111.501/en-US/b6fe6846e9c7488e8c7e6d42...
In addition, you may also check the latest SAP Note that includes the necessary security settings including roles & authorizations for the generation 2 operator: 3100673 .
If you want to perform table-based extraction, there are generally two different ways of extracting data similar to generation 1 operators leveraging SLT functionality via the DMIS add-on:
This new target system or target scenario is available in addition to the existing scenario “SAP Data Intelligence“ , which can still be used with the generation 1 operators.
In addition to CDS View and table-based based extraction the generation 2 Read Data from SAP System operator also offers the capability to extract data via the ODP interface similar to what is being offered with the generation 1 ODP Reader" operator.
Within ODP there are different so called ODP contexts through which data can be exposed for data extraction. In this case the following two ODP contexts are supported by the operator:
Note: Not supported in this scenario is the extraction via context ABAP_CDS, which is currently mainly used by SAP Business Warehouse to perform data extraction directly out of SAP S/4HANA on premise.
Additionally, the extraction out of ODP requires the ODP API version 2.0 as well as a minimum version of the SAP_BW component. Detailed list of prerequisites is briefly listed here :
where the DMIS version is as always depending on the NetWeaver release of the SAP System.
Whereas ODP as extraction interface is mainly used for SAP Business Warehouse systems, it is also offered in SAP Business Suite systems as well as (still) in SAP S/4HANA systems as well. In SAP S/4HANA the recommended extraction interface are CDS Views, but ODP is still supported as well with some restrictions that are documented in the following SAP Note.
The main pre-requisite of using ODP comes with the DMIS Add On when talking about SAP Business Suite and SAP Business Warehouse systems and a respective minimum version of SAP S/4HANA (>=1909 plus TCI). More information can be found in other sections of this blog as well as in the following SAP Note.
The capabilities for performing initial as well as delta load are also depending on the definition of the ODP object itself, which means that the object definition inside the source ABAP system needs to be configured for delta extraction in order to use it inside the generation 2 operator for implementing a delta load data replication inside a pipeline.
The below figure provides a more granular view on which kind of data sets (CDS View, table or ODP objects) can be extracted from different SAP systems via direct ingestion or via dedicated SLT server in case of table-based extraction to SAP Data Intelligence using the Generation 2 operator Read Data from SAP System:
* Extraction enabled CDS Views
** Considering the SLT version dependency in the SLT installation guide: Link
As already briefly mentioned in the footnote of some diagrams above, if you use a dedicated SLT server it is important that you consider the SLT version dependency between the DMIS version of the connected SAP source system and the dedicated SLT server. More information can be found in the SLT installation guide here:
https://help.sap.com/viewer/product/SAP_LANDSCAPE_TRANSFORMATION_REPLICATION_SERVER/3.0.06/en-US
Now we want to take a closer look on the generation 2 operator Read Data from SAP System by describing the main functionality as well as necessary configuration parameters.
Configuration Settings for Read Data from SAP System operator:
ABAP Connection – Select your SAP system
The connection created in the SAP Data Intelligence Connection Management (using connection type ABAP) against your SAP source system that meets the necessary prerequisites mentioned in the beginning of this blog post.
Version – Browsing of operator in the source SAP system
Similar to the generation 1 operators like SLT Connector, CDS Reader, and ODP Reader, the generation 2 operator is technically running in the connected SAP source system as part of the ABAP Pipeline Engine (APE). The APE provides the technical foundation for connecting your SAP system with SAP Data Intelligence. This means that the operator in SAP Data Intelligence is acting like a shell, which is pointing to the actual operator that is located in the SAP source system.
Object name - browsing the data set for data extraction (CDS view, table or ODP)
The generation 2 operator “Read Data from SAP System” is combining the functionality of the SLT Connector, the CDS Reader and ODP Reader operators in one operator. This means that you perform table-based, ODP based as well as CDS view-based extraction with the same operator. The availability of the source objects you can extract from a source system depends on the SAP system connection you are using in the ABAP connection parameter of the operator. Please check the previous paragraphs for the support of CDS view, ODP and table-based extraction from the different SAP source systems.
What is different compared to the generation 1 operators is that you are now able to browse and select your data set similar to how it is being done in the Metadata Explorer instead of entering the CDS view or table name manually as it was required with the generation 1 operators like CDS Reader or SLT Connector.
Transfer mode
The supported transfer modes for the generation 2 operator are similar to the ones provided by the generation 1 operators CDS Reader, ODP Reader and SLT Connector: initial load, replication (covering initial load + delta) and delta load.
Before moving on to the sample scenario, I would like to briefly describe two aspects that may be of interest to users who have worked with generation 1 operators:
First, in contrast to the generation 1 operators, the generation 2 operator currently does not allow for customization of the package size / records per roundtrip. Instead, it uses a default value of ~ 10 Mb (depending on the individual source data set, there may be differences in the actual size).
Second, it is no longer necessary f or an SAP Data Intelligence user to specify a Subscription ID, as this feature has been replaced with an optimized one that is now part of the snapshot feature in generation 2: When a user starts a graph with snapshots using the generation 2 operator “Read Data from SAP System”, the system automatically takes care of the subscription-related aspects in the background. We will see later what this looks like in an SAP Data Intelligence graph based on a sample scenario.
Now that’s it for Part 1 of my blog series. I hope I could give you some useful information about the overview of the topic of ABAP integration with SAP Data Intelligence using generation 2 operators.
In part 2 of this blog series, I will show you a concrete example scenario to better illustrate how this could be realized in generation 1 and generation 2, including some of the new features in generation 2, such as the pipeline resiliency feature.
Please feel free to add your feedback and question in the comments section of this blog.
First of all, however, I will briefly recap on the existing generation 1 operator scope and compare the different handling when you want to switch from an existing generation 1 graph to a new generation 2 graph in the context of CDS View extraction from SAP S/4HANA, ODP based extraction from SAP Business suite, SAP S/4HANA on Premise and SAP Business Warehouse systems as well as table-based extraction from SAP Business suite systems.
Recap and brief summary of generation 1 ABAP integration with SAP Data Intelligence
Before we get started with the generation 2 operator, I want to provide a brief summary of the existing Generation 1 operator scope of the ABAP integration with SAP Data Intelligence. Below you can find an overview of the different integration use cases using generation 1 operators:
*Minimum DMIS 2011 SP17 / DMIS 2018 SP02 depending on NetWeaver release, but it is highly recommended to use latest SP-level.
**Considering the SLT version dependency in the SLT installation guide: Link
For more information, see the following blog post from Karine Farias: https://blogs.sap.com/2021/08/19/beginners-guide-how-to-start-with-abap-integration-within-data-inte...
The main scope of Generation 1 ABAP Integration looks as follows:
- Integration of SAP S/4HANA Cloud and on-premise
Use the CDS Reader operator to integrate data out of extraction & delta enabled CDS views into a graph in SAP Data Intelligence using initial and delta mode. - Integration of SAP Business Suite systems and SLT with DMIS add-on
Mainly table-based extraction in initial and delta load mode leveraging SLT technology using the SLT Connector operator directly, for example, from SAP Business Suite systems or via a dedicated SLT server.
When directly integrating with SAP Business Suite systems (no separate SLT server), ODP-based integration of business extractors via ODP/ODQ framework is also possible using the ODP Reader operator in an SAP Data Intelligence graph. - Integration of SAP Business Warehouse (SAP BW) Systems via ODP interface
Integrate various SAP BW objects that can be exposed by the ODP interface into an SAP Data Intelligence graph, such as ADSO, DSO, or InfoProvider , by using the ODP Reader operator in initial and delta load mode. - Custom ABAP Integration
Usage of Custom ABAP Operator for use cases that go beyond the current scope of standard operators, and involve, for example, executing function modules.
Overview of ABAP Integration with generation 2 in SAP Data Intelligence
In contrast, the figure below describes the current scope of the generation 2 “Read Data from SAP System” operator: It lists the systems that can be integrated with this operator and provides a summary of the high-level prerequisites in terms of minimum versions of the connected SAP ABAP source systems that I will further describe in the next paragraph below.
* Minimum DMIS 2018 SP06 with TCI or DMIS 2020 SP03 or newer
** Considering the SLT version dependency in the SLT installation guide: Link
Detailed view on requirements for SAP ABAP systems to use generation 2 Operator:
- High-level Information is available in the central SAP Note for ABAP integration: 2890171
- More information is also available in the overview Release Note for each release:
- SAP S/4HANA 1909: 2830276
- SAP S/4HANA 2020: 2943599
- SAP S/4HANA 2021: 3085579
- SAP ECC / SLT with DMIS 2018: 3110660
- SAP SLT for SAP S/4HANA with DMIS 2020
- SAP Business Warehouse via DMIS AddOn
Out of these SAP Notes, we can derive the following prerequisites for using generation 2:
- SAP S/4HANA on Premise >= 1909 for CDS View extraction via embedded ABAP Pipeline Engine
- SAP S/4HANA 1909 with TCI Note 3105890
- SAP S/4HANA 2020 with TCI Note 3105880
- SAP S/4HANA 2021 with TCI Note 3115128
- SAP S/4HANA Cloud 2202 or higher for CDS view extraction via embedded ABAP Pipeline Engine
- SAP S/4HANA Cloud 2202 or later
- SAP Business Suite systems, SAP S/4HANA Foundation and SAP SLT systems where ABAP Pipeline Engine comes with the DMIS add-on
- Systems with NetWeaver Version 7.52 with DMIS 2018 SP06 & TCI 3110660 or newer
- SAP Business Warehouse Systems for ODP based extraction using DMIS version 2011 SP23 or DMIS 2018 SP08 or a higher level SP
- Systems with SLT for SAP S/4HANA using DMIS 2020 SP03 or newer (e.g. SAP S/4HANA Foundation)
- Systems with NetWeaver versions starting from 7.00 and lower than 52 using DMIS 2011
Note: These systems cannot be integrated directly with SAP Data Intelligence, but the connection can be established through a dedicated SLT server that is running on DMIS 2018 / DMIS 2020, and then the generation 2 operator can also be used. The prerequisites are the same as mentioned above. In this case, the dedicated SLT server is connected to SAP Data Intelligence and uses the DMIS 2011 based system as a source in the SLT configuration.
Please note that for these scenarios you need to consider the SLT version dependency that is available in the SLT installation guide: link.
Note: Please always check the general SAP Release Note for each system as mentioned in the previous paragraph to learn about all latest SAP Notes that might be required in addition.
Use Case: Extracting data out of CDS views with Generation 2 “Read Data from SAP System”
For extracting CDS Views with the new generation 2 operator, the CDS Views need to be extraction enabled as well as enabled for change data capturing via the respective analytics annotations similar to the Generation 1 CDS Reader operator.
If you plan to use an SAP S/4HANA system (on-premise), you can not only use standard CDS Views that are extraction enabled, but you can also create custom CDS Views with the required analytics annotations and consume them with this operator as well. More information about the required analytics annotations is available in the following documentation and blog posts: https://help.sap.com/viewer/3a65df0ce7cd40d3a61225b7d3c86703/Cloud/en-US/55b2a17f987744cba62903e97dd... & the following block posts from Simon Kranig and Martin Boeckling : https://blogs.sap.com/2019/12/16/cds-based-data-extraction-part-ii-delta-handling/ & https://blogs.sap.com/2021/01/21/abap-cds-replication-in-data-intelligence/ .
If you are integrating CDS Views from SAP S/4HANA Cloud, you can only integrate standard CDS Views that are extraction enabled and C1 released. More information can be found here: https://help.sap.com/viewer/0f69f8fb28ac4bf48d2b57b9637e81fa/2111.501/en-US/b6fe6846e9c7488e8c7e6d42...
In addition, you may also check the latest SAP Note that includes the necessary security settings including roles & authorizations for the generation 2 operator: 3100673 .
Use Case: Extracting data out of tables with generation 2 “Read Data from SAP System”
If you want to perform table-based extraction, there are generally two different ways of extracting data similar to generation 1 operators leveraging SLT functionality via the DMIS add-on:
- Direct ingestion from the source via DMIS add-on version 2018 or 2020
In this case, SAP Data Intelligence is directly connected to the actual SAP source system, e.g. SAP Business Suite system or SAP S/4HANA Foundation, for table-based extraction using a predefined SLT configuration.
For detailed information about the required versions, please check the paragraphs above.
- Ingestion via a dedicated SLT server via DMIS add-on
In this case, SAP Data Intelligence is connected to a dedicated SLT Server as a source system. Within this SLT Server, an SLT configuration can be created for table-based extraction from various different SAP sources that are connected via an RFC connection. This RFC connection needs to be specified as a source in the SLT configuration setup.
This new target system or target scenario is available in addition to the existing scenario “SAP Data Intelligence“ , which can still be used with the generation 1 operators.
Use Case: Extracting data out of ODP objects with Generation 2 “Read Data from SAP System”
In addition to CDS View and table-based based extraction the generation 2 Read Data from SAP System operator also offers the capability to extract data via the ODP interface similar to what is being offered with the generation 1 ODP Reader" operator.
Within ODP there are different so called ODP contexts through which data can be exposed for data extraction. In this case the following two ODP contexts are supported by the operator:
- ODP_SAPI (e.g. used by business extractors in SAP ECC systems)
- ODP_BW (mainly used by SAP Business Warehouse systems)
Note: Not supported in this scenario is the extraction via context ABAP_CDS, which is currently mainly used by SAP Business Warehouse to perform data extraction directly out of SAP S/4HANA on premise.
Additionally, the extraction out of ODP requires the ODP API version 2.0 as well as a minimum version of the SAP_BW component. Detailed list of prerequisites is briefly listed here :
- SAP S/4HANA 1909 or higher (with new TCI)
- SAP System with DMIS 2011 SP23 or higher level SP
- SAP System with DMIS 2018 SP08 or higher level SP
- SAP System with DMIS 2020 SP04 or higher level SP
where the DMIS version is as always depending on the NetWeaver release of the SAP System.
Whereas ODP as extraction interface is mainly used for SAP Business Warehouse systems, it is also offered in SAP Business Suite systems as well as (still) in SAP S/4HANA systems as well. In SAP S/4HANA the recommended extraction interface are CDS Views, but ODP is still supported as well with some restrictions that are documented in the following SAP Note.
The main pre-requisite of using ODP comes with the DMIS Add On when talking about SAP Business Suite and SAP Business Warehouse systems and a respective minimum version of SAP S/4HANA (>=1909 plus TCI). More information can be found in other sections of this blog as well as in the following SAP Note.
The capabilities for performing initial as well as delta load are also depending on the definition of the ODP object itself, which means that the object definition inside the source ABAP system needs to be configured for delta extraction in order to use it inside the generation 2 operator for implementing a delta load data replication inside a pipeline.
The below figure provides a more granular view on which kind of data sets (CDS View, table or ODP objects) can be extracted from different SAP systems via direct ingestion or via dedicated SLT server in case of table-based extraction to SAP Data Intelligence using the Generation 2 operator Read Data from SAP System:
* Extraction enabled CDS Views
** Considering the SLT version dependency in the SLT installation guide: Link
As already briefly mentioned in the footnote of some diagrams above, if you use a dedicated SLT server it is important that you consider the SLT version dependency between the DMIS version of the connected SAP source system and the dedicated SLT server. More information can be found in the SLT installation guide here:
https://help.sap.com/viewer/product/SAP_LANDSCAPE_TRANSFORMATION_REPLICATION_SERVER/3.0.06/en-US
Introduction & Usage of Generation 2 “Read Data from SAP System” Operator
Now we want to take a closer look on the generation 2 operator Read Data from SAP System by describing the main functionality as well as necessary configuration parameters.
Configuration Settings for Read Data from SAP System operator:
ABAP Connection – Select your SAP system
The connection created in the SAP Data Intelligence Connection Management (using connection type ABAP) against your SAP source system that meets the necessary prerequisites mentioned in the beginning of this blog post.
Version – Browsing of operator in the source SAP system
Similar to the generation 1 operators like SLT Connector, CDS Reader, and ODP Reader, the generation 2 operator is technically running in the connected SAP source system as part of the ABAP Pipeline Engine (APE). The APE provides the technical foundation for connecting your SAP system with SAP Data Intelligence. This means that the operator in SAP Data Intelligence is acting like a shell, which is pointing to the actual operator that is located in the SAP source system.
Object name - browsing the data set for data extraction (CDS view, table or ODP)
The generation 2 operator “Read Data from SAP System” is combining the functionality of the SLT Connector, the CDS Reader and ODP Reader operators in one operator. This means that you perform table-based, ODP based as well as CDS view-based extraction with the same operator. The availability of the source objects you can extract from a source system depends on the SAP system connection you are using in the ABAP connection parameter of the operator. Please check the previous paragraphs for the support of CDS view, ODP and table-based extraction from the different SAP source systems.
What is different compared to the generation 1 operators is that you are now able to browse and select your data set similar to how it is being done in the Metadata Explorer instead of entering the CDS view or table name manually as it was required with the generation 1 operators like CDS Reader or SLT Connector.
Transfer mode
The supported transfer modes for the generation 2 operator are similar to the ones provided by the generation 1 operators CDS Reader, ODP Reader and SLT Connector: initial load, replication (covering initial load + delta) and delta load.
Before moving on to the sample scenario, I would like to briefly describe two aspects that may be of interest to users who have worked with generation 1 operators:
First, in contrast to the generation 1 operators, the generation 2 operator currently does not allow for customization of the package size / records per roundtrip. Instead, it uses a default value of ~ 10 Mb (depending on the individual source data set, there may be differences in the actual size).
Second, it is no longer necessary f or an SAP Data Intelligence user to specify a Subscription ID, as this feature has been replaced with an optimized one that is now part of the snapshot feature in generation 2: When a user starts a graph with snapshots using the generation 2 operator “Read Data from SAP System”, the system automatically takes care of the subscription-related aspects in the background. We will see later what this looks like in an SAP Data Intelligence graph based on a sample scenario.
Conclusion
Now that’s it for Part 1 of my blog series. I hope I could give you some useful information about the overview of the topic of ABAP integration with SAP Data Intelligence using generation 2 operators.
In part 2 of this blog series, I will show you a concrete example scenario to better illustrate how this could be realized in generation 1 and generation 2, including some of the new features in generation 2, such as the pipeline resiliency feature.
Please feel free to add your feedback and question in the comments section of this blog.
- SAP Managed Tags:
7 Comments
