Exploring Integration Options in SAP Datasphere with the focus on using SAP extractors

[ClaudiaFiess]

Overview of the blog series

The main reason for this blog series was the realization that with the Introduction of SAP Datasphere there seems to be so many integration options when it comes to leveraging SAP Extractors (SAPI Data sources & ABAP CDS view extractors).

I wanted to illustrate options in a graphical way which can be useful as a starting point for implementing new use cases. From experience, besides determining the appropriate SAP Extractor to be used, another important factor to highlight is the capability of full versus delta load when it comes to data persistency.

Introduction to this blog

SAP Datasphere provides numerous integration possibilities when utilizing SAP Extractors.

As some of you might have already experienced, there are numerous factors and aspects to be considered when it comes to the decision whether you are going to leverage SAPI (Service API) extractors (also known as SAP Data Sources) or SAP ABAP CDS (Core Data Services) views or potentially both.

My intention with this post is to summarize and graphically present the Integration Options in SAP Datasphere utilizing SAP Extractors - this could then serve as a starting point for a project or implementation of new use cases or enhancements of existing use cases.

Most likely there will be many factors and aspects to be considered when it comes to the decision which SAP extractor to use, but in this post the emphasis will be on the possibilities of full and delta data load capability into SAP Datasphere.

As a side note, I highly recommend the creation of a design decision paper to document your options, considerations and reasons why particular design decisions were eventually chosen (or not). Especially as options might change over time, it is always great to be able to go back and look why certain things were done in a certain way at the time.

Scenario Overview:

The following 4 Integration scenarios are being covered during this blog series:

1. Direct Integration into SAP Datasphere [Blog 1 – focus within this Blog post]
2. Leveraging SAP Datasphere, SAP BW bridge [part of a future Blog post]
3. Leveraging SAP BW/4 Model transfer option [part of a future Blog post]
4. Leveraging ABAP connection integration with SAP BW systems [part of a future Blog post]

For each of the scenarios, I will be exploring integration option into SAP Datasphere by using Remote Table, Data Flow and Replication Flow functionality.

Please note that the following elements will not be included in this blog:

• Comprehensive explanation of the three integration alternatives (remote table, data flow, replication flow).
• Additional modeling options such as developing Dimension views, Fact views or Analytic Models on top of the remote or local table within SAP Datasphere.

Important Facts

Before looking at specifics of any of the scenarios it is important to highlight a few key facts for both types of extractors related to:

1. Availability
2. Operational Data Provisioning (ODP)
3. How to check: Primary Keys
4. How to check: Extractor Delta Capability
5. LIS Extractor specifics

Note:
Information in below Table 1 to Table 5 is relevant for the entire blog series.

Availability

SAPI Extractor	SAPI Extractors are available in SAP On-premise Business Suite systems (such as SAP ECC) as well as in SAP S/4HANA systems. It is important to know that not all SAPI extractors have been released for SAP S/4HANA (see SAP note 2500202) and in SAP S/4HANA cloud there are no SAPI extractors at all.
ABAP CDS view Extractor	SAP ABAP CDS view are only available in SAP S/4HANA On-premise (including PCE) as well as SAP S/4HANA Cloud systems, but not in any of the SAP Business suite systems (e.g. ECC). To be able to use SAP ABAP CDS views for exaction requires data extraction annotations (see SAP Help - CDS Views Enabled for Data Extraction)

                                             Table 1: Important Facts – Availability

Operational Data Provisioning (ODP)

SAPI Extractor	SAPI Extractors can be leveraged via the ODP framework. ODP enablement (see SAP note 2232584) is prerequisite for using SAPI Extractors in SAP BW/4HANA or SAP Datasphere (with and without SAP BW Bridge) via the ODP context for SAPI extractors (ODP_SAPI).
ABAP CDS view Extractor	ABAP CDS view extractors can be leveraged via the ODP framework. If the ABAP CDS view contains the required annotations for analytics, they can be used for extraction via the ODP context for ABAP CDS_VIEWS (ODP_CDS). See SAP Help - Transferring Data from SAP Systems via ODP (ABAP CDS Views)

                                             Table 2: Important Facts - Operational Data Provisioning (ODP)

How to check: Primary Keys

SAPI Extractor

There are different ways to find out whether the SAPI extractor has primary keys in the SAP source system: 1. In your Source system- use transaction RSA2 for SAPI Extractor and check tab “fields” for key fields. OR 2. a) In your Source system- use transaction SE16 / table ROOSOURCE and OLTPSOURCE equals SAPI Extractor(s): Use ROOSOURCE-EXSTRUCT and Run Transaction SE16 for table DD03L and TABNAME = ROOSOURCE-EXSTRUCT. Check KEYFLAG is ‘X’ for primary key(s). 2. b) In your Source system- use transaction SE16 / table ROOSFIELD and OLTPSOURCE equals the SAPI Extractor. Check KEYFLAG_DS is ‘X’ for primary key(s).

ABAP CDS view Extractor

Following Blog describes steps on how to analyse ABAP CDS views: Finding the right CDS Extractor in SAP S/4HANA Example (S/4HANA on premise): 1.      In Eclipse – display ABAP CDS view: 2.      In Source system – transaction SE16:

                                             Table 3: Important Facts - How to check: Primary Keys

How to check: Extractor Delta Capability

SAPI Extractor	In your Source system – use Transaction SE16 / table ROOSOURCE and OLTPSOURCE equals SAPI Extractor. Check that field DELTA is not empty.
ABAP CDS view Extractor	Following Blog describes steps on how to analyse ABAP CDS views: Finding the right CDS Extractor in SAP S/4HANA Example (S/4HANA on premise): use Transaction SE16: Table IXTRCTNENBLDVW (as mentioned in above blog)

                                             Table 4: Important Facts - How to check: Extractor Delta Capability

LIS Extractor specifics

SAPI Extractor

Standard SAP SAPI Extractors starting with the “2” in the technical names are commonly known as “Logistics Cockpit extractors.” (e.g., 2LIS_02_ITEM – Purchasing Line item). As preparation for a data load initialization as well as for a full data load, application specific setup tables need to be filled. Before filling the setup tables, it is recommended to delete the existing setup tables before executing new setup runs to avoid duplicate records for the same selections.

                                             Table 5: Important Facts – LIS Extractors

Let’s now focus on getting into the details of Scenario A, while the other scenarios will be covered in future blogs.

Scenario A: Direct Integration into SAP Datasphere

Direct integration using SAP Extractors with SAP Datasphere Native offers the capability for both - FULL and DELTA data load integration -provided certain prerequisites are met.

Direct Integration into SAP Datasphere using SAPI Extractors

As a general remark – please keep in mind that the SAPI Extractors have been developed for the purpose of data extraction into a BW system. As you can see below, it is possible to use these extractors directly for data extractions into SAP Datasphere but be mindful of prerequisites and possible limitations.
As an alternative you can either explore the option of leveraging an equivalent or similar ABAP CDS view extractor (Standard or custom) or use the SAPI extractor via the SAP BW bridge functionality [more details about SAP BW bridge functionality in the next Blog post].

As shown in Figure 3 – LIS extractors have the following limitations and therefore considered not suitable to be used for direct integration into SAP Datasphere:

• Federation and Full load would be using LIS Set-up tables, but these are usually are not updated on a regular basis.
• 2LIS extractors do not have primary keys defined.

The Recommendation for scenarios using LIS-Extractors [as per Figure 3] is therefore to leverage SAP Datasphere, SAP BW bridge functionality [more details about this scenario in the next Blog post].

In the following Table 6 (column “No.” refers to equivalent number in Figure 4) I have summarized the most relevant information/prerequisites. As functionality is changing and evolving, please check mentioned references for updates.

No.	Prerequisites & References
	Overall Prerequisites for SAPI Extractor: SAPI Extractor is ODP enabled. References: see SAP note 2232584 Release of SAP extractors for ODP replication (ODP SAPI)
[1]	Prerequisites: SAPI Extractor allows virtual access: In your SAP source system: Check table ROOSOURCE (via transaction se16): If Field VIRTCUBE has either entries 1 (without preaggregation) or 2 (with preaggregation), virtual access is generally available. References: SAP Help Portal: Import Remote Tables See Data Integration for ABAP Source Systems   –> section: S-API DataSources / Extractors - Virtualization Recommendation (as mentioned in Data Integration for ABAP Source Systems ) Due to limited push-down capability of SAP extractors and related performance implications, Virtualization via S-API extractors is generally not recommended – hence persistency should be considered.
[2]	Prerequisites: SAPI Extractor allows virtual access – same as [1] References: SAP Help Portal: Import Remote Tables See Data Integration for ABAP Source Systems   –> section: S-API DataSources / Extractors - Extraction / Data Replication Recommendation (as mentioned in Data Integration for ABAP Source Systems ) Certain S-API Data Source / Extractors (e.g. FI-AA Business Content) may send duplicate records during snapshot replication or real-time initialization. As the Remote Table replication runs with an optimized INSERT processing on Data Source key level this will lead to “unique constraint violation” errors. For these cases, it is recommended to use Data Flow [4] with the Remote Tables as source and the target table set to Mode "APPEND" with the flag "Update Records By Primary Key (UPSERT)” enabled.
[3]	Prerequisites: SAPI Extractor allows virtual access - same as [1] Real-time Replication prerequisites for ODP_SAPI (see link) SAPI Extractors are delta enabled (delta method ADD* not allowed) SAPI Extractor must have primary key(s). References: SAP Help Portal: Import Remote Tables SAP Help Portal: Replicate Data Changes in Real-Time See Data Integration for ABAP Source Systems   –> section: S-API DataSources / Extractors - Extraction / Data Replication See Table 3: How to check: Extractor Delta Capability See Table 4: How to check: Primary Keys
[4]	Prerequisites: See overall prerequisites for SAPI Extractor References: SAP Help Portal:  Creating a Data Flow
[5]	Prerequisites: SAPI Extractor must have primary keys. References: SAP Help Portal: Creating a Replication Flow SAP note 3297105 - Important considerations for SAP Datasphere Replication Flows Blog post https://blogs.sap.com/2023/11/16/replication-flow-blog-series-part-1-overview/ See Table 4: How to check: Primary Keys
[6]	Prerequisites: SAPI Extractor must have primary key(s). SAPI Extractors must be delta enabled. References: Same references as mention in [5] See Table 3: How to check: Extractor Delta Capability

                                        Table 6: Prerequisites and References for SAPI Extractors

Direct Integration into SAP Datasphere using SAP ABAP CDS view Extractors

In the following Table 7 (column “No.” refers to equivalent number in Figure 5) I have summarized the most relevant information/prerequisites. As functionality is changing and evolving, please check mentioned references for updates.

No.	Prerequisites & References
	Overall Prerequisites for ABAP CDS view Extractor: ABAP CDS view must include the annotation: @analytics.dataExtraction.enabled: true. ABAP CDS view must include the annotation for a data category. References: SAP Help Portal Analytics Annotations Check blog Finding the right CDS Extractor in SAP S/4HANA
(7)	Prerequisites: See overall prerequisites for ABAP CDS view Extractor References: SAP Help Portal: Import Remote Tables See Data Integration for ABAP Source Systems   –> section: ABAP CDS Views - Virtualization Recommendation (as mentioned in Data Integration for ABAP Source Systems ) Known limitation (that could lead to performance degradation in virtual access (e.g. group by statements are either not pushed down by SAP Datasphere or not applied by the extractor) – hence persistency should be considered.
(8)	Prerequisites: See overall prerequisites for ABAP CDS view Extractor References: SAP Help Portal: Import Remote Tables See Data Integration for ABAP Source Systems   –> section: ABAP CDS Views - Extraction / Data Replication
(9)	Prerequisites: Real-time Replication prerequisites for ODP_CDS (see link) ABAP CDS view must have a primary key. ABAP CDS views must be delta-enabled. References: SAP Help Portal: Import Remote Tables SAP Help Portal: Replicate Data Changes in Real-Time See Data Integration for ABAP Source Systems   –> section: ABAP CDS Views - Extraction / Data Replication See Table 3: How to check: Extractor Delta Capability See Table 4: How to check: Primary Keys
(10)	Prerequisites: See overall prerequisites for ABAP CDS view Extractor References: SAP Help Portal: Creating a Data Flow
(11)	Prerequisites: ABAP CDS view must have primary keys. References: SAP Help Portal: Creating a Replication Flow SAP note 3297105 - Important considerations for SAP Datasphere Replication Flows Blog post https://blogs.sap.com/2023/11/16/replication-flow-blog-series-part-1-overview/ See Table 4: How to check: Primary Keys
(12)	Prerequisites: ABAP CDS view must have primary keys. ABAP CDS views must be delta-enabled. References: Same references as mention in [11] See Table 3: How to check: Extractor Delta Capability

                                     Table 7: Prerequisites & References for SAP ABAP CDS view Extractors

Conclusion

As initially stated, this post is not intended to provide a comparison or firm conclusion on which option is best (or worst). Numerous factors come into play in any specific implementation or project – such as the version of the source system, availability and use of Business content, development guidelines, prerequisites of each extractor type, preference of integration functionality.

As a result, there might not be a one-size-fits-all integration option in your situation, and you may find yourself implementing several different options tailored to specific use cases.

Please let me know, if you have any detailed question regarding the integration options into SAP Datasphere.

Stay tuned for the next integration scenario post.

Regards, Claudia

[Mandar_Damle]

This blog came so late considering the lack of information available in SAP standard documentation about the consumption of S-API extractor into the Datasphere; nevertheless, it is an important and helpful blog post. Particularly, Tables 1-5 are very handy for checking S-API-based data source settings.

I have one question though -

I have one generic extractor based on VBAK and VBELN is not a key.

But I don't see a provision in the extract structure where I can mark it as a key. Because of this I cannot use delta functionality via remote tables in Datasphere.

Thanks again for this blog post!

Regards,

Mandar

[EduardS]

Hi Claudia,

a very, very well documented topic which gives a good overview of the integration possibilities.

Best Regards,

Eduard

[SilvioCanha]

Fantastic post @ClaudiaFiess !

Do you know if there is a similar post for non-SAP sources?

As per documentation, to achieve data federation with non-sap sources, the only connectors currently supported are generic OData and JDBC. Is this still the case?

Any available options (to achieve federation) for when source systems only provide traditional REST API's (non-OData) ?

Thanks!

Silvio Canha

SAP Datasphere 

[ClaudiaFiess]

@SilvioCanha  unfortunately I am not sure if there is a similar post for non-SAP sources.
I would suggest to have a look at below links - these allow you to apply filter for specific source system connections (e.g. Generic ODATA or Generic JDBC):

• Integrating Data and Managing Spaces in SAP Datasphere
  -> Connection Types Overview
• Administering SAP Datasphere
  -> Preparing Connectivity for Connections

[ClaudiaFiess]

@Mandar_Damle 
Assuming you are talking about the scenario in Table 6 No. [3]:
FYI - I just tested this by creating a generic extractor with delta extraction based on table VBAK -  I was able to leverage this extractor  in SAP Datasphere for all 3 remote table functionalities (federation/snapshot/real-time replication). 

[MKreitlein]

Hello @ClaudiaFiess,

what @Mandar_Damle writes initially have also been my first thoughts:

This Blog is really great, and I would have loved to read this already one year ago in the official SAP Help pages... but what you found there made you wonder, rather understand 😞

Why do we, as customers, see such information only in Blogs and not initially in the official help pages?

BR, Martin

[Mandar_Damle]

Hello @ClaudiaFiess 

I have an observation -

If you create a generic data source based on the table VBAK then it is possible to have VBELN as key.

But if you create a generic data source based on the FM then it is NOT possible to have VBELN as key.

Thanks!

Regards,

Mandar

[FrankRiesner]

Hi Claudia
thanks for sharing these insights. I hope you will provide the other blog posts for scenarios B, C and D soon! Cheers - Frank

[sujangade1]

Great Post when is the next blog for LIS extractors with Datasphere published?

[albertosimeoni]

The strange thing to me is that ever method has its own drawbacks more than the advantages it offer...

Dataflows => simple, (too simple,) no delta support, no preload / postload commands, works in batch mode with no recovery for the committed rows. To make the most simple fake/delta method require at least 2 dataflows per table (delete+insert). It can read from files but without dynamic/runtime instantiation of sources (if you need to upload 10 different files with the same structure => 10 different dataflows or 1 with 9 union blocks and much time loss in developing it).

Replication Flows => unfeasible/unmanageable (every changes like adding a source will restart all delta from zero), They need to provide a capability to modify only part of the replication flow (only some couple of source-target) without restarting the delta from zero for non modified objects. So the best practice is: add every possible columns of every possible table you need and you will need in future (simply not feasible).

Remote Table replication => Only real time because is pushed by dpAgent (with non trivial sizing requirements), a disconnection may broke it and really rare use case for real time replication.

The only solution is to have replication flows with more capabilities to be managed after the first deploy ( to modify a single source-target couple ).

Best Regards,

Alberto