SAP Datasphere + SAP S/4HANA: Your Guide to Seamless Data Integration

[jeetendrakapase]

Introduction:

Achieving seamless data integration between SAP S/4HANA, your core operational ERP system, and SAP Datasphere, the business data platform is a critical requirement for most SAP customers. This integration unlocks powerful reporting capabilities across various business functions.
This technical article aims to empower you with expert tips and strategies to simplify and enhance your SAP S/4HANA integration for operational reporting, analytical insights, and planning & forecasting purposes. You are following the best practices and knowing what is supported and when to use what. you can streamline data integration unlock the full potential of this integration and bring more time to value.

SAP S/4HANA Data Integration Explained:

This diagram illustrates various methods for integrating data with SAP S/4HANA, covering both data federation and replication scenarios.

Source Object Types:

1. ABAP CDS Views: It provides business data along with semantic(context) information (hierarchies, text, etc.), security, associations, and improved performance.
2. ODP Extractors: Specialized tools designed to extract specific datasets from SAP S/4HANA.
3. ABAP Tables: The traditional SAP master and fact tables are used for raw business data storage.

Delta Handling Mechanisms During Data Replication:

1. Database Trigger-based CDC Delta: This method leverages database triggers to capture changes in real-time. It utilizes the well-established SAP Landscape Transformation Replication Server (SLT) technology.ABAP Pipeline Engine (APE): This framework provides flexibility for building custom CDC pipelines suited to specific needs.
2. Generic Timestamp/Date-based Delta: This approach relies on a designated timestamp/date column within your ABAP CDS views to identify data modifications.

ODP Technical Infrastructure:

• Operational Data Provisioning (ODP) Framework: This is the modern framework for CDC, replacing the older RSA7 (BW Delta Queues) approach. It simplifies data extraction and management.
• Operational Delta Queue (ODQ): This queue stores changes made to data until they are replicated.

Data Builder Objects in SAP Datasphere facilitate your integration with SAP S/4HANA:

1. Remote Tables: Enable access to SAP S/4HANA data in real-time or through snapshots.
2. Replication Flows: These define the path for data replication from source to target system.
3. Data Flows: These specify the transformations applied to the data during replication.

Model Import Wizard:

This tool imports valuable metadata, context, and semantics from your CDS views, including associations, logic, and security information. Convert the SAP S/4HANA CDS semantics to SAP Datasphere data and business builder entities using the wizard, take a quick look at the below image for the I_PRODUCT CDS view same can also be done for the Sales Order, and Purchase Order CDS views. It is automatic, and you can imagine how much time/effort is required to build something similar to a standalone. This is something you will only get with Datasphere. 

Now you may have a question what is the benefit of getting a data model available in SAP S/4HANA into SAP Datasphere? Let's look at the use case, I strongly believe one will apply to your organization's data platform needs.

1. Integrate with Non-SAP Data, It is obvious that you want to keep your SAP ERP clean for business transactions while needing a data platform to combine the SAP ERP business data with other business data segments with the context.
2. Strategic or Historical Reporting, over time you may plan to achieve the data from SAP S/4HANA hence for the historical data along with the business semantics for the reporting needs.
3. Leverage the Hyperscalar capabilities like ML/AI, with SAP Business data with semantic information.

Source Object Types Support and Compatibility:

The top diagram conveniently highlights which source object types are compatible with specific data integration objects within the SAP Datasphere.

Prerequisites and Additional Software Components (Middleware):

The diagram also indicates any prerequisites or middleware components needed for specific functionalities. For instance, Remote Tables require installing, configuring, and setting up the SAP Data Provisioning (DP) Agent with the ABAPAdapter. Refer to the help guide on Preparing Connectivity for Connections and filter based on the Connection Type = SAP S/4HANA.

Optional Components:

• SAP BW bridge: This component (optional) creates a data staging layer specifically for supporting SAP S/4HANA ODP extractors. This is mainly for the existing SAP BW and BW/4HANA customers to enable a path towards conversion with SAP Datasphere, BW bridge, or Hybrid use case
  • Transferring Data using ODP Options
    
• SAP Landscape Transformation Replication Server (SLT): This server replicates tables from SAP S/4HANA to the SAP Datasphere and can be integrated into your Replication Flow.

Further Resources:

For a deeper understanding of ABAP CDS delta mechanics and the ODP framework, refer below:

• CDS-based data extraction with delta handling mechanism.  
• Understanding ODP, The Functionality of ODP, and Using ODP Effectively.

Recommendations for choosing the right approach for SAP S/4HANA Data Federation and Replication into SAP Datasphere:

1. Recommended option is ABAP CDS Views: If the ABAP CDS is available with the desired data needs then establish a direct connection between SAP Datasphere and the source system. Leveraging the available CDS views is the BEST and RECOMMENDED Approach. Recommend to use released CDS views as they are upgrade stable.
   • Individual C1-released CDS views across various business areas.
     • 
   • SAP Business Accelerator Hub List of Released CDS Views. Here you can filter based on the capabilities of Data Source for Data Extraction to find extraction-enabled released ABAP CDS views.
     
     • 
   • Finding the right CDS Extractor in the SAP S/4HANA System. In SAP S/4HANA On-Prem or Private cloud >=2020, You can use Tcode SE16N with CDS SQL View: IXTRCTNENBLDVW (I_DataExtractionEnabledView) or View Browser Fiori App.
     • How to find the released Sales Document Item CDS view using SE16N GUI Tcode?
     •  How to find the released Sales Document Item CDS view using the View Browser Fiori App? You will need the business role: SAP_BR_ANALYTICS_SPECIALIST. Tcode /UI2/FLP -> View Browser -> Released Views for Cloud Development
       •  
       • 
   • Recommended using standard CDS views with extension capabilities, and pre-built business contents over creating custom CDS views as long-term custom CDS may not be a sustainable option with the upgrades and may lead to performance implications. However, all requirements cannot be met with the standards hence only create custom CDS when it becomes business required and no standard alternative is available.

   • Unavailable CDS Views: The required CDS views haven't been released yet. Missing Data Extraction Settings: The data extraction settings are not configured even if the CDS views are available. There are two approaches to resolving these issues:

     1. Customer Influence Request: Raise a customer influence request or upvote the existing one at SAP S/4HANA Cloud Public Edition, CDS View Extraction Enablement. OR Open a Product Support Ticket For High Priority Request: A ticket can be submitted to the relevant component for a long-term solution. Be aware that this process may take some time.

     2. Build a Custom CDS View: Create a custom CDS view replicating the missing one. You can then enable data extraction on this custom view for your use. How to Create a Custom ABAP CDS View with Change Data Capture Delta Mechanism.
     3. How to deal with “Not released” predefined VDM in S/4HANA.
   • DATA FEDERATION FIRST THEN DATA REPLICATION, The approach for using the CDS view as a data federation goes back to the basics 🙂
     1. Select the required columns from the projection. e.g. CDS view may have 200 columns however business may require 75 columns. hence select only 75 columns. 
     2. Filter the data as early as possible based on attributes like company code, sales org, plant, dates, or periods.
     3. Business logic and data aggregation can be handled in the source CDS view, you do not have to do it on SAP Datasphere, unless required. 
     4. Use the view persistency option intelligently to get optimized data and high performance.
     5. Middleware components like DPAgent need to be sized properly S-M-L, go with a higher side like 16GB, 32GB memory vs 4GB default and 8 cores CPU.
   • Restrictions: 2890171 - SAP Data Intelligence / SAP Datasphere - ABAP Integration
     1. Not supported for data transfer if the CDS view not having the "dataExtraction" annotation.
     2. CDS view for the delta load requires the "dataExtraction" and "changeDataCapture" annotations.
     3. CDS Views having parameters and a table function are not supported.
     4. CDS Views include Tables with a client field as a key, not a client-dependent field.
     5. Data extraction halts with an error if the structure of a table or CDS view changes mid-load or replication. Unfortunately, resuming after such an error isn't currently possible, requiring a full restart of the data extraction process.
2. An alternative option is ODP extractors: If the ABAP CDS view is unavailable then prefer ODP objects over the direct ABAP table replication. A direct connection can be established provided SAP S/4HANA has the DMIS add-on.
3. The discouraged option is Direct ABAP Table Access and Replication: Bypassing CDS views to replicate tables into SAP Datasphere directly is not recommended. This approach involves replicating raw data (master and transactional) without its inherent business context and relationships. Essentially, you'd be replicating the data and then manually recreating the associations, logic, and hierarchies – a time-consuming and error-prone process. The preferred approach is to leverage the existing, comprehensive CDS data models whenever possible. However, if replicating tables is unavoidable due to specific business or compliance requirements, then the recommendations below should be followed.
   1. Standalone SAP LT Replication Server: For replicating tables, the recommended approach.
   2. Embedded(Add-on) SAP LT Replication in SAP S/4HANA: This option is discouraged because it's tightly coupled with the SAP S/4HANA version. Upgrading the server requires a complete SAP S/4HANA upgrade, limiting flexibility. Downporting new functionalities to older versions is also not feasible.
   3. Restrictions:
      1. Replicating tables with redirect views (e.g., MBEW, MARD, COEP) from your SAP S/4HANA source system is not supported.
      2. Declustering of INDX-like tables (e.g. STXL, PCL1, ...) during the transfer is not supported.

SAP S/4HANA Connection Supported SAP Datasphere Integration Features:

	Supported Objects Types	Delta Load Supported?	Prerequisite Connectivity Components?
Remote Tables	ABAP Tables Extractors: ABAP_CDS ODP_BW ODP_SAPI Model Import	YES - Real-time data replication. The object in the source has to be enabled for change data capture (CDC). For source views, ABAP Dictionary tables real-time replication is not supported. For ABAP ODP sources, real-time replication is supported in the following cases: ODP-BW: InfoProviders with a change log.object type ADSO), Standard DataStore objects (classic) (object type ODSO), InfoObjects ODP-CDS: For all ABAP CDS views with primary key and that are delta-enabled ODP-SAPI: For all extractors that are delta enabled (delta method ADD* not allowed) and have a primary key. For an overview of all S-API extractors released for ODP replication. refer:   2232584 - Release of SAP extractors for ODP replication (ODP SAPI)  For remote tables using an SAP HANA smart data access connection (SAP HANA on-premise via Cloud Connector or SAP HANA Cloud), you can replicate your data changes in real-time but with some restrictions as some remote object types are not supported: The remote object needs to be a table of type "COLUMN TABLE" (other table types like Row Tables or Virtual Tables are not supported) Some data types and SAP HANA table features are not supported. For more information, see Unsupported Data Types and Other Limitations. Note You can’t pause the replication for connections if SAP HANA smart data access is used. For remote tables connected via an SAP HANA smart data integration Cloud Data Integrator (CDI) adapter, we recommend that you use a DP Agent with the latest version.	DP(Data Provisioning) Agent
Replication Flows	CDS_EXTRACTION (ABAP CDS Views Enabled for Data Extraction) ODP_BW ODP_SAPI SLT (add-on DMIS )	YES - Load type: Initial and Delta The default delta load interval is 60 minutes. You can change the integer between 0 and 24 for hours and 0 and 59 for minutes, respectively. If you enter 0 hours and 0 minutes, the system replicates any source changes immediately. Performance Issues: 3360905 - SAP Data Intelligence / SAP Datasphere - ABAP Integration - Performance when using RMS / R...	SAP S/4HANA On-Premise: Cloud Connector SAP S/4HANA Cloud: None
Data Flows	CDS_EXTRACTION SLT (add-on DMIS )	NO SAP Note: 3335277 - Datasphere - No Delta or Init in Data Flow	SAP S/4HANA On-Premise: Cloud Connector SAP S/4HANA Cloud: None
Import Entities	ABAP CDS Views Entities. e.g.: I_PRODUCT, I_BillingDocumentItem etc.	N/A Even though this object already exists from a prior import, any changes to its structure or meaning within the source system will be automatically detected, triggering an update and redeployment within your system. By default objects created will be as data is federated (remote tables), you can switch to replication when required.	SAP S/4HANA On-Premise: DP(Data Provisioning) Agent, Cloud Connector SAP S/4HANA Cloud: None

Further Resources:

For a deeper understanding of the SAP S/4HANA and ECC integration scenarios with persistence options: Exploring Integration Options in SAP Datasphere with a focus on using SAP extractors

Pre-built Business Content for SAP S/4HANA:

Overall, SAP Datasphere Business Content for SAP S/4HANA helps you unlock the full potential of your data by providing a faster, easier, and more reliable way to access, integrate, and analyze your business information. e.g. Finance Foundation for SAP S/4HANA and SAP S/4HANA Cloud, Sales Analysis for SAP S/4HANA and SAP S/4HANA Cloud, etc.

Faster Time to Value: Pre-built dashboards, data models, tables, and views with pre-defined business context eliminate the need for extensive development and configuration from scratch. 

Simplified Data Integration: Business Content streamlines the process of integrating data from SAP S/4HANA with other applications and data sources. This allows you to create a unified view of your data for more comprehensive analysis. With minimum effort, you will only have to replace the SAP S/4HANA server details in the connection placeholder and you are ready to consume the continent or if you want to replicate the data then just create the required replication flow and you are done. all details are provided as a part of the help guide.

Preserved Business Context: Business Content captures the business meaning of the data, including units, hierarchies, and other relevant information. This ensures that users understand the data in its proper context and can make informed decisions.

Alignment with Industry Standards: Business Content is often aligned with industry best practices and key performance indicators (KPIs), allowing you to benchmark your performance against industry standards

 

Key Takeaways:

• This breakdown highlights the various options in SAP Datasphere for data federation, extracting, transforming, and replicating data from SAP S/4HANA.The most suitable method(feature) depends on factors like real-time data federation or real-time updates vs. snapshots, data type, data volume, and the desired level of data transformation or conversions.
• The data source object recommended precedence is ABAP CDS View > ODP Extractors > ABAP Tables. ABAP CDS view object type from SAP S/4HANA is the first-class citizen as a data source or provider from the ERP system. Customers can also prefer to create a custom CDS view with CDC capabilities even for the master data where date or timestamp columns are unavailable.
• SAP S/4HANA real-time operational reporting can be done based on the Remote tables, and master data (smaller data sets) with Snapshot. For Mass data replication from SAP S/4HANA into SAP Dataspher Prefer, use the New Replication Flow.
•  The import model wizard provides the capability that scans the semantically rich CDS data model with associations and automatically creates the data builder and business builder objects with the default data federation scenario.
• SAP S/4HANA Business Content provides time to value and quick win use case that can be enhanced.
• Optional components like the BW bridge and SLT server can further enhance data integration capabilities and Safeguard existing on-prem investments and skillsets.