Recap — SAP Data Unleashed 2024

[TuncayKaraca]

Yesterday I attended the SAP Data Unleashed 2024 event. It's SAP's marketing event. It's not a webinar, nor a webcast, nor a technical event! I would like to summarize some notes here.

Photo by Conny Schneider on Unsplash

It's a one-hour online event. We all watched a video as participants and there was a Q&A chat box. It's a passive event with the video and it's an interactive event with the Q&A. While watching the video, I tried to follow and watch the questions and answers. The questions and answers flowed very quickly. Honestly, there were a lot of participants - maybe thousands of them - the questions were answered by several SAP team members & experts. I was trying to watch the video and not miss maybe some important announcements. I thought maybe the video was not released to the public. Because all video materials etc. seem to be very confidential; there was a link on the online event platform for copyright information.

Anyway, I was wrong! After the event SAP first released the news SAP Data Unleashed: The Future Belongs to Data-Driven Organizations with highlights. Then another news Unleashing the Latest SAP Data and Analytics Innovations with more information. It's all news and marketing, but let me highlight what the event was all about and the important points.

SAP Data Unleashed 2024 — SAP HANA Cloud, SAP Datasphere, SAP Analytics Cloud

The stars of the Data Unleashed event are  SAP HANA Cloud,  SAP Datasphere and SAP Analytics Cloud. Among these three products, all of which are of course SAP Business Technology Platform, the announcements are listed below.

SAP Data Unleashed 2024 — SAP Datasphere Knowledge Graph

• SAP Datasphere Knowledge Graph — automatically creates an ontology representing the relationships in the data, including the inherent business context from SAP application sources like SAP S/4HANA. … Knowledge graphs can provide better context to large language models (LLMs) and prevent AI-generated “hallucinations.” SAP Datasphere knowledge graph will enable organizations to unleash the power of AI for even the most complex queries.
• SAP HANA Cloud Vector Engine — provides the ability to store and compare vectors using SQL. Determining similarity between vectors enables use cases such as retrieval-augmented generation (RAG), recommendations, classifications, clustering, and more. … customers to build intelligent data applications that combine the natural intuition of human interaction with powerful machine learning and multi-model processing capabilities.
• Simplified Consumption of Data Products in SAP Datasphere — customers will find ready access to SAP and non-SAP data products and easily install them into their SAP Datasphere instance. The procedure establishes relevant connections and creates the necessary artifacts in SAP Datasphere without the user having to enter those details.
• SAP Datasphere Semantic Onboarding Enhancements — provides a seamless import of semantically rich objects from SAP S/4HANA and SAP BW/4HANA into SAP Datasphere. This preserves the semantic information such as measures, attributes, hierarchies, aggregations, and currencies/units, while keeping SAP HANA Cloud and SAP Datasphere objects in sync…
• Collibra AI Governance Integration with SAP Solutions — The partnership with Collibra is being expanded to address the data and AI governance needs of joint customers via Collibra AI Governance that was launched last week and is available to interested customers.
• SAP Datasphere Confluent Integration — enables real-time streaming of data to and from SAP Datasphere… It allows connections on the application level without additional custom application development…
• SAP Analytics Cloud for Planning Deep Integration to SAP Datasphere — SAP Analytics Cloud planning models can be deployed into SAP Datasphere, bringing plan and actuals data together for real-time steering… With this, SAP Datasphere will become the platform on which planning processes and tools can integrate.

SAP Data Unleashed 2024 — SAP Analytics Cloud Joule Co-Pilot

• SAP Analytics Cloud Generative AI Co-Pilot — … generative AI for … data discovery, dashboard creation, planning model maintenance, and more… run analytical queries, simulate driver impacts, plan resource allocations, and auto-generate dashboards and calculations… A preview of Joule co-pilot functionality in SAP Analytics Cloud is planned to be available in Q2 2024.
• SAP Analytics Cloud Compass — provides Monte Carlo simulation capabilities with a business user-friendly interface…A preview of SAP Analytics Cloud compass is planned to be available at the end of Q3 2024.

Today SAP also uploaded the video Explore the Future of Data, AI, and Planning | SAP Data Unleashed 2024

As you see in the video there are two customers Hugo Boss and Bosch, two technology partners Collibra and Confluent, two implementations partner Deloitte and Accenture.

Although myself I asked a question regarding SAP Datasphere's capabilities like Data Intelligence I could not see the answer while all Q&A box very crowded and trying to read all answers. Namely some attendees asked “Unable to focus on the Ask Questions and also listen the session. Is it possible to check the QA later ?” and the answer was “The Q&A will remain open for 30-minutes after the broadcast concludes, so you have a chance to check the Q&A during that time.” Probably you could not get a chance read all questions and answers, so I would like to add some questions and answers here:

• Question: Can we know the Datasphere Roadmap in the future and upcoming innovation aligned to this product?
• Answer: The roadmap is available online and aligned to the other products.

• Question: Does the Datasphere is going to have any Excel add-in are it can be connected through AO?
• Answer: Yes, Datasphere artifacts can be consumed in Excel through the SAC add-in for Excel. More information here.

• Question: What option I have for Near Real Time Storage when pooling all my system data into Datasphere for reporting?
• Answer: In SAP Datasphere a dedicated service called Replication Flows enables near real time replication into SAP Datasphere. Transformation Flow then allow further data processing/transformation capabilities.

• Question: What is the pricing model of SAP Datasphere?
• Answer: SAP Datasphere pricing is based on capacity units. Usage of various services can used per month, each with a specific capacity unit value.

• Question: Is there a path or tool provided to move ABAP programs from SAP BW 7.5 systems to SAP Datasphere?
• Answer: SAP Datasphere and the BW Bridge currently don’t support custom ABAP programs. For transformation logic and variable exits can be migrated into SQL logic. For other applications you always have the BTP (e.g. SAP Build) available to build apps.

• Question: Is there anyway to generate delta loads from Datasphere Tables and use these as a source in a Replication Flow?
• Answer: This planned for Replication Flows when using local tables with delta capture as a source. Please take a look on our roadmap explorer to have our latest updates available.

• Question: Who will be working with these features? IT or Business? Do we need specific skills to manage them?
• Answer: SAP Datasphere provides features for data professionals, but also for business users. So, it really depends on which task a user tries to fulfill. Overall, SAP Datasphere aims at making all activities easier to complete.

• Question: We receive an excessive number of requests from clients for access to SAP data via data dumps. We provide a large and diverse portfolio of analytic solutions but clients want to build their own reports using their own (non-SAP tools). What is SAPs recommendation to handle these requests?
• Answer: Datasphere can expose the artifacts for consumption either via a SQL endpoint or an OData endpoint. One of the two ways can be leveraged to consume exposed artifacts. Please refer to the documentation for more information.

• Question: Can Tableau be connected to the Datasphere?
• Answer: As Datasphere exposes its artifacts for consumption via a SQL endpoint or an OData API, one of the two ways can be leveraged to consume data. Please refer to the help documentation for more about this.

• Question: What is the difference between Datasphere and Data Intelligence?
• Answer: SAP Datasphere is the evolution of SAP Data Warehouse Cloud and is equipped with selected SAP Data Intelligence capabilities (e.g. Replication Flows, Data Catalog) and will be enhanced in the future with additional data integration capabilities.

• Question: How data literate does a user need to be to leverage the capabilities of SAP Datasphere?
• Answer: Data literacy plays a pivotal role in the SAP Datasphere catalog’s functionality. When exploring a data asset within the catalog, users can access comprehensive metadata, including details on columns, data lineage to trace its origin, a business glossary providing insights at the object or column level, and associated key performance indicators (KPIs) for a holistic understanding. Additionally, tags and comments are available to assist end-users in gaining insights into the data assets.

• Question: It is now possible to increase Datasphere resources by self service. Last year there was an item on the roadmap which should enable us to see the financial impact of the increased resource allocation (somehow called Financial governance). This item is not on the roadmap anymore. Anything planned in future for this requirement?
• Answer: Yes, we will continue increasing the scope of what administrators can configure in SAP Datasphere Flexible Tenant Configuration as this is a key priority for our customers.

• Question: In BW/4HANA, we can extract data from S/4HANA and stage it in ADSO. For example : we can replicate ACDOCA table we can store billion of records and if it reaches 2 billion limit , we can do partitioning of ADSO. How it is going to be handled in SAP Datasphere ? Will Datasphere is capable of storing billion of records in cloud and can we do partitioning in SAP Datasphere?
• Answer: SAP Datasphere is an SQL data warehouse, it uses tables instead of DSO. The tables in in Datasphere can be partitioned to accommodate the ACDOCA. In addition the data could also be spread across multiple tables.

• Question: Can we try Joule?
• Answer: Joule isn’t available in SAP Analytics Cloud yet, following the successful release of the just ask NLQ feature in Q1 2024, SAP is currently planning initial GenAI deliveries in H2 2024, please see the product roadmap for more information.

• Question: Will SAP still invest and provide feature and functionality upgrades to BW4/HANA 2.0 OR is it put more on support track only?
• Answer: We are investing in SAP BW/4HANA and deliver new features like we did end of last year with the release SAP BW/4HANA 2023. So you are 2 releases behind the latest version.

• Question: Any resources that we can use to understand knowledge graph better, in terms of how it works?
• Answer: The feature is currently in preview and we are excited to share more in the coming months.

• Question: SAP Datasphere involves as much configuration as when I started doing BW more than 2 decades ago — time consuming. How will you utilize Gen AI to handle the repetitive tasks involved in configuring Datasphere?
• Answer: We will infuse generative AI into workflows and features in SAP Datasphere to improve the efficiency of user tasks. This will extend across the feature and administrative areas within SAP Datasphere.

• Question: How the Collibra is leveraging the SAP info types as Business catalog for Data governance?
• Answer: Through the integration with SAP Datasphere catalog, Collibra can display the metadata and lineage of SAP data and assets like SAP Analytics Cloud stories and models, and SAP Datasphere tables, models, and more.

• Question: In SAP HANA, we can use SDI or SDA or may be Integration Suite to connect to outside systems, and combine data from multiple data sources, and perform required modeling. What is the advantage of Datasphere in this scenario, if we already have SAP HANA?
• Answer: SAP Datasphere is equipped with additional capabilities for Mass Data Replication and additional connectivity (like Kafka and Confluent) which complements what SAP HANA SDI and SDA. It is also possible to leverage additional data transformation capabilities in SAP Datasphere Transformation Flows with dedicated user experience.

• Question: How is data secured in Datasphere?
• Answer: Data access controls in Datasphere let you secure your data on a row-level which can be applied on data layer models so that the user can only see the rows which they are authorized to see.

• Question: How is the integration of knowledge graph to SAP Analytics Cloud look like? What is possible with this integration?
• Answer: The knowledge graph capability is focused on Datasphere modeling, and SAP Analytics Cloud will benefit from easier to create data models.

• Question: Is there a best practice method for the life cycle management in Datasphere? DEV — PROD
• Answer: The recommendation for setting up multiple environments is through multiple tenants that are Productive as well as Non-productive.

• Question: How will Datasphere work with the core insurance systems like for example claims system or underwriting system?
• Answer: If these system provide an interface that SAP Datasphere can read from. ABAP Bases system typically provide ODP as an interface to read, other options could be JDBC or OData.

• Question: How Different is Datasphere from the Regular ETL tools like DS, CPI etc.?
• Answer: SAP Datasphere is SAP’s Business Data Fabric that offers Data Integration, Data Management and Data Discovery capabilities in a self-service manner.

• Question: What are the space limitation in Datasphere?
• Answer: The space thresholds can be set by a space or tenant administrator in terms of Disk and Memory consumption as well as workload level limits.

• Question: How is this Datasphere different than Microsoft Fabric?
• Answer: SAP Datasphere provides a business data fabric that connects easiest to SAP data while also incorporating 3rd party data (e.g., via the data marketplace). Looking at SAP sources, SAP Datasphere offers semantic onboarding to easily integrate SAP data into your SAP Datasphere while retaining semantics (such as associations between tables).

• Question: How can the Business User access the Data/Reports in Datasphere? Is SAP Analytics Cloud still necessary as frontend?
• Answer: Access to reports/data in Datasphere depends on the specific implementation of the system. Datasphere allows users to access and manage reports directly through its own interface. However, it may also integrate with other tools like SAP Analytics Cloud (SAC), or other visualization software which can act as the front-end interface for more detailed data analysis and visualization.

• Question: How much data is needed in order to properly utilize the SAP Datasphere?
• Answer: SAP Datasphere is a cloud solution, therefore customers are able to scale their SAP Datasphere tenant according to their need — starting with smaller instances up to bigger implementations.

• Question: Are Datasphere knowledge graphs creates automatically? How can one access the knowledge graphs? Where can we find more information this?
• Answer: Correct, the effort and cost of manually creating knowledge graphs is typically very high and SAP Datasphere knowledge graph overcomes this by pre-populating the knowledge graphs using all the valuable content already available in SAP Datasphere. It is currently in preview, and we are excited to share more in the coming months.

• Question: In SAP Analytics Cloud, predictive analytics can do machine learning. How large the size of the dataset should be, how much time does it take to train the model and how cost is calculated?
• Answer: For time series forecasting — a very popular ML use case — the size of the historical data depends directly on the number of predictions you want to do in the future. A good ratio admitted is 5:1. This means that if you have 100 historical cases, you predict 20 values of your target variable in the future.

• Question: Can we use SAP HANA smart data integration Data Provisioning Agent to connect to our on premise BW7.5 SP14 and bring in data towards Datasphere?
• Answer: Yes, SAP Datasphere uses for Remote Tables SDI and SDA. SDI connection use the dpAgent. In addition you can use the Cloud Connector for SDA, Replication Flows and Data Flows to connect to BW7.5

• Question: What are some of the successful use cases between SAP Analytics Cloud, Datasphere, and Google Cloud?
• Answer: SAP Discovery Center > SAP Datasphere

• Question: Is it possible to extract data from S/4HANA tables to Datasphere without using CDS Views?
• Answer: You can perform CDS View based replication, which is our recommended approach. Table-based replication from S/4HANA is also possible leveraging Replication Flows in SAP Datasphere.

• Question: How BW/4HANA and Datasphere are different?
• Answer: There are lots of differences and lots of commonalities. The major difference is that SAP Datasphere is public cloud software as a service offering, whereas SAP BW/4HANA is a private cloud and on-premise offering.

• Question: Will you provide for Datasphere standard data lake storage without compute and not only HANA based?
• Answer: Yes, we are working to offer data lake storage in Datasphere, so that data does not need to be store in HANA.

• Question: Is Datasphere now available in NS2 environment?
• Answer: SAP Datasphere is not available on NS2 right now. Please follow the Roadmap Explorer to understand Data Center rollouts.

• Question: Is it possible to use SAP Datasphere Knowledge Graph from an On-Prem HANA DB, hosted in a private network?
• Answer: SAP Datasphere knowledge graph pre-populates ontologies based on SAP S/4HANA and other Line of Business applications. It is currently in preview, and we are excited to share more specifics in the coming months.

• Question: What are the limitation on data federation (in terms of huge data volumes)?
• Answer: The limitation is more dependent on the source system as well as the model that is created in Datasphere and the resources available on the Datasphere tenant. There are no set limitations for data federation.

• Question: How is Collibra data governance used? Do we partner with Collibra directly, or will this be available within SAP products or via BTP?
• Answer: It is achieved through the Collibra Data Intelligence Platform. Through the integration with SAP Datasphere catalog, Collibra can display the metadata and lineage of SAP data and assets like SAP Analytics Cloud stories and models, and SAP Datasphere tables, models, and more.

• Question: How would Datasphere add value if we already have S4 HANA on prem?
• Answer: SAP Datasphere is a Business Data Fabric, built to bring data from different source system together and democratize it. S/4 HANA is a ERP solution and can be one possible data source. In most cases, you want to ad data from other sources to your Analytics as well. This is where SAP Datasphere comes into play, besides a variety of other use cases.

• Question: Existing BW system when migrated to Datasphere, BW Bridge, BW queries cannot be reused. Is there any plans in Datasphere roadmap to migrate BW queries to analytic models in Datasphere for reuse.
• Answer: SAP Datasphere offers the Model Transfer to migrate BW Queries into Analytic Models.

• Question: What are the requirements for the S/4HANA system connection (on-premise) with Datasphere? I mean, for example, which adapter?
• Answer: Please check the following documentation that provides some detailed information about the pre-requisites: Prepare Connectivity to SAP S/4HANA On-Premise

• Question: Are you also enabling the integration to SAP IBP?
• Answer: We are planning to support a live connection with IBP in Q2 (IBP2405).

• Question: How it is expected to remove AI bias in the Datasphere context?
• Answer: Collibra AI Governance provides detailed information on the data fueling an AI model, and alerts AI owners to the possible risks associated with the data.

• Question: Are there already AI functions in Datasphere?
• Answer: Our partnerships with DataRobot and Databricks enable customers to build machine learning and artificial intelligence models.

• Question: Which features or functionalities of BW4/HANA 2.0 not available in Datasphere?
• Answer: Currently two structures, NCUM (planned to be delivered in Q2), variables (still missing some features like replacement path), DSO activation logic (planned). This is not a complete list. In addition please check the Note https://me.sap.com/notes/2932647/E

• Question: Can we save result of the flow — either transformation or data flow as a file on SFTP server?
• Answer: SFTP as target system is currently not supported. Please feel free to submit your feature request with your concrete use case here: https://influence.sap.com/sap/ino/#campaign/3433

• Question: Can we visualize Report in Power BI from SAP Datasphere?
• Answer: Please refer to the documentation for integration between SAP Datasphere and Power BI: Consume Data in Power BI and Other Clients, Tools, and Apps via an OData Service

• Question: What are the plans regarding leveraging SAP Datasphere as a backend for SAP Analytics Cloud planning? The current solution necessitates acquiring data in SAC planning from SAP Datasphere for planning capabilities, resulting in data duplicity and a poorly designed system for any complex calculations.
• Answer: Please check the following roadmap item for more information.

• Question: What are the possible ways to consume SAP Datasphere data?
• Answer: SAP Analytics Cloud is the perfect complement for consumption of SAP Datasphere data. In addition, there are SQL and OData APIs to connect other tooling for consumption.

• Question: Does the BW Bridge in the Datasphere have the same functionalities as a SAP BW on HANA? If not, what are the biggest differences?
• Answer: The biggest difference is that the BW Bridge does not have an OLAP, the OLAP tasks are done inside of Datasphere. I addition the BW Bridge does not allow add-ons or custom applications.

• Question: By when can we read from KAFKA? Does Datasphere also support delta parquet files as source? By when will JSON-Document store be available in Datasphere? Will JSON-Document usage store cost in addition? Is SAP HANA Cloud Vector Engine also available in Datasphere By when are the new AI functions in SAC included? Does they need additional config or reconfigurations? By when will BW bridge be enhanced to extend the length of info provider names to get rid of the lousy 10 digits?
• Answer: Consuming data from Confluent in SAP Datasphere will be possible mid of 2024. No time line available for the support of JSON-Document store. We introduce Knowledge graphs in SAP Datasphere. Please check out the official roadmap of SAP Analytics Cloud features and find corresponding SAP Help Documentation about the topics after they are released.

• Question: Are there any plans (or existing tools) that support the Datasphere developers — like versioning, formula overviews and such?
• Answer: We do have plans to support Technical Versioning in the near future. Please refer to the roadmap explorer as the planned capabilities will be visible here.

• Question: Is there a plan to support REST calls within flows in Datasphere — like we could do in Data Intelligence?
• Answer: Rest API can currently be done via Open Connectors. We are already working toward an Datasphere Rest API connection.

• Question: So Datasphere is the future BW and SAC is the new BO?
• Answer: SAP Datasphere and SAP Analytics Cloud do far more than SAP BW and SAP BusinessObjects BI respectively, thousands of users of SAP BW or SAP BusinessObjects have already adopted SAP Datasphere and SAP Analytics Cloud for modern data and analytics.

• Question: In regards Datasphere connectivity to Data Lakes like Snowflake when we can expect Cloud to Cloud connectivity? Similar like in SAP DI — ODBC connection? Really current option to utilize JDBC via Data Provisioning Agent (which today does not support even High Availability) is not optimal and not usable for enterprise solutions.
• Answer: Data Integration capabilities in SAP Datasphere will be enhanced over time to overcome gaps and limitations.

• Question: Can the Datasphere in future have planning capabilities?
• Answer: Seamless Planning will support sharing models between SAP Datasphere and SAP Analytics Cloud for a seamless planning experience.

• Question: Is Non-cumulative key figure already available in SAP Datasphere?
• Answer: Support for non-cumulative key figures in Analytic Model is planned to be released in Q2.

• Question: What is basic release/SP-level for SAP ECC to connect to Datasphere?
• Answer: 4.6C with Replication Flows and SLT.

• Question: Will Datasphere allow for live connections to Google Sheets?
• Answer: Currently, no timeline available.

• Question: Does Datasphere have OLAP Engine like BW BEX for aggregation?
• Answer: Analytic Models in Datasphere provide the analytical engine capabilities.

• Question: Does Datasphere have the capabilities of data transformation like ETL tools have?
• Answer: Yes. There are three types of transformations Transformation Flows (a little like BW Transformations), Data Flows more ETL like and SQL Views with persistency.

• Question: Do we need to move data from SAP or non SAP system to Datasphere for modelling OR data can reside in source system can be fetched in real-time?
• Answer: Federated scenarios are supported in Datasphere where in remote tables can be leveraged for modeling which lets you model and consume without having to necessarily move the data.

• Question: What exactly does the Knowledge Graph? Why is it different from current Datasphere offering? In simple terms.. can you provide an use case example?
• Answer: With a knowledge graph you can uncover relationships within your data and also establish a schema-free approach to data allowing simple queries that would otherwise require complex joins of tables. Moreover, it allows grounding of GenAI models. I would advise you to check out some documentation (SAP and non-SAP) to do research on this technology.

• Question: What is recommended, (replication flow vs remote table with replication) to replicate data from S/4HANA to Datasphere?
• Answer: SAP Datasphere Replication Flows is the recommendation.

• Question: Will BW Bridge be decommissioned at some point? Should companies move their BW Bridge models to Datasphere native artifacts at some point?
• Answer: You can move your models to native models. There is no plan at the moment to decommission the SAP BW bridge as an optional part of SAP Datasphere.

• Question: Is there a plan to provide cloud transport integration with Datasphere transports? like the way charm works?
• Answer: We do have plans to support Cloud transport service with Datasphere in 2024 H2.

• Question: In Datasphere / SAC is there a way to see the predictive / AI model that is being used and or to change it?
• Answer: SAP Analytics Cloud has full model debriefing in Smart Predict with the ability to deconstruct individual forecasts and examine accuracy. Just Ask/ Joule also has model management for Admins to manage rules, model training and synonyms.

• Question: RAW data type supported in SAP Datasphere? if not when it will be added?
• Answer: Datasphere supports Binary, LargeBinary and hana.Binary. They can not be displayed in Datasphere one processed. Column Data Types

• Question: How can apply means to encrypt or to make fields anonymous? Is there any console or other function available?
• Answer: Data Anonymization is supported using HANA Cloud functions through the Open SQL schema.

• Question: What is the different between planning in Datasphere vs planning using SAC models?
• Answer: Planning in Datasphere will bring the planning process closer to the data itself improving overall process agility and responsiveness through access to real-time data through the use of data federation.

• Question: Datasphere can be used as data migration tool? Since SAP Data services is getting decommissioned. Will bring some features from Data Services?
• Answer: At the moment we are not positioning Datasphere as data migration tool for SAP S/4HANA conversions.

References

1. Recap — SAP Data Unleashed 2024 at medium.com/@tncykarc

[yoav_]

Excellent coverage @TuncayKaraca!

[tmeyer]

👍

[jakobflaman]

Thx Tuncay. Great Blog