Kisha Gulley was once kicked out of a Facebook group for mothers with autistic children after a c...Read More
This blog post is about a strategic feature of SAP Data Warehouse Cloud, namely the Data Marketplace. It helps data providers and data consumers to exchange data in clicks, not projects to heavily reduce the data integration efforts which currently costs Time, Budget Motivation in analytics projects. It consequently addresses use cases ranging from external data integration and harmonization to cross-company data collaboration between business partners that use SAP Data Warehouse Cloud. For the time being, commercial transactions remain outside of the data marketplace and an embedded license management allows a BYOL (bring your own license) scenario to onboard existing Data Provider-Data Consumer relationships or new ones done on the SAP Store or via the Data Providers Sales channel.
This blog provides an overview of the data marketplace as well as shows the core steps of the processes for data consumers and data providers with links to detailed blogs.
The Data Marketplace serves as a strategic element of SAP Data Warehouse Cloud to ease the consumption of external data in order to combine it with internal (SAP) data.
Timing Information: The Data Sharing Cockpit of the Data Marketplace to list Data Products is already productive with wave 2202.1 on selected SAP Data Warehouse Cloud tenants. With Wave 2202.2 – delivered in CWC in 2022, the Data Marketplace will be enabled in all SAP Data Warehouse Cloud tenants. In this blog post, certain topics that are foreseen but not yet communicated on the roadmap are stated with “in the product vision”.
From the realization of outside-in use cases to the creation of a holistic data platform/warehouse, the necessity to include external data is more important than ever. What do we mean by external data? Let us give you some examples use cases :
External Data is sourced from Commercial and Public Data Providers as well as from a Business Partner to harmonize with internal data and perspectives.
In a nutshell, with external data, we mean all data that you cannot extract from your company’s internal applications but get access to from an external party which can be a commercial data provider, an open data source, or a company where data is traded as part of a business process.
While this data is super helpful – especially in times of market disruption through innovation, pandemic times, or economic climate changes where historical data does not help you to project the future – integrating it can be super cumbersome and is today often the hurdle for data-driven innovation and cross-company collaboration. Why can this be the case?
Integration Challenges leave the data management and data consumption potentials on the table
In a nutshell, for enterprise-grade integration of external data with the current setup, the integration work lies with the data consumer, and almost always an IT project is required which in reality leads to the fact that the use case is not realized and potentials remain unaddressed.
With the Data Marketplace, this changes completely for the data consumer in three steps:
CONSUME. The data consumer can access external data in a few clicks into his SAP Data Warehouse Cloud without doing any integration work. All of the integration work is done by the Data Marketplace (after the Data Provider has once onboarded his views). This holds true for the initial load as well as for updates published by the data provider. As a consequence, the Data Marketplace allows you to standardize the inbound flow of external data where in a company today a mess of different setups makes it impossible to scale.
COMBINE. Once the data is integrated via the Data Marketplace, you benefit from the entire data management functionality of SAP Data Warehouse Cloud to cleanse, harmonize and prepare the data for consumption. One explicit feature to call out here is the Intelligent Lookup that helps you to bring together datasets that do not have a singular, technical JOIN condition which is almost always the case with external data and which is introduced in the following blog by Jan Fetzer. And ultimately, you can then easily consume the data with SAP tools such as SAP Analytics Cloud, SAP HANA Cloud, SAP Data Intelligence or 3rd Party Tools such as Jupytier Notebook, PowerBI, etc.
CONTROL. Ultimately – by being tightly embedded into the SAP Data Warehouse Cloud platform – the Data Marketplace allows you to setup an infrastructure with which you can manage your data inbound processes at scale, e.g. by leveraging spaces. Within the Data Marketplace, you benefit from Access control mechanisms that manage who can access which Data Product. This way a provider can represent his existing contracts as a digital twin and onboard his existing customer relationships while new ones can be created via the SAP Store.
All further questions to the Data Consumer perspectives can be found below in chapter 5 or within our FAQ. Link to consumer FAQ on help.sap.com will be added with the GA release in CW2, 2022.
In order to realize the above-described data consumer perspective, the data providers obviously play a crucial role. But why should they care?
How does the Data Marketplace for SAP Data Warehouse Cloud help?
All further information on the Data Provider perspective shall be answered in the E2E processes below or in the Provider FAQ on the SAP Help Page right here. Link to provider FAQ on help.sap.com will be added with the GA release in CW3, 2022.
The best way to understand the value proposition of the Data Marketplace is to see it in action. Feel free to take a look at the following 25-minutes long end-to-end demo video where you see how 2 data products are loaded and an additional one is listed and delivered. In addition, the Intelligent Lookup is shown to demonstrate how the native SAP Data Warehouse Cloud functionality helps you to work with the acquired data in an SAP context.
In order to truly understand the benefits of the Data Marketplace – as a data consumer or data provider, it is important to know how it operates. This chapter should give an overview and understanding of the setup while the following chapter will give more context for the processes.
The Data Marketplace connects all SAP Data Warehouse Cloud Tenants via a central catalog while the data delivery is orchestrated in a decentral fashion.
First of all, all participants in the Data Marketplace need access to an SAP Data Warehouse Cloud tenant. For companies that only want to use SAP Data Warehouse Cloud to offer and deliver data in the Data Marketplace and do not want to use it internally as a data warehouse solution, a special offer is available for €830 per month that can be retrieved by contacting email@example.com.
As a Data Provider, you use the Data Sharing Cockpit to list one or multiple SAP Data Warehouse Cloud artifacts as a so-called Data Product. For this Data Product, you can manage access via the License Management and updates via the Publishing Management. Based on this definition, a hidden “Data Product Space” is created in the Providers Tenant.
When a Data Consumer now discovers such a Data Product that he wants to load, he needs a license key to activate it (unless it is a free product) and can select the target SAP Data Warehouse Cloud Space in which he wants to consume the Data Product. Based on this selection, the Data Marketplace now creates a database connection between the generated “Data Product Space” and the selected target space.
Subsequently, the defined artifacts are automatically created and the data is replicated. Every time a new update is made available to the data, a new replication is triggered. Federated access is currently being investigated while replication was the priority to mirror the current process of flat-file delivery where a physical copy is shipped as part of the data contract.
This approach scales especially if multiple customers consume the same product (or products based on the same data) as the Data Provider only once needs to connect, list, publish and update the product while all data consumers that have subscribed to the product benefit immediately (or manually if they choose to manually control the update flow).
Consequently, the Data Marketplace connects all SAP Data Warehouse Cloud customers (currently within the same landscape, e.g. EU10, US10, etc.) in a matter of clicks, without an IT project required.
In this chapter, you get a high-level walkthrough of the main data marketplace processes. Further step-by-step guidance and field-level explanations can be found in more detailed blogs then.
In order to make data available on the Data Marketplace, you need a deployed graphical or SQL view in your SAP Data Warehouse Cloud populated with the data that you want to ship. Based on the origin and complexity of your data product, different onboarding mechanisms are possible. In a nutshell, you can distinguish between the following 5 archetypes:
From Manual Entry to Federated Access to a database – all types of SAP Data Warehouse Cloud connectivity are available for you to connect data for sharing with the Data Sharing Cockpit
In a nutshell, multifold data onboarding options are available and you can find a full list of supported sources systems right here.
Once the views are created that you want to list in the Data Marketplace, the processes in the Data Sharing Cockpit start. You need to create a Data Provider Profile or assign your user to an existing one via an activation key. With the Data Provider profile, you can describe your area of expertise, connect your LinkedIn page and tag the Industries, Data Categories, Regions that you serve with your data. In addition, you maintain the contact details to help curious data consumers to reach out.
With the Data Provider Profile, you set the frame for all further activity in the Data Sharing Cockpit
Data Product Listing is the process of defining the marketing mix of your data offering. For interested data consumers, you can provide all “4Ps” as you would call it in economics: Product, Price, Promotion, Placement.
It starts with descriptive information. You can maintain a free text that describes the product, its use case, its sources, etc. With the same taxonomy as in the Data Provider profile, you can tag your product with the applicable Data Category, SAP Application, Industry, and Country to optimize the search experience and likelihood to be found. In addition, you can maintain images as well as data documentation such as a metadata catalog or KPI definition document.
A further very important asset is the Sample Data representation. This is currently been achieved with a JSON upload (link with a tutorial video will be added soon) while later this will be realized with a filter on the actual data set. One or multiple samples can be maintained – ideally one for each artifact that is part of the data product – with each maximum 1000 records.
A further key setting is the maintenance of the delivery information. You can decide between the following data shipment types:
Furthermore, commercial information can be maintained. In general, the Data Marketplace runs on a “Bring your own License” setup. You can represent your existing license in the Data Marketplace to authorize access. This can be a license that a provider sells in his own sales channels or via SAP channels, the SAP Store. In addition, you can set your product to “Free of Charge” which consequently does not require a license key to access a product. For license products, you can then maintain price information as well as terms and conditions.
Last but not least and most importantly, you select the artifacts that you want to ship in an automated fashion (in case you have a direct delivery data product). In the most simplistic fashion, it can be one, entire view that you can ship as it is. At the same time, with one data product, you can also ship multiple views at once, e.g. transactional and master data views that can be joined on the consumer side. Furthermore, you can use data filter or column selection to base multiple data products on the same base view to accommodate the portfolio and GTM strategy.
Once you have defined these Data Product listings aspects, you are ready to list your product on the data marketplace. The entire process can be done in several minutes in case the Data Provider knows what exactly he wants to list and deliver.
The next two steps are only required for specific Data Product setups:
Most Data Products are being updated frequently or infrequently. This is where non-integrated setups create efforts for both data providers and data consumers. Data Providers need to create new data slices and data consumers need to find ways to ingest new incoming data. To reduce the friction of releasing and consuming data updates was one of the biggest motivations to build the Data Marketplace. This is why a dedicated module is available in the Data Sharing Cockpit with the Publishing Management.
For each Data Product, the Data Provider can easily define a new release that provides the Data Marketplace with the information the new content is available. Data Consumers that have activated the Data provide immediately receive the new data into their existing artifacts without any intervention required. At the same time, a “Manual Update” mode is available as well on the consumer side that allows the data consumer to decide when the new data shall be ingested. The Data Provider has also the option to lock a data product for the time where he updates his view to make sure no inconsistent data is shipped (e.g. to prevent data from being shipped to the customer while it is updated by the data provider).
In the first release of the Data Marketplace, a new data release always provides the full data set in the provider’s view(s) to be transferred. In the product vision, you will be able to define and ship incremental releases as well.
Finally, the Data Provider is able to set up licenses that make the commercial setup actionable. Therefore the Licensing Management module exists in the Data Sharing Cockpit. A license entitles one or multiple users to access one or multiple data products for a limited or unlimited time. In addition, a domain check can be set up to only allow users with a certain mail address, e.g.@sap.com, to activate a specific license. For a given license, the data provider can generate as many activation keys as he likes and can send them to the data consumer that can enter the license on the specific product page or in the “My Licenses” section.
There are four major licensing models that you can execute with the License Management:
As the Data Products and the License Management is decoupled, any combination is possible. In the product vision, you will also be able to set a data filter within the license to restrict data access via the license to a certain data scope.
The Data Consumer part is under construction and will be released with the General Availability of the Data Marketplace in CW3/2022.
Data Consumer – 1. Find Data Products
Data Consumer – 2. Elaborate Data Products
Data Consumer – 3. Load Data Products
Data Consumer – 4. Track Data Deliveries
Data Consumer – 5. Use Data Product’s artifacts
Many that are new to the topic of external data and data sharing are curious about the use cases that can be covered with this cross-tenant data exchange technology. In the following, I would like to give an overview of the archetypes of use cases that are possible. Check the chapter below on the Data Provider and Data Product Portfolio below.
under construction for GA in CW3/2022
under construction for GA in CW3/2022
You can find the overall roadmap for SAP Data Warehouse Cloud on the Roadmap Explorer right here.
As a Data Provider, reach out to firstname.lastname@example.org in case you want to become a Data Provider on the Data Marketplace. There are different ways to get your listing in front of our SAP customers based on your data assets and GTM strategy.
As a Data Consumer, you can load data with any kind of access to SAP Data Warehouse Cloud. If you are not yet a customer, just check out our up to 90 days trial offer right here.