Execute SQL queries against delta format tables in Azure Data Lake using Jupyter Lab notebook

[jeffelliott]

Scenario: We have delta format tables residing in an Azure data lake. We want to be able to run SQL queries against the delta tables from within a Jupyter notebook in Jupyter lab in SAP Data Intelligence. We would like to be able to use the Data Intelligence connection object to the Azure data lake, if possible.

Questions:

1. Is it possible to achieve the above scenario using Python in a Jupyter notebook?
2. If yes to #1, how is this accomplished? Are there examples or documentation showing the method?

Thanks.

[MarioDeFelipe]

Hello Jeff

Assuming you have gone through this and it falls short

https://help.sap.com/docs/SAP_DATA_INTELLIGENCE/97fce0b6d93e490fadec7e7021e9016e/a4a74c394c874270a28...

Can you explore the option to run the Azure connection through SAP DataSphere?

The way I see it; we use Data Intelligence Cloud for replication options, CDC, Kafka, and similar. DataSphere for federated queries, Integration, and Management. Check out if this provides you with more information;

https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/eb85e157ab654152bd68a87140...

[jeffelliott]

Hi Mario,

Thanks for the information.

Yes, you are correct. I have looked at the options in DI Modeler as well. Specifically, the SQL Consumer V2 doesn't appear to support Azure Data Lake Storage v2.

I am not familiar with Datasphere. Additionally, at present, we do not have instance to allow us to explore and try anything out. I will keep this in mind, however, in case the opportunity arises in the future.

I appreciate the feedback and ideas.