Create custom Jupyter Lab Python kernels on SAP Data Intelligence

[LarissaH]

In this tutorial, I will show you how to create additional python kernels on the SAP Data Intelligence to be used within the Jupyter Lab and the ML Scenario Manager.

I, like many of you probably, really like working with the SAP Data Intelligence. What I probably use most, is the Jupyter Lab application, which can be activated and used just like any other known notebook server. Here, you can access your datasets and data sources, you can visualize data and try out code, and produce first results.

But maybe you will be in the same situation as I am: You are working on several projects that require different versions of python libraries, or their dependencies require different versions. You need to control exactly, what you have installed and what is not installed, so this gets pretty messy in one single environment.

The solution: Use different python kernels for different use cases / projects!

When you open a Jupyter notebook in the Jupyter Lab, you will see the label "Python 3" on the top right corner. This indicates on which kernel your notebook is currently running. If you click on it, a small popup window will appear, giving you the option to switch between different kernels. But: There are no other proper kernels (except for kernels from other sessions, which will not be permanent).

So, let's start. Let's assume that you want to use LightGBM on the SAP Data Intelligence. Currently, it is not installed.

Step 1: Open a terminal and activate conda.

Python and python libraries on the SAP Data Intelligence are managed with conda (instead of pip, which might be known to a broader audience). When you use conda on the DI for the first time, you will have to initialize it, simply typing:

conda init bash

Or any other shell option that you want to use. It might be the case that you have to re-load the terminal, that you are using. In this case, simply close the terminal tab and open a new one. Be sure that now the environment has loaded, this is indicated by a name in parenthesis at the beginning of each new terminal line. Most probably, it is called "base":

(base) labuser@jupyter-<some-cryptical-numbers>:~>

If this is not the case, you can activate the environment by typing

conda activate <env-name>

Otherwise, if you want to start completely new, you can create your own environment with the following command and activate it later on:

conda create --name <env-name> python=3.6

In case of questions or problems, please also refer to the official and extensive conda documentation.

Step 2: Install wanted libraries.

This is really simple, as you can just install all the libraries you want to use via conda, for example like this:

conda install lightgbm

Step 3: Install and activate ipython for your kernel.

You need to install ipykernel inside of your environment, in order to connect it to the Jupyter Lab in the next step. This can be done by typing:

conda install ipykernel

python3 -m ipykernel install --user --name <env-name> --display-name "Python 3 (<env-name>)"

In the last line, you need to replace <env-name> with the name of the environment that you installed your wanted libraries in, so for example "base" or your environment name, if you created one by yourself beforehand.

Step 4: Reload Jupyter Lab.

Now you can reload your Jupyter Lab page and review, if the new environment is there for selection.

Be aware: If your Jupyter Lab instance is reloaded, you will lose all environments you just created and everything will be reset to default.

I hope, that this tutorial helped you proceed with the SAP Data Intelligence and opened the door for new possibilities. Let me know if you have any questions!

[Cocquerel]

How do you open the "Launcher" tab in Jupyter Lab ?

[LarissaH]

You can find the Launcher by simply clicking on the "+"-sign on the top left corner of the Jupyter Lab application!

I hope, this helps 🙂

[Cocquerel]

Thanks, it was not obvious to me

[RaulFlorian]

Hi Larissa, exist some form that the environment is not delete? How prevent the jupyter kernel reload?

[LarissaH]

Hi Raúl,

there will be situations, where you need to restart your jupyter instance. Up until this moment, the environment will stay stable (as long as you do not have to restart the instance).

Unfortunately, I have no idea how to prevent the reset in case of a restart ... Maybe other users have an idea about that?

All the best,
Larissa

[Vitaliy-R]

Thank you for documenting these steps. In the past, I tried it as well to get a kernel but gave up describing it, when found that it disappears after the restart.

I thought about looking into directories, if I can find and configure directories that are preserved between restarts, but I had no time for that.

Cheers and looking forward to more excellent content from you!

-Witalij

[Cocquerel]

I have created a new conda environment as described in this blog. I'm missing the package "notebook-hana-connector" in this new environment. I was not able to install it neither using conda install nor pip install. It seems this package does not exist in any of the default channel. is there a way to copy it from the "base" environment ?

[marcus_schiffer]

Hi,

 

is the issue solved that the environment is always deleted when the instance is restarted e.g. after the regular maintenance ?

That problem leaves a lot of additional work when we need to restore the env every time.

 

Any help appreciated.

 

Regards

Marcus