Deploy JupyterHub and JupyterLab on Ubuntu Server

Prateek Verma / 11 May 2023

computational python

This guide was compiled from documentations from JupyterHub [1], JupyterLab [2], Conda, Ubuntu and other sources such as StackOverflow.

Prerequisites

• Ubuntu Server with SSH access to shell
• Python (modern Ubuntu servers come bundled with Python)

Goals

• JupyterHub + JypyterLab will be installed as a webservice in /opt using Python.
• Conda will be installed globally. Conda is intended to be the preferred Python environment to serve the actual computing tasks of the users.
• A shared conda environment will be created for all users. They can use it, but not modify it.
• Demonstrate how users can create their own private conda environments.

Steps

Create python virtual environment for JupyterHub

Create a Python virtual environment in /opt for jupyterhub.
sudo python3 -m venv /opt/jupyterhub/
Tip: You may need to install python3-venv first to be able to create a virtual environment.

Install JupyterHub and JupyterLab

• Install jupyterhub, jupyterlab, and UI related packages in the jupyterhub virtual environment we just created.
  sudo /opt/jupyterhub/bin/python3 -m pip install wheel
sudo /opt/jupyterhub/bin/python3 -m pip install jupyterhub jupyterlab
sudo /opt/jupyterhub/bin/python3 -m pip install ipywidgets
  Tip: You can specify the version of the package like this: pip install jupyterhub==3.1 but that may create incompatibility issues and problems with starting of the JupyterHub service successfully. I installed the latest versions at this step, but went back and downgraded jupyterlab according to my specific needs.
• Install nodejs, npm and then, configurable-http-proxy systemwide. These are required by jupyterlab.
  sudo apt install nodejs npm
sudo npm install -g configurable-http-proxy

Configure JupyterHub

To keep things together, create any configurations under the virtualenv, i.e. under /opt/jupyterhub/etc/.

• Create a folder for jupyturhub configurations and navigate to it.
  sudo mkdir -p /opt/jupyterhub/etc/jupyterhub
cd /opt/jupyterhub/etc/jupyterhub
• Generate the default configuration file.
  sudo /opt/jupyterhub/bin/jupyterhub --generate-config
• Edit the configuration file to make JupyterLab interface start by default, rather than Jupyter Notebook.
  sudo nano jupyterhub_config.py
  And update the following parameter to:
  c.Spawner.default_url = '/lab'

Set up JupyterHub as a system service

Create a folder for systemd service file.
sudo mkdir -p /opt/jupyterhub/etc/systemd
Create a service file.
sudo nano /opt/jupyterhub/etc/systemd/jupyterhub.service

Paste the following into the service file:

[Unit]
Description=JupyterHub
After=syslog.target network.target

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin"
ExecStart=/opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target 

Create a symlink to this sevice file from the system-wide systemd folder.
sudo ln -s /opt/jupyterhub/etc/systemd/jupyterhub.service /etc/systemd/system/jupyterhub.service

Tell systemd to reload its configuration files, and then enable and start the service.
sudo systemctl daemon-reload
sudo systemctl enable jupyterhub.service
sudo systemctl start jupyterhub.service
sudo systemctl status jupyterhub.service # check status of service

You can now test the service by navigating to http://<your-server-ip>:8000 in your browser. You should see the JupyterLab interface.

Install Conda

Install conda systemwide. This involves a series of simple commands, more details of which can be found here [3].
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
sudo install -o root -g root -m 644 conda.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" | sudo tee /etc/apt/sources.list.d/conda.list
sudo apt update
sudo apt install conda
sudo ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh

This installs conda in /opt/conda. If you wish to use the conda command in terminal now, without restarting the shell/computer, it is available from /opt/conda/bin/conda.

Configure Conda

Install a default conda environment for all users. Note that this is different from the base environment that is included when Conda is first installed. It's a good practice to not work in the base environment.

• Create a folder for new conda environments.
  sudo mkdir /opt/conda/envs
  Tip: At this time, also give write permissions to the user/group intended to install packages to this environment. Otherwise conda install will fail with the EnvironmentNotWritableError error.
  sudo chown -R root:<groupname> /opt/conda/envs
sudo chmod -R 775 /opt/conda/envs
• Create a new conda environment in this folder. Here we will install tensorflow (which includes Python).
  sudo /opt/conda/bin/conda create --prefix /opt/conda/envs/cpuenv tensorflow
• We need to install ipykernel in this environment so that it can be used with Jupyter. Be careful not to install ipykernel in the default base environment.
  sudo /opt/conda/bin/conda install --prefix /opt/conda/envs/cpuenv ipykernel
• Make this environment visible to Jupyter by installing the kernel spec system-wide by putting it into /usr/local. This step makes the environment visible on the JupyterHub UI. Read more here [4].
  sudo /opt/conda/envs/cpuenv/bin/python -m ipykernel install --prefix /usr/local/ --name 'cpuenv' --display-name "CPU Environment"
  A folder /usr/local/share/jupyter/kernels/cpuenv will be created.

Per-user Conda environments

To let users set up their own environments, they should login into shell and run conda init or /opt/conda/bin/conda and thereafter use conda to create their own environment. I believe they should create the environment in their home directory (to which they have write permissions to) instead of the /opt/conda/envs directory. They will also need to install ipykernel (required by Jupyter) in the new environment. Finally, they can enable the kernel to appear in JupyterHub UI by running the following command:
/path/to/env/bin/python -m ipykernel install --user --name 'user-env' --display-name "User Environment"

Additional Tasks

Install Git on the Ubuntu server

sudo apt install git-all

Create users and user groups on Ubuntu

Before a user can use JupyterLab, they need to be an Ubuntu user first. For convenience, we will also add them to a group called jupyterhub-user.
sudo adduser <username>
sudo addgroup jupyterhub-user
sudo adduser <username> jupyterhub-user

Set a user as admin on JupyterHub

Navigate to /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py and add the following line:
c.Authenticator.admin_users = {'<username>'}
Tip: make sure to use {} instead of set()

Restart the service:
sudo systemctl restart jupyterhub.service

Disable conda from automatically activating its base environment

The following command will add a .condarc file in the home directory of the user.
conda config --set auto_activate_base false

Create a read only shared folder for all users

Create a folder in /srv. You can add data to this folder however you like.
sudo mkdir /srv/data

Note that the permission of this folder are drwxr-xr-x by default. Next, optionally, create a symlink to this folder in each user's home directory. To automate this, you can add this link in /etc/skel; anything placed in this folder shows up in the home directory of any new user created.
sudo ln -s /srv/data /etc/skel/data

Note 1: Make sure to do this for existing users' home dirs manually.
Note 2: You will want to grant at least one user (besides root) the ability to write (upload) to the /srv/data folder. This user can then add data from the JuoyterLab UI, and remaining users can read from it.
sudo chown -R <username>:<groupname> /srv/data

Create a folder for users to share data with each other

Each uploader will have write access to their files (only), while everyone else will have read access to those files.
sudo mkdir /srv/shared
sudo chown root:<jupyterhub-users-group> /srv/shared

Using chmod g+s tells new files to use the default permissions for the group jupyterhub-users-group which is rw-r--r--.
sudo chmod 777 /srv/shared
sudo chmod g+s /srv/shared

Finally, add a symlink to this folder in /etc/skel/ for future users and in each existing user's home directory.

Let users change their password

<your_server_ip>/hub/auth/change-password

Install git extension for JupyterLab

Make sure you have git >= 2.x installed in Ubuntu and JupyterLab >= 3.0 installed in the python3 virtual environment that houses JupyterLab.
Note that jupyterlab-git is not compatible with jupyterlab 4 yet.
sudo /opt/jupyterhub/bin/pip install --upgrade jupyterlab jupyterlab-git

Downgrade jupyterlab if needed.
sudo /opt/jupyterhub/bin/pip install jupyterlab==3.6.3
Restart the JupyterHub service.
sudo systemctl restart jupyterhub.service
The JupyterLab UI should show the git button and the menu item. Additionally, it should also show the git extension already enabled/installed in the left sidebar.

You can now clone a git repository from the JupyterLab UI. When asked for a password, make sure to provide a fine-grained (preferred) or a personal access token (not preferred) instead of your account password. GitHub no longer supports third-party authentication using passwords. You can create a token from your GitHub account settings.

Make your custom python package available to all users

I prefer the strategy of adding our package's folder path to a conda environment installation instead of adding the path to users' home directories or add the package itself to the site-packages directory of the environment [5].

First, create a folder and set ownership for the desired user so they can upload their package to this folder.
sudo mkdir /srv/_packages
sudo chown -R <username>:<groupname> /srv/_packages

Upload files to this folder however you like. I cloned my package from GitHub so that my folder looks like:
/srv/_packages/my_package_1

To make this package available to other users (note that they have read access to the _packages folder by default, which is perfect), we will add the path to this folder to conda's site-packages directory, for every environment that is created. Just adding the path to the base environment's site-packages, did not add it to other environments for me [5].
Create a *.pth file in the site-packages directory of the base environment.
sudo nano /opt/conda/lib/python3.10/site-packages/my_packages.pth
Add the path to the package folder in the file. Save and exit.
/srv/_packages

Create a symlink to this file in the site-packages directory of every new environment that is created. (I am not sure if this is intended or I am doing something unnecessary, but I was able to import packages from cpuenv environment only after I created this symlink.)
sudo ln -s /opt/conda/lib/python3.10/site-packages/my_packages.pth /opt/conda/envs/cpuenv/lib/python3.11/site-packages/my_packages.pth

Things I am still figuring out

• How to remove the default python kernel showing up in JupyterLab UI for the current version of JupyterLab.
• Successfully set up a static ip address for the server in your local network using netplan. Once a static ip address is obtained, you can also use LetsEncrypt to get a free SSL certificate for your server. This will allow you to access your server using https instead of http and also get rid of the Not Secure warning in the browser.