de.NBI summer school 2017 on cloud computing

As part of the de.NBI Summer School on Cloud Computing for Bioinformatics from 25th June - 1st July 2017 in Giessen (see also http://www.denbi.de/22-training-cat/training-courses/256-cloud1) PhenoMeNalist Christoph Ruttkies (IPB Halle) shows how to lift a PhenoMeNal CRE on the local de.NBI cloud (see also the earlier blog post http://phenomenal-h2020.eu/home/2017/01/13/phenomenal-meets-denbi/).

OpenStack PhenoMeNal Installation Tutorial

Part I: Set up Deploy Node in OpenStack

The Deploy Instance Node will be used to deploy PhenoMeNal in OpenStack as it provides all prerequisites needed (docker, git).

• set up an instance in OpenStack with the provided "css_ubuntu_git_docker"
• assign a floating IP for external access
• assign a SSH key pair of your local machine

After the creation of the Deploy node:

• login to the running instance via

ssh ubuntu@FLOATING_IP_OF_DEPLOY_NODE
# root permissions are required as we need to use docker
sudo su -
# create a new folder in which you will put all necessary files in the next steps
mkdir css2017 && cd css2017

Part II: Install prerequisites for cloud deployment

• Checkout the cloud deployment from GitHub

git clone --recursive https://github.com/phnmnl/cloud-deploy-kubenow.git
cd cloud-deploy-kubenow
git checkout development/bucetin
git submodule update

Part III: Create the KubeNow configuration for your OpenStack

• Download and source the “Download OpenStack RC File v3” (Compute -> Access & Security), which is a small shell script setting a number of environment variables required for access to the OpenStack API
• Copy cloud-deploy-kubenow/config.ostack.sh-template to cloud-deploy-kubenow/config.ostack.sh
• Next, we need to modify the cloud-deploy-kubenow/config.ostack.sh to match our OpenStack installation. The required information is spread across several OpenStack horizon pages, or available through the OpenStack command line client

#!/usr/bin/env bash

# Cloud Prefix Name
export TF_VAR_cluster_prefix="your_unique_prefix"
# Path to "Download OpenStack RC File v3" from https://cloud.computational.bio.uni-giessen.de/horizon/project/access_and_security/?tab=access_security_tabs__api_access_tab
export OS_CREDENTIALS_FILE="/path/to/phenomenal-giessen-openrcv3.sh"

# Specific for de.NBI Giessen openstack
export TF_VAR_floating_ip_pool="NETWORK_NAME"
export TF_VAR_external_network_uuid="NETWORK_ID"

# If your cloud provider is not allowing external nameservers, please specify here or
# uncomment and leave empty for provider automatic configuration
# export TF_VAR_dns_nameservers=""

# Master configuration
# Note: too small flavors might cause diffuse errors on your installation
export TF_VAR_master_as_edge="true"
export TF_VAR_master_flavor="de.NBI.default"

# Node configuration
export TF_VAR_node_count="3"
export TF_VAR_node_flavor="de.NBI.default"

# Gluster configuration
export TF_VAR_glusternode_count="1" # 1 - 3 depending on preferred replication factor
export TF_VAR_glusternode_flavor="de.NBI.default"
export TF_VAR_glusternode_extra_disk_size="100" # Size in GB

# Galaxy
export TF_VAR_galaxy_admin_email="yourname@domain.org"

# create password with e.g. head /dev/urandom | sha256sum | head -c 16 ; echo
export TF_VAR_galaxy_admin_password="yournewpassword"

# Jupyter 
# create password with e.g. head /dev/urandom | sha256sum | head -c 16 ; echo
export TF_VAR_jupyter_password="yournewpassword"

Part IV: Lifting up PhenoMeNal

• source the downloaded OpenStack RC File v3 (you are asked for your password)

source /path/to/phenomenal-giessen-openrcv3.sh

• deploy the cluster to OpenStack

./phenomenal.sh deploy ostack -c CONFIG_FILE

• At this stage you should see the progress in the "Network -> Network Topology -> Graph" of your OpenStack project
• Terraform talks to openstack
• Ansible talks to virtual hosts
• Launching kubernetes / helm -> packaging for kubernetes

(Optional) Part V: Uninstalling PhenoMeNal

./phenomenal.sh destroy ostack -c CONFIG_FILE

PhenoMeNal Cloud Administration

You can manage the Kubernetes (Phenomenal) cluster via command line or by using the Kubernetes Dashboard.

Dashboard Administration

• Access the Kubernetes Dashboard in your browser

http://dashboard.FLOATING_IP_ADDRESS_OF_MASTER.nip.io/

• Check available settings e.g. Nodes

(Optional) Command line Administration

• Login on the master node using the private key file vre.key in deployments

ssh -i ~/css2017/cloud-deploy-kubenow/deployments/id-phnmnl-config.ostack/vre.key ubuntu@FLOATING_IP_ADDRESS_OF_MASTER

• some useful commands (just for your information)

# show all pods
kubectl get all
kubectl get all --all-namespaces

# get more detailed information about pods
kubectl describe $pod

# restart galaxy (try not to use this for the moment ;))
con=$(kubectl get pods | grep Running | cut -d" " -f1 | grep galaxy)
kubectl delete pods/$con
sleep 4
con=$(kubectl get pods | grep Running | cut -d" " -f1 | grep galaxy)
kubectl exec -i -t $con -- /galaxy/run.sh --restart

# delete a whole pod (try not to use this for the moment ;))
kubectl delete $pod --all

Usage of PhenoMeNal Cloud e-Infrastructure via Galaxy

All Phenomenal tools are accessible by the Galaxy environment shipped by the cloud installation. You can access Galaxy by the provided URL:

http://galaxy.FLOATING_IP_ADDRESS_OF_MASTER.nip.io/

• login with the provided Galaxy email address and password (cloud-deploy-kubenow/config.ostack.sh)

Part I: Use Case - Statistical (Sacurine) Workflow

• Access the workflow file and input files for the Statistical Workflow and download them to your local machine

Part II: Use Case - MetFrag Workflow