Container Service: CloudStack

The CloudStack Container Service (CCS) orchestrates provisioning of Kubernetes managed container clusters integrated in your CloudStack environment. CCS builds container clusters Kubernetes running on CoreOS (Stable Channel) VMs. Once provisioned, you can configure the cluster and deploy containers using standard Kubernetes tools such as kubectl. Currently, CCS provides the following features for managing container clusters:

  • Embeds the Kubernetes 1.11.3 Dashboard in the CloudStack Console to deploy containerized applications
  • Injects CloudStack managed SSH keys into Kubernetes clusters
  • Monitors Kubernetes cluster health
  • Connects Kubernetes clusters to CloudStack isolated networks with integrated DNS (using kubedns)
  • Support for CoreOS guests


Prerequisites for creating CloudStack containers

Each container cluster has a kubernetes master and some kubernetes nodes. To create a container cluster, you have to:

  • create/register an SSH key pair (which is injected into kubernetes master and nodes and used for logging into them) 
  • create a service offering (which is used in creating kubernetes master and nodes) 
  • optionally, prepare a CoreOS template (which kubernetes master and nodes are created from) 
  • optionally, create an isolated network (where kubernetes master and nodes are created on)

Step 1: Create/Register SSH Key Pair

To create/register an SSH key pair, please perform the following steps:

  1. On the left panel, select Accounts.
  2. Choose "SSH Key Pairs" in "Select view" dropdown.
  3. Click button  “+ Create a SSH Key Pair” on the right, you will see a dialog:

    If public key is set, CloudStack will register the public key. You can use it through your private key. If public key is not set, CloudStack will create a new SSH Key pair. In this case, please copy and save the private key. CloudStack will not keep it.

    For detailed instructions on generating keys for a user, please visit "Registering Keys for Users".

Step 2: Create a service offering

Container service requires a service offering with at least 1 CPU and 1024MB RAM.

For detailed instructions on creating a service offering, please visit "Service Offerings ".

Step 3: Prepare a CoreOS template (optional)

CoreOS Container Linux is the leading container operating system designed to be managed and run at massive scale, with minimal operational overhead.Containers are key to the modern data center. For developers, it has never been easier to ship new application versions. Containers easily plug into your CI/CD pipeline for automated build, test, and deployment environment with an audit trail.The container engines Docker and rkt are configured out of the box, ready to run your applications. Through the continuous stream of updates, Docker and rkt are automatically and continuously updated with the operating system.

We’ve prepared a CoreOS template used for container cluster, which is downloaded from CoreOS. You can also create a CoreOS template and upload it to CloudStack so it can be used to create a container cluster.

If CoreOS template is not specified when creating a container cluster, a default CoreOS template with Kubernetes 1.11.3 build-in will be used.

Step 4: Create an isolated network (optional)

The isolated network should

  • Support services: SourceNat, UserData, Firewall, PortForwarding, Dhcp
  • Be implemented
  • Port 443 not in use (by any firewall rule, port forwarding rule and load balancing rule).
  • Egress traffic are allowed. If default egress policy is Deny, then add egress rules. If default egress policy is Allow, do not add egress rules. Kubernetes master and nodes will download necessary packages and configurations from official kubernetes repo.

For detailed instructions on creating an isolated network, please visit "CloudStack Network ".

If network is not specified when create a container cluster, an isolated network named “<container name>-network” will be created with default network offerings for containers.

Creating a CloudStack container cluster

To create a container cluster, please perform the following steps:

Before creating a container, you must perform the steps mentioned under "Prerequisites for creating CloudStack containers".

  1. On the left panel, select “Container Service”.
  2. Click the “+ Add container cluster” button.
    The "Add container cluster" dialog box displays.

  3. Enter the following information and click "OK".

    Field NameRequired/OptionalEditable afterwardsDescription



    The name for the container cluster.




    The description for the container cluster.

    ZoneRequiredNoThe zone where the container cluster is.

    Service Offering



    The Compute Offering to use for kubernetes master and nodes in the container cluster.




    The network to use for the container cluster. If empty, then {container name}-network will be created.

    HA EnabledOptionalYesIndicates if multiple master nodes (3 master nodes) has to be created to provide High Availability feature for clusters.




    The template to use for kubernetes master and nodes in the container cluster. If empty, it will use default CoreOS template "ccs-template-leaseweb4-1106".

    Root Disk size



    The size of root disk for kubernetes master and nodes in the container cluster.

    Cluster Size



    The number of kubernetes nodes in the container cluster (excludes the kubernetes master)

    SSH keypair



    The SSH key pair to use to log into the kubernetes master and nodes in the container cluster.

    Private registry



    Whether or not to use a private container registry. By default, the Docker public registry will be used.

    Checked the “Private Registry” option to use an external or private container registry.

    UsernameThis is your Docker username
    PasswordThis is your Docker password
    URLThis is your Private Docker Registry FQDN
    EmailThis is your Docker email
  4. When a container cluster is created successfully, you can see the list of created clusters:

Viewing a CloudStack container cluster

You are able to get more information of a container cluster by clicking it in the list.

It displays the cluster details, dashboard, instances, and firewall.

Viewing details of container cluster

Click the "Details" tab to view the details of a container cluster.

Here are the explanation of each field (except the fields in creating a container cluster):

Field NameDescription


The name for the container cluster.

Zone Name

The zone in which the cluster is deployed.

# of CPU Cores:

Total CPU cores used in the container cluster.

Memory (in MB):

Total memory cores used in the container cluster.


The current state of the container cluster.

API Endpoint

The API endpoint for the container cluster. This endpoint is used to connect kubectl to the container cluster.

Dashboard Endpoint

The URL for the Kubernetes Dashboard for the container cluster


The username used to authenticate to the container cluster. This username is used to authenticate when connecting to the cluster with kubectl.


The password used to authenticate to the container cluster. This password is used to authenticate when connecting to the cluster with kubectl.

Viewing Kubernetes Dashboard

Click the "Dashboard" tab to view the Kubernetes dashboard.

If the page is empty, then click “Pop-out” on the right-up, a new page will pop up.

Viewing Instances

Click the "Instances" tab to view the list of all instances in that container cluster.

Viewing the Firewall

Click the "Firewall" tab to view the the network configurations of the container cluster .

You are also be able to add firewall rules, load balancing rules or port forwarding rules to access your Kubenetes master or nodes and your services

For detailed instructions on configuring a network, please visit "Managing Networks ".

For detailed instructions on accessing kubernetes master or nodes, please visit "Accessing the Kubernetes master and nodes ".

For detailed instructions on accessing services, please visit "Accessing the Kubernetes service ".

Managing a CloudStack container cluster

Once you create a container cluster, you can operate it. The state of container cluster will change accordingly.

For example, for a Running container cluster, you can stop, destroy, edit, and resize (see below, buttons from left to right) it.


For a Stopped container cluster, you can start, destroy, edit, resize, and change service offering it.

Different states of a container cluster

Here are a list of all the states of a container cluster.



Initial state of a container cluster when has been defined but no resources consumed


Necessary resources are provisioned and container cluster is in operational ready state to launch containers


State of the failed to create container clusters


Resources needed for container cluster are being provisioned and the container cluster is being configured and started


Resources for the container cluster are being destroyed


All resources for the container cluster are destroyed, Container cluster may still have resource like persistent volumes provisioned


Transient state in which resources are either getting scaled up/down


State to represent container clusters which are not in expected desired state (operationally in-active control place, stopped master/nodes etc)


State in which container cluster is recovering from alert state


State in which resources for the container cluster is getting cleaned up or yet to be cleaned up by garbage collector


End state of container cluster in which all resources are destroyed, cluster will not be useable further

Stopping a container cluster

Dependent on the current state of the container cluster, this action stops it.

All resources for the container cluster will be destroyed.

Starting a container cluster

Dependent on the current state of the container cluster, this action starts it.

Resources needed for container cluster will be provisioned and the container cluster will be configured and started.

Destroying a container cluster

All resources for the container cluster will be destroyed and cleaned up.

Recovering a container cluster

If container cluster is in an "Alert" state, then you can recover it from alert state to normal state by clicking the below icon

Scaling In/Scaling Out a container cluster

Click the "Resize container cluster" button to resize a container cluster that is in Running/Stopped/Alert state.

There are two options: scale in and scale out. The state will be changed to Scaling and back to Running if it is in Running state.

  • Scale out
    Allocate new kubernetes nodes with configurations. 
    Select Action “Scale Out”, enter new “Cluster size”, and click “OK”. 

    The Cluster size is the number of kubernetes nodes (except kubernetes master)

    Once done, kubernetes nodes are created and added to kubernetes cluster automatically.

  • Scale in
    Destroy existing kubernetes nodes. 
    Select Action “Scale In”, enter “Cluster size”, and click “OK”. 

    The Cluster size is the number of kubernetes nodes (except kubernetes master)

    Once done, the kubernetes nodes with bigger number (the ones that were created later than others) will be destroyed. If some kubernetes nodes are expunged out of band (for example, in Cloudstack dashboard), then CCS will remove these kubernetes nodes from container cluster at first.

Editing Cluster Name and Root disk size

  1. For the selected cluster, in the "Details" tab, click the "Edit" button.
  2. In the "Name" field, enter a new name for the cluster.

  3. In the "Root disk size (GB)" field, enter the new size for the disk.

    The disk of all VMs (including Kubernetes master and nodes) will be resized, even if they are Running or Stopped.

    As we are using KVM, it is only possible to enlarge the size. Due to limitation of CoreOS, the operation system in running VMs can NOT recognize the new disk size.  CoreOS system will recognize the new disk size and enlarge the file system, after a reboot.

Changing service offering of a container cluster

The container cluster should be in Stopped state.

Click the "Compute Offering" icon.

Select a “Compute offering” from the list and click “OK”. 

The new compute offering will be applied on all Kubernetes master and nodes. 

Configuring network of a container cluster

Once you select a container cluster, from the "Firewall" tab, you can view and configure the network.

Alternatively, you can also access the page from Networks page

Accessing a CloudStack container cluster

Accessing the Kubernetes dashboard

To access the kubernetes dashboard of container cluster, you have to download the certificate and import certificate in your browser. The certificate is self-signed and used for internal communication between kubernetes master and nodes, and also http server for kubernetes dashboard.

  1. Download certificate
    In the "Container Service" screen, click the “Download CA Certificate” button, and click “OK”.
    the certificate will be stored in your system.

  2. Import certificate in your browser
    You need to import the certificate in your browser (eg IE, Chrome, Firefox). 
    Once it is done, a certificate named “cloudstack” will be added in your certificates list. If you are not able to add it, please remove the certificate with “cloudstack” and retry.

  3. Kubernetes dashboard
    You are able to access the kubernetes dashboard if the certificate is imported. By clicking the “Dashboard” tab in the details page, you can view the dashboard.

    If the screen is empty, then click “Pop-out” on the top-right of the screen, a new page will pop up.

Accessing the Kubernetes API server

To access the Kubernetes API server, you need to download kubectl and execute the following command.

kubectl <COMMAND> -s <endpoint> --username=<username> --password=<password> -insecure-skip-tls-verify=true

The endpoint, username and password can be found in the details of container cluster. For example,

Accessing the Kubernetes master and nodes

To access the Kubernetes master and nodes, you need to add port forwarding rules and firewall rules in the network configurations in CloudStack.

  1. Add port forwarding rules

    Please note, port 443 is used for kubernetes-dashbaord

  2. Add firewall rules (port 443 is used for kubernetes-dashbaord)

  3. You can now to log into kubernetes master and nodes, using the SSH key you specified when creating the container cluster.

    The username is “core” not “root”. To modify the files or execute commands in the system, you have to use “sudo”.

Accessing the Kubernetes service

You can access your services by creating load balancing rules or port forwarding rules, along with firewall rules.

For example, if you create an ngnix service in the container cluster.

  1. Create nginx service in Kubernetes.

  2. The port of nginx is 80. The NodePort is randomly picked up from range 30000-32767. 
    In this example, the NodePort is 32446. 

  3. Create a load balancing rule (public port is 8080, private port is 32446/NodePort) to all Kubernetes nodes.

  4. Open the public port 8080 on firewall.

  5. You can access the nginx service by the public IP and public port.

  6. You can also specify the NodePort (in range 30000-32767) by editing kubernetes service on Kubernetes dashboard.

FAQs about CloudStack Container Service

How to use your domain (name, ssl, and key) for Kubernetes dashboard

By default, the kubernetes dashboard URL is using the Public IP of the container cluster.

For example, https://<Public IP address>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/

It is also possible to use your own certificate, key and domain.

  1. Log into kubernetes master
  2. Put your server certificate into /srv/kubernetes/leasewebcloud.crt (/opt/bin/leasewebcloud.crt in Alpha version, you can use any other filename). If there are certificate chains, put intermediate certificate right after server certificate.
  3. Put your key into /srv/kubernetes/leasewebcloud.key.
  4. Add a new line in /etc/systemd/system/kube-apiserver.service (Please make corresponding change if you use other filenames).
    “--tls-sni-cert-key=/srv/kubernetes/leasewebcloud.crt,/srv/kubernetes/leasewebcloud.key \”
  5. Restart kube-apiserver by “sudo systemctl restart kube-apiserver”

If the domain name can be resolved, you will be able to access the kubernetes dashboard by URL https://<DomainName>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/

For more information, click here.

How to upgrade Kubernetes version

Due to the rapid development of the upstream open source project, the Kubernetes version LeaseWeb delivers will lag behind the most recent community release [8]. The supplied versions in the alpha and beta environments and the different regions will most likely differ as well.

  1. Get current version
    You can determine the Kubernetes version on master and nodes by issuing the following command

    Get kubernetes version
    # for kubernetes master
    kubectl version -s <endpoint> --username=<username> --password=<password> -insecure-skip-tls-verify=true |grep "Server Version"
    # for kubernetes nodes
    kubectl get nodes -s <endpoint> --username=<username> --password=<password> -insecure-skip-tls-verify=true

    and nodes

    Alternatively, you can log into kubernetes master, download kubectl from URL to /opt/bin/kubectl if it does not exist in kubernetes master and execute the commands without endpoint,username and password.

    Get kubernetes version via kubectl
    sudo wget -N -P /opt/bin
    (If you need different version, please change 1.11.3 to version you want)
    chmod +x /opt/bin/kubectl
    # for kubernetes master
    kubectl version | grep "Server Version"
    # for kubernetes nodes
    kubectl get nodes

    And nodes

  2. To upgrade Kubernetes master,

    • Log into kubernetes master
    • Execute the following commands (suppose we are upgrading Kubernetes from 1.8.4 to 1.8.6)

      Upgrade kubernetes master
      sudo systemctl stop kube-apiserver
      sudo sed -i "s/1.8.4/1.8.6/g" /opt/bin/install-kube-addons
      sudo /opt/bin/install-kube-addons
      sudo wget -N -P /opt/bin$version/bin/linux/amd64/kube-apiserver
      sudo wget -N -P /opt/bin$version/bin/linux/amd64/kube-controller-manager
      sudo wget -N -P /opt/bin$version/bin/linux/amd64/kube-scheduler
      sudo systemctl restart kube-apiserver
    • Check if kubernetes master has been upgraded successfully

  3. To upgrade Kubernetes nodes,
    • Log into kubernetes nodes

    • Execute the following commands(suppose we are upgrading Kubernetes from 1.8.4 to 1.8.6)

      Upgrade kubernetes nodes
      sudo systemctl stop kube-proxy
      sudo systemctl stop kube-kubelet
      sudo wget -N -P /opt/bin$version/bin/linux/amd64/kube-proxy
      sudo wget -N -P /opt/bin$version/bin/linux/amd64/kubelet
      sudo systemctl restart kube-proxy
      sudo systemctl restart kube-kubelet 
    • Check if kubernetes nodes have been upgraded successfully

Kubenetes might not work after upgrade. Sometimes we have to change the settings of kube-apiserver on master and kubelet on nodes.

This guide is only applicable for Kubertnetes 1.8.X and later.

How to upgrade CoreOS version

We use CoreOS template in stable release channel in cluster container service. Normally the CoreOS system will be updated automatically. You have to reboot the VMs manually so the new OS will be applied, because the reboot-strategy is set to off in our settings.

You are also be able to upgrade CoreOS manually at any time.

  1. Execute command “update_engine_client -check_for_update” to check the new CoreOS versions.
  2. The log can be found by command “journalctl -f -u update-engine”

    If there are newer version available, you will see this message at the end.

  3. Execute command “update_engine_client -update” to check the state.

  4. Reboot the OS.

There will be some seconds downtime with services in the container.

How to use High Availability(HA) feature in containers

Currently CloudStack containers support High Availability feature where three master nodes will be created. If one of the master node goes down, the other nodes will take over the responsibility of the master node to ensure that services are not interrupted for the end user.

To use this feature, select the "HA Enabled" option while creating the container cluster. The steps are mentioned in the section "Creating the container cluster". If the option is selected, three master nodes will be created.

We need to make sure that all the masters will have the same configuration in /srv/kubernetes

How to use external load balancer feature in container clusters

CloudStack containers supports external load balancers instead of port forwarding. To use this feature, make sure that the "api-key" and the "secret key" for the users are not empty. If this is empty, create it by performing the following steps:

  1. Click on "Accounts" tab in left side of the CloudStack dashboard. 
  2. Select the account to which the user belongs to. 
  3. Click "View users" on the top right corner, and select the particular user who will be creating the containers.
  4. Click on the icon which says "Generate keys" to generate the keys which will be used while creating the container clusters.

You can also refer to this link to setup the api key access.

What are the supported Kubernetes versions and CoreOS template version?

Currently we are supporting the following Kubernetes versions:

  • Kubernetes v1.9.4
  • Kubernetes v1.10.6

  • Kubernetes v1.11.3

Kubernetes is running on the following CoreOS version

  • CoreOS stable 1800.5.0 (Kubernetes: 1.10.6 Pre-Installed)