The CloudStack Container Service (CCS) orchestrates provisioning of Kubernetes managed container clusters integrated in your CloudStack environment. CCS builds container clusters Kubernetes running on CoreOS (Stable Channel) VMs. Once provisioned, you can configure the cluster and deploy containers using standard Kubernetes tools such as kubectl. Currently, CCS provides the following features for managing container clusters:

  • Embeds the Kubernetes 1.8.4 Dashboard in the CloudStack Console to deploy containerized applications
  • Injects CloudStack managed SSH keys into Kubernetes clusters
  • Monitors Kubernetes cluster health
  • Connects Kubernetes clusters to CloudStack isolated networks with integrated DNS (using kubedns)
  • Support for CoreOS guests

CloudStack Container Service is currently in beta.

Contents

Prerequisites for creating CloudStack containers

Each container cluster has a kubernetes master and some kubernetes nodes. To create a container cluster, you have to:

  • create/register an SSH key pair (which is injected into kubernetes master and nodes and used for logging into them) 
  • create a service offering (which is used in creating kubernetes master and nodes) 
  • optionally, prepare a CoreOS template (which kubernetes master and nodes are created from) 
  • optionally, create an isolated network (where kubernetes master and nodes are created on)

Step 1: Create/Register SSH Key Pair

To create/register an SSH key pair, please perform the following steps:

  1. On the left panel, select Accounts.
  2. Choose "SSH Key Pairs" in "Select view" dropdown.
  3. Click button  “+ Create a SSH Key Pair” on the right, you will see a dialog:


    If public key is set, CloudStack will register the public key. You can use it through your private key. If public key is not set, CloudStack will create a new SSH Key pair. In this case, please copy and save the private key. CloudStack will not keep it.

     

    For detailed instructions on generating keys for a user, please visit "Registering Keys for Users".

Step 2: Create a service offering

Container service requires a service offering with at least 1 CPU and 1024MB RAM.

For detailed instructions on creating a service offering, please visit "Service Offerings ".

 

Step 3: Prepare a CoreOS template (optional)

CoreOS Container Linux is the leading container operating system designed to be managed and run at massive scale, with minimal operational overhead.Containers are key to the modern data center. For developers, it has never been easier to ship new application versions. Containers easily plug into your CI/CD pipeline for automated build, test, and deployment environment with an audit trail.The container engines Docker and rkt are configured out of the box, ready to run your applications. Through the continuous stream of updates, Docker and rkt are automatically and continuously updated with the operating system.

We’ve prepared a CoreOS template used for container cluster, which is downloaded from CoreOS. You can also create a CoreOS template and upload it to CloudStack so it can be used to create a container cluster.

If CoreOS template is not specified when creating a container cluster, a default CoreOS template with Kubernetes 1.8.4 build-in will be used.

Step 4: Create an isolated network (optional)

The isolated network should

  • Support services: SourceNat, UserData, Firewall, PortForwarding, Dhcp
  • Be implemented
  • Port 443 not in use (by any firewall rule, port forwarding rule and load balancing rule).
  • Egress traffic are allowed. If default egress policy is Deny, then add egress rules. If default egress policy is Allow, do not add egress rules. Kubernetes master and nodes will download necessary packages and configurations from official kubernetes repo.

For detailed instructions on creating an isolated network, please visit "CloudStack Network ".

If network is not specified when create a container cluster, an isolated network named “<container name>-network” will be created with default network offerings for containers.

Creating a container cluster

To create a container cluster, please perform the following steps:

Before creating a container, you must perform the steps mentinoed under "Prerequisites for creating CloudStack containers".

  1. On the left panel, select “Container Service”.
  2. Click the “+ Add container cluster” button.
    The "Add container cluster" dialog box displays.
  3. Enter the following information and click "OK".

    Field NameRequired/OptionalEditable afterwardsDescription
    Name

    Required

    Yes

    The name for the container cluster.

    Description

    Optional

    No

    The description for the container cluster.

    ZoneRequiredNoThe zone where the container cluster is.

    Service Offering

    Required

    No

    The Compute Offering to use for kubernetes master and nodes in the container cluster.

    Network

    Optional

    No

    The network to use for the container cluster. If empty, then {container name}-network will be created.

    Template

    Optional

    No

    The template to use for kubernetes master and nodes in the container cluster. If empty, it will use default CoreOS template.

    Root Disk size

    Required

    Yes

    The size of root disk for kubernetes master and nodes in the container cluster.

    Cluster Size

    Required

    No

    The number of kubernetes nodes in the container cluster (excludes the kubernetes master)

    SSH keypair

    Required

    No

    The SSH key pair to use to log into the kubernetes master and nodes in the container cluster.

    Private registry

    Optional

    No

    Whether or not to use a private container registry. By default, the Docker public registry will be used.

    Checked the “Private Registry” option to use an external or private container registry.

    UsernameThis is your Docker username
    PasswordThis is your Docker password
    URLThis is your Private Docker Registry FQDN
    EmailThis is your Docker email
  4. When a container cluster is created successfully, you can see the list of created clusters:

Viewing a container cluster

You are able to get more information of a container cluster by clicking it in the list.

It displays the cluster details, dashboard, instances, and firewall.

Viewing details of container cluster

Click the "Details" tab to view the details of a container cluster.

Here are the explanation of each field (except the fields in creating a container cluster):

Field NameDescription

ID

The name for the container cluster.

Zone Name

The zone in which the cluster is deployed.

# of CPU Cores:

Total CPU cores used in the container cluster.

Memory (in MB):

Total memory cores used in the container cluster.

State

The current state of the container cluster.

API Endpoint

The API endpoint for the container cluster. This endpoint is used to connect kubectl to the container cluster.

Dashboard Endpoint

The URL for the Kubernetes Dashboard for the container cluster

Username

The username used to authenticate to the container cluster. This username is used to authenticate when connecting to the cluster with kubectl.

Password

The password used to authenticate to the container cluster. This password is used to authenticate when connecting to the cluster with kubectl.

Viewing Kunernetes Dashboard

Click the "Dashboard" tab to view the kubernetes dashboard.

If the page is empty, then click “Pop-out” on the right-up, a new page will pop up.

Viewing Instances

Click the "Instances" tab to view the list of all instances in that container cluster.

Viewing the Firewall

Click the "Firewall" tab to view the the network configurations of the container cluster .

You are also be able to add firewall rules, load balacing rules or port forwarding rules to access your kubenetes master or nodes and your services

For detailed instructions on configuring a network, please visit "Managing Networks ".

For detailed instructions on accessing kubernetes master or nodes, please visit "Accessing the Kubernetes master and nodes ".

For detailed instructions on accessing services, please visit "Accessing the Kubernetes service ".

Managing a CloudStack container cluster

Once you create a container cluster, you can operate it. The state of container cluster will change accordingly.

For example, for a Running container cluster, you can stop, destroy, edit, and resize (see below, buttons from left to right) it.

 

For a Stopped container cluster, you can start, destroy, edit, resize, and change service offering it.

Different states of a container cluster

Here are a list of all the states of a container cluster.

StateDescription

Created

Initial state of a container cluster when has been defined but no resources consumed

Running

Necessary resources are provisioned and container cluster is in operational ready state to launch containers

Error

State of the failed to create container clusters

Starting

Resources needed for container cluster are being provisioned and the container cluster is being configured and started

Stopping

Resources for the container cluster are being destroyed

Stopped

All resources for the container cluster are destroyed, Container cluster may still have resource like persistent volumes provisioned

Scaling

Transient state in which resources are either getting scaled up/down

Alert

State to represent container clusters which are not in expected desired state (operationally in-active control place, stopped master/nodes etc)

Recovering

State in which container cluster is recovering from alert state

Destroying

State in which resources for the container cluster is getting cleaned up or yet to be cleaned up by garbage collector

Destroyed

End state of container cluster in which all resources are destroyed, cluster will not be useable further

 

Stopping a container cluster

Dependent on the current state of the container cluster, this action stops it.

All resources for the container cluster will be destroyed.

 

Starting a container cluster

Dependent on the current state of the container cluster, this action starts it.

Resources needed for container cluster will be provisioned and the container cluster will be configured and started.

 

Destroying a container cluster

All resources for the container cluster will be destroyed and cleaned up.

 

Scaling In/Scaling Out a container cluster

Click the "Resize container cluster" button to resize a container cluster that is in Running/Stopped/Alert state.

There are two options: scale in and scale out. The state will be changed to Scaling and back to Running if it is in Running state.

  • Scale out
    Allocate new kubernetes nodes with configurations.
    Select Action “Scale Out”, enter new “Cluster size”, and click “OK”.

    The Cluster size is the number of kubernetes nodes (except kubernetes master)

    Once done, kubernetes nodes are created and added to kubernetes cluster automatically.

  • Scale in
    Destroy existing kubernetes nodes. 
    Select Action “Scale In”, enter “Cluster size”, and click “OK”. 

    The Cluster size is the number of kubernetes nodes (except kubernetes master)

    Once done, the kubernetes nodes with bigger number (the ones that were created later than others) will be destroyed. If some kubernetes nodes are expunged out of band (for example, in Cloudstack dashboard), then CCS will remove these kubernetes nodes from container cluster at first.

 

Editing Cluster Name and Root disk size

  1. For the selected cluster, in the "Details" tab, click the "Edit" button.


  2. In the "Name" field, enter a new name for the cluster.


  3. In the "Root disk size (GB)" field, enter the new size for the disk.

    The disk of all VMs (including Kubernetes master and nodes) will be resized, even if they are Running or Stopped.

    As we are using KVM, it is only possible to enlarge the size. Due to limitation of CoreOS, the operation system in running VMs can NOT recognize the new disk size.  CoreOS system will recognize the new disk size and enlarge the file system, after a reboot.

 

Changing service offering of a container cluster

The container cluster should be in Stopped state.

Click the "Compute Offering" icon.

Select a “Compute offering” from the list and click “OK”.

The new compute offering will be applied on all kubernetes master and nodes. 

Configuring network of a container cluster

Once you select a container cluster, from the "Firewall" tab, you can view and configure the network.

Alternatively, you can also access the page from Networks page

 

Accessing a CloudStack container cluster

Accessing the Kubernetes dashboard

To access the kubernetes dashboard of container cluster, you have to download the certificate and import certificate in your browser. The certificate is self-signed and used for internal communication between kubernetes master and nodes, and also http server for kubernetes dashboard.

  1. Download certificate
    In the "Container Service" screen, click the “Download CA Certificate” button, and click “OK”.
    the certificate will be stored in your system.

     

  2. Import certificate in your browser
    You need to import the certificate in your browser (eg IE, Chrome, Firefox).
    Once it is done, a certificate named “cloudstack” will be added in your certificates list. If you are not able to add it, please remove the certificate with “cloudstack” and retry.

  3. Kubernetes dashboard
    You are able to access the kubernetes dashboard if the certificate is imported. By clicking the “Dashboard” tab in the details page, you can view the dashboard.

    If the screen is empty, then click “Pop-out” on the top-right of the screen, a new page will pop up.

Accessing the Kubernetes API server

To access the kubernetes API server, you need to download kubectl and execute the following command.

kubectl <COMMAND> -s <endpoint> --username=<username> --password=<password> -insecure-skip-tls-verify=true

The endpoint, username and password can be found in the details of container cluster. For example,

 

Accessing the Kubernetes master and nodes

To access the kubernetes master and nodes, you need to add port forwarding rules and firewall rules in the network configurations in CloudStack.

  1. Add port forwarding rules

    Please note, port 443 is used for kubernetes-dashbaord

  2. Add firewall rules (port 443 is used for kubernetes-dashbaord)


  3. You can now to log into kubernetes master and nodes, using the SSH key you specified when creating the container cluster.

    The username is “core” not “root”. To modify the files or execute commands in the system, you have to use “sudo”.

Accessing the Kubernetes service

You can access your services by creating load balancing rules or port forwarding rules, along with firewall rules.

For example, if you create an ngnix service in the container cluster.

  1. Create nginx service in kubernetes.


  2. The port of nginx is 80. The NodePort is randomly picked up from range 30000-32767.
    In this example, the NodePort is 32446. 


  3. Create a load balancing rule (public port is 8080, private port is 32446/NodePort) to all kubernetes nodes.


  4. Open the public port 8080 on firewall.


  5. You can access the nginx service by the public IP and public port.


  6. You can also specify the NodePort (in range 30000-32767) by editing kubernetes service on kubernetes dashboard.

 

FAQs about Cloudstack Container Service

How to use your domain (name, ssl, and key) for Kubernetes dashboard

By default, the kubernetes dashboard URL is using the Public IP of the container cluster.

For example, https://<Public IP address>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/

It is also possible to use your own certificate, key and domain.

  1. Log into kubernetes master
  2. Put your server certificate into /srv/kubernetes/leasewebcloud.crt (/opt/bin/leasewebcloud.crt in Alpha version, you can use any other filename). If there are certificate chains, put intermediate certificate right after server certificate.
  3. Put your key into /srv/kubernetes/leasewebcloud.key.
  4. Add a new line in /etc/systemd/system/kube-apiserver.service (Please make corresponding change if you use other filenames).
    “--tls-sni-cert-key=/srv/kubernetes/leasewebcloud.crt,/srv/kubernetes/leasewebcloud.key \”
  5. Restart kube-apiserver by “sudo systemctl restart kube-apiserver”

If the domain name can be resolved, you will be able to access the kubernetes dashboard by URL https://<DomainName>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/

For more information, click here.

How to upgrade Kubernetes version

Due to the rapid development of the upstream open source project, the Kubernetes version LeaseWeb delivers will lag behind the most recent community release [8]. The supplied versions in the alpha and beta environments and the different regions will most likely differ as well.

  1. Get current version
    You can determine the Kubernetes version on master and nodes by issuing the following command

    Get kubernetes version


    and nodes



    Alternatively, you can log into kubernetes master, download kubectl from URL to /opt/bin/kubectl if it does not exist in kubernetes master and execute the commands without endpoint,username and password.

    Get kubernetes version via kubectl


    And nodes

     

  2. To upgrade Kubernetes master,

    • Log into kubernetes master
    • Execute the following commands (suppose we are upgrading Kubernetes from 1.8.4 to 1.8.6)

      Upgrade kubernetes master
    • Check if kubernetes master has been upgraded successfully

     

  3. To upgrade Kubernetes nodes,
    • Log into kubernetes nodes

    • Execute the following commands(suppose we are upgrading Kubernetes from 1.8.4 to 1.8.6)

      Upgrade kubernetes nodes
    • Check if kubernetes nodes have been upgraded successfully



Kubenetes might not work after upgrade. Sometimes we have to change the settings of kube-apiserver on master and kubelet on nodes.

This guide is only applicable for Kubertnetes 1.8.X and later.

 

How to upgrade CoreOS version

We use CoreOS template in stable release channel in cluster container service. Normally the CoreOS system will be updated automatically. You have to reboot the VMs manually so the new OS will be applied, because the reboot-strategy is set to off in our settings.

You are also be able to upgrade CoreOS manually at any time.

  1. Execute command “update_engine_client -check_for_update” to check the new CoreOS versions.
  2. The log can be found by command “journalctl -f -u update-engine”


    If there are newer version available, you will see this message at the end.

  3. Execute command “update_engine_client -update” to check the state.

  4. Reboot the OS.

 

There will be some seconds downtime with services in the container.