How to deploy a Docker container to Kubernetes using wercker and Quay.io

Introduction

In this post, I will introduce how to setup a cloud native development environment with wercker, kubernetes, quay and Google Cloud.

The post here is evolved from http://devcenter.wercker.com/docs/quickstarts/deployment/kubernetes (the tutorial is out of date, about two years ago, and it is a defacto guideline for a experienced developer, rather a tutorial for a new beginner). That is why I wrote this post to track my learning and also hope it can help if you are also going to play with it)

I assumed you have done,

I assumed you have knowledge:

  • Docker. https://docs.docker.com/get-started/
  • Kubernetes. Install a minikube in your local machine and play with it. Should have some concepts about cluster, pods, nodes, replication controllers.
  • Knowledge about CI/CD tool, like Jenkins.

Setup

Google cloud

Go to Google cloud to register a account. They provided a 12 months trial with $300 in cap.

Logon Google cloud to create a cluster.

1

I have created acluster ‘cluster-1’ by following the ‘CREATE CLUSTER’ wizard step by step.

Click ‘Connect’ button,

2.png

Logon your laptop, and install Google Cloud SDK.

Then run commands as below (copied from above browser dialog)

3.png

The kubectl proxy command will help you logon your Google Cloud Kubernetes cluster from your local desktop!

4.png

It has no replication controllers created. In the next steps, Wercker will create it automatically.

5

You can see it also has only system’s kubernetes service.

You can also use kubectl to manage the cluster directly in command line,

6

GitHub

Fork my repo https://github.com/luohuahuang/DEPRECATED-wercker-kubernetes-quay

7.png

You need to update the Quay.io information with yours in create_cities-controller.json.sh and wercker.yml

(Do NOT fork it from wercker. Its wercker.yml is outdated)

Quay.io

Go to create a Quay.io account and create a registry

8.png

Its path will be quay.io/luhuang/demo in this case. You need to update the values to yours in create_cities-controller.json.sh and wercker.yml

Wercker

Bind your GitHub account.

9.png

Go to create the pipelines. To make it simple, I created all of them as pipelines as below,

10.png

The “build” will be triggered by Git push while the rest pipelins will be based on pipelines hook. Since I don’t define a workflow here, I will have to trigger these three pipelines manually on demand.

All of the pipelines require Quay.io parameters and K8s parameters so I created them as application environment variables. You can also add variables into specified pipelines.

11.png

KUBERNETES_MASTER, KUBERNETES_USERNAME, KUBERNETES_PASSWORD can be found from your Google Cloud console,

24.PNG

Run

OK, it is ready to run.

I am going to commit a change to GitHub. Take a note about the commit#.

12.png

build

The build pipeline will be triggered immediately automatically by wercker as I configured its hook type as ‘Git push’.

13.png

See more details,

14.png

It firstly pulls the latest code, then setup environment, run test, build, then store the artifacts.

15.png

push-to-registry

The build pipeline has built and stored the artfacts successfully. I am going to trigger the push-to-registry manually (its hook type is not Git Push) to use Docker to push the image to Quay.io.

After the pipeine completes, go to Quay.io.

16.png

deploy-kubernetes

After the new image is gone to Quay.io, go to trigger the deploy-kubernetes.

It will run kubectl command (create_cities-controller.json) setup the Replication Controllers. See from the kubnernetes console,

17.png

(you can see wercker created ‘cities’ replication controllers in my Google cloud cluster)

And then it will run kubectl command (cities-service.json) to expose the services. See from your Kubernetes console,

18.png

(you can see cities has been exposed as services. We should be able to visit it from internet via http://#external endpoints#/cities.json)

Let’s see more details from Google Cloud console. You can choose one of the three replication instances and check the log.

19.png

(Listening on the host: xxx is the log printed by the application. https://github.com/luohuahuang/DEPRECATED-wercker-kubernetes-quay/blob/master/main.go)

You can also visit the service via browser:

25

From the console,

26.PNG

You jsut need to run ‘deploy-kubernetes’ for one time only. The next time you have any code changes and want to deploy. You don’t need to run ‘deploy-kubernetes’ again as the replication controllers and all related settings already available. You should run ‘rolling-update’.

rolling-update

https://github.com/luohuahuang/DEPRECATED-wercker-kubernetes-quay/commit/db3a58bdee726e0769962023840211568086a64f (this is a new push. It will print one more line ‘Hello World’)

20.png

The ‘build’ is triggered successfully and after it completes, I triggered the ‘push-to-registry’.

Okie, now I will NOT run ‘deploy-to-kubernetes’, instead, I will just run ‘rolling-update’ to ask replication controller to update ALL of the three pods in my Google cluster to use the image#db3a58bdee726e0769962023840211568086a64f

Look at the log from rolling-update:

21.png

I recommend you read into the document about how Kubernetes uses replications controller to sync up Pods, and how Kubernetes to update/sync up applications among pods…

Now go to Kubernetes console,

22.png

See I just ran the ‘rolling-update’ and it helped me update all of the three pods. Go to all of the three pods,

23.png

‘Hello World’s is printed for the same.

Conclusion

In this wiki, you can see how we can setup a cloud native development by deploying a Docker container to Kubernetes using wercker and Quay.io on a Google Cloud environment.

I think it is kinda cool DX (developer experience) of cloud native development:

  1. dev pushes a code to GitHub and then go to have a coffee.
  2. Wercker triggers a build to pull the code and use Docker to generate image
  3. Publish the image to Quay.io
    1. If Dev wants to test it locally, for example dev has Minikube installed on his local machine. He can deploy the new image to his minikube to do testing.
  4. Deploy the image from Quay.io to Google Kubernetes Cluster

It is a cool experience. Dev pushes code by a click ‘GO’ and then all will be in cloud.

Pass HTTP_PROXY when creates Docker VMs

If your docker operations are behind a firewall. You have to setup proxies when you create Docker VMs. The following command will not work,

docker-machine create -d virtualbox myvm1

If you run docker search ubuntu, it will return time-out.

You should pass HTTP_PROXY and related setting with below format:

docker-machine create -d virtualbox –engine-env HTTP_PROXY=http://www-proxy.xxx.com:80 –engine-env HTTPS_PROXY=http://www-proxy.xxx.com:80 –engine-env NO_PROXY=localhost,127.0.0.1,xxx.com myvm1

swarm deamon port VS. swarmkit port

I have two VMs (myvm1 and myvm2) and myvm1 is set as swarm manager.
scaab101:~ uadmin$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 – virtualbox Running tcp://192.168.99.100:2376 v17.06.1-ce
myvm2 – virtualbox Running tcp://192.168.99.101:2376 v17.06.1-ce
scaab101:~ uadmin$ docker-machine ssh myvm1 “docker swarm init –advertise-addr 192.168.99.100:2376”
Swarm initialized: current node (8g5gscgtpvnc4kg4rgqqp4czi) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join –token SWMTKN-1-24nwzfj1jww1w4b4sle3udr6a99rh7hhyca5w22scmpi8q9oh5-3rzbf69pv3yhe80b1xpfaczeg 192.168.99.100:2376

To add a manager to this swarm, run ‘docker swarm join-token manager’ and follow the instructions.
scaab101:~ uadmin$ docker-machine ssh myvm2 “docker swarm join” \
> –token SWMTKN-1-24nwzfj1jww1w4b4sle3udr6a99rh7hhyca5w22scmpi8q9oh5-3rzbf69pv3yhe80b1xpfaczeg \
scaab101:~ uadmin$ docker-machine ssh myvm2 “docker swarm join \
> –token SWMTKN-1-24nwzfj1jww1w4b4sle3udr6a99rh7hhyca5w22scmpi8q9oh5-3rzbf69pv3yhe80b1xpfaczeg \
> 192.168.99.100:2376”
Error response from daemon: rpc error: code = 13 desc = connection error: desc = “transport: remote error: tls: bad certificate”
exit status 1

(The error here is because I use 2376 to try to join myvm2 to swarm as a worker. 2376 here is swarm daemon port. I should use a different one, say, 2377 to join. 2377 here is a swarmkit port)
scaab101:~ uadmin$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 – virtualbox Running tcp://192.168.99.100:2376 v17.06.1-ce
myvm2 – virtualbox Running tcp://192.168.99.101:2376 v17.06.1-ce
scaab101:~ uadmin$ docker-machine ssh myvm1 “docker swarm leave –force”
Node left the swarm.
scaab101:~ uadmin$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 – virtualbox Running tcp://192.168.99.100:2376 v17.06.1-ce
myvm2 – virtualbox Running tcp://192.168.99.101:2376 v17.06.1-ce
scaab101:~ uadmin$ docker-machine ssh myvm1 “docker swarm init –advertise-addr 192.168.99.100:2377
Swarm initialized: current node (eq8iugdzo3aez24dhpj3au6hj) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join –token SWMTKN-1-1np3bem2tuziipajx9hnh8yj10pkovx046b4itinru1uazeq57-6e5noxm3to8lvmwbanqee048r 192.168.99.100:2377

To add a manager to this swarm, run ‘docker swarm join-token manager’ and follow the instructions.

scaab101:~ uadmin$ docker-machine ssh myvm2 “docker swarm join –token SWMTKN-1-1np3bem2tuziipajx9hnh8yj10pkovx046b4itinru1uazeq57-6e5noxm3to8lvmwbanqee048r 192.168.99.100:2377”
This node joined a swarm as a worker.
scaab101:~ uadmin$

Set proxies for Docker

Firstly, set the proxies in Preference Panel.

proxy-1.JPG

Secondly, you need to add proxy info to your Dockerfile.txt like this,

bash-3.2$ cat Dockerfile
# Use an official Python runtime as a parent image
FROM python:2.7-slim

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
ADD . /app

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt –proxy http://www-xxxxx.com:80 –trusted-host pypi.python.org

# Make port 80 available to the world outside this container
EXPOSE 80

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD [“python”, “app.py”]

Thirdly, if you are going to run docker-machine command to create VM. You will have to set below so it can download related stuff.

export https_proxy=htp://www-proxy.xxx:80