Tutorial: Spin up Your Kubernetes-in-Docker Cluster and They Will Come

Originally published at: https://www.conjur.org/blog/tutorial-spin-up-your-kubernetes-in-docker-cluster-and-they-will-come/

Online demos and tutorials are a great way to introduce Kubernetes-native applications to potential users and collaborators. Often, however, these online demos and tutorials use a “Bring Your Own Cluster & Clients” approach. In other words, they require that the participant have a pre-configured Kubernetes cluster available and specific versions of clients such as kubectl and helm. This blog will show you how to create a containerized demo for…

2 Likes

Hi @dane ,

Thanks for the very useful post. There’s one thing that confused me. In the post, you say:

When kind spins up a cluster, it also creates (or modifies) a kubectl config file that allows kubectl access to the kind cluster. Normally, kind sets up port forwarding between the Kubernetes master container and the host and uses a Kubernetes access IP of 127.0.0.1 (localhost) in the kubectl config. When kind is run from inside a demo container, the host’s localhost address of 127.0.0.1 no longer applies.

I don’t understand why it doesn’t apply. Wouldn’t it just forward the port to the demo container’s localhost in this case? The demo container would try to send requests to the forwarded port on its 127.0.0.1 which would end up on the kind cluster. Am I missing something?

1 Like

Hi @emre-aydin,

Thank you for reading my blog, and thank you for the challenging question!!!

I probably could have explained the kubectl config fixup more clearly, but it’s a bit difficult to explain clearly in one paragraph as part of a blog. Let me try to explain it while digging deeper into the details.

The first thing we should cover, and probably the key to understanding everything else, is the concept of “Docker-on-Docker” operation. (This is different than “Docker-in-Docker” or “DinD”: see Using Docker-in-Docker for your CI or testing environment? Think twice. for an explanation of DinD).

For this “Kubernetes-in-Docker” demo, we want to spin up a client container that:

  • Conveniently contains all of the client binaries we need: kubectl, docker(client), helm, kind, etc.
  • Allows us to run KinD to spin up other containers that serve as Kubernetes nodes.

Now in order for KinD to spin up other containers from inside the client container, we need a docker server. But rather than add a docker server/agent running inside the client container (i.e. the Docker-in-Docker approach, which is very difficult to pull off), we simply use the host’s Docker server by doing a bind-mount of the host’s Docker socket from the client container (using the arg -v /var/run/docker.sock:/var/run/docker.sock). This is what I refer to as “Docker-on-Docker” (container’s Docker client on host’s Docker server).

We can see what this looks like from inside the client container:

root@secretless-k8s-demo:~/.kube$ # From inside the container
root@secretless-k8s-demo:~/.kube$ docker version --format {{.Server.Version}}
20.10.7
root@secretless-k8s-demo:~/.kube$ docker version --format {{.Client.Version}}
18.06.1-ce
root@secretless-k8s-demo:~/.kube$ 

So the client container’s Docker client version is different than the Docker server version that “it sees”.

And the Docker server version that the container “sees” is the same as the host’s server version:

dane@dane:~/.kube$ # On the host
dane@dane:~/.kube$ docker version --format {{.Server.Version}}
20.10.7
dane@dane:~/.kube$ docker version --format {{.Client.Version}}
20.10.7
dane@dane:~/.kube$ 

Also, if we run docker ps on both the container and the host, we see the same containers running since they’re both sharing the same Docker server:

root@secretless-k8s-demo:~/.kube$ # In the container
root@secretless-k8s-demo:~/.kube$ docker ps --format {{.Names}}
secretless-kube-worker2
secretless-kube-control-plane
secretless-kube-worker
secretless-demo-client
root@secretless-k8s-demo:~/.kube$ 

and:

dane@dane:~/.kube$ # On the host
dane@dane:~/.kube$ docker ps --format {{.Names}}
secretless-kube-worker2
secretless-kube-control-plane
secretless-kube-worker
secretless-demo-client
dane@dane:~/.kube$ 

Cool!

So the takeaway so far:
Docker commands in the container are served by the Docker server on the host.

Now, when we run kind (the client for KinD) from inside the container, it will make Docker requests, but it has no awareness that the Docker server it’s using is the host’s Docker server. It’s sort of being fooled, or “faked out”

So now when KinD requests that a container be created to be used as a Kubernetes master node, it asks the Docker server to use port forwarding to expose that new container’s port 6443 (the Kubernetes API server port) on a random port on the host (important: here the term “host” is from Docker server’s perspective, not the KinD client’s perspective).

Let’s see what this looks like using Docker commands. From inside the container:

root@secretless-k8s-demo:~/.kube$ # From inside the container
root@secretless-k8s-demo:~/.kube$ docker container port secretless-kube-control-plane
6443/tcp -> 127.0.0.1:49153
root@secretless-k8s-demo:~/.kube$

But to whom does this 127.0.0.1:49153 address belong? This address actually applies to the host, not to the container!!!

To see proof of this, let’s try curling this address both from inside the container and from the host. From inside the container:

root@secretless-k8s-demo:~/.kube$ # From inside the container
root@secretless-k8s-demo:~/.kube$ curl -k https://127.0.0.1:49153
curl: (7) Failed to connect to 127.0.0.1 port 49153: Connection refused
root@secretless-k8s-demo:~/.kube$ 

So no response.

Whereas on the host, we do get a response (although we’re missing an API token, so we get an unauthorized response):

dane@dane:~/.kube$ # On the host
dane@dane:~/.kube$ curl -k https://127.0.0.1:49153
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}dane@dane:~/.kube$ 

After KinD creates the container to serve as the Kubernetes master node, it creates a Kubernetes config file (at ~/.kube/config) that points to the address/port at which it THINKS it can reach the Kubernetes API server (i.e. 127.0.0.1:49153), but as shown above, that address/port is not reachable from inside the container.

So at what address/port CAN the Kubernetes API server be reached from inside the container? It turns out that the Kubernetes API server can be reached at the address/port of:

     <Master-Node-Containers-IP-Address>:6443

To see this, let’s fetch that IP address, and try to curl to that address/port:

root@secretless-k8s-demo:~/.kube$ # From inside container
root@secretless-k8s-demo:~/.kube$ docker inspect secretless-kube-control-plane | grep '"IPAddress"'
            "IPAddress": "172.17.0.4",
                    "IPAddress": "172.17.0.4",
root@secretless-k8s-demo:~/.kube$ curl -k https://172.17.0.4:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}root@secretless-k8s-demo:~/.kube$ 

So THIS address, 172.17.0.4:6443, (and NOT 127.0.0.1:49153) is the address that needs to be loaded in the ~/.kube/config file inside the container, so that when kubectl commands are run from inside the container, kubectl is able to connect with the Kubernetes API server running in the Kubernetes master node container.

I hope this all makes sense! Please let me know if you have any more questions.

Cheers,
Dane

Hi @dane,

It makes perfect sense! Thanks for the great explanation. The missing part for me was that the actual port forwarding is happening via the Docker agent. For some reason I thought that it was done on the iptables level. I got it now.

Cheers,
Emre

1 Like

Right!

Along those lines, another thing I could have added in the explanation is looking at who is actually listening on that random port 49153?

The container is NOT listening on 49153:

root@secretless-k8s-demo:~/.kube$ # From inside container
root@secretless-k8s-demo:~/.kube$ netstat -atunp | grep 49153
root@secretless-k8s-demo:~/.kube$ 

But the host IS listening on 49143:

dane@dane:~/.kube$ # On the host
dane@dane:~/.kube$ sudo netstat -atunp | grep 49153
tcp        0      0 127.0.0.1:49153         0.0.0.0:*               LISTEN      1066583/docker-prox 
dane@dane:~/.kube$