Note: If you have missed my previous articles on Docker and Kubernetes, you can find them here.
Application deployment models evolution.
Getting started with Docker.
Docker file and images.
Publishing images to Docker Hub and re-using them.
Docker- Find out what's going on.
Docker Networking- Part 1.
Docker Networking- Part 2.
Docker Swarm-Multi-Host container Cluster.
Docker Networking- Part 3 (Overlay Driver).
Introduction to Kubernetes.
Kubernetes- Diving in (Part 1)-Installing Kubernetes multi-node cluster.
Kubernetes-Diving in (Part2)- Services.
Kubernetes- Infrastructure As Code with Yaml (part 1).
Kubernetes- Infrastructure As Code Part 2- Creating PODs with YAML.
Kubernetes Infrastructure-as-Code part 3- Replicasets with YAML.
Kubernetes Infrastructure-as-Code part 4 - Deployments and Services with YAML.
Deploying a microservices APP with Kubernetes.
Kubernetes- Time based scaling of deployments with python client.
Kubernetes Networking - The Flannel network explained.
Kubernetes- Installing and using kubectl top for monitoring nodes and PoDs
Kubernetes Administration- Scheduling
Kubernetes Administration- StorageAdministration is an important part of maintaining any infrastructure be it physical, virtual, or container-based. Major administration tasks involve managing resources like compute (CPU, Memory), storage, users, etc. In Kubernetes Administration- Scheduling I gave a brief overview of requesting CPU and Memory resources for PoDs. Kubernetes Administration- Storage gives an overview of storage administration. In this article, I will attempt to give an overview of user administration in Kubernetes.
User administration generally involves the following:
Who can Access resources? i.e Authentication
What can they do ? i.e Authorization
Role-Based Access control is a method of authorization where a user's role in the organization determines what he can do with resources. For example, users with role admins might have complete control of the resources, whereas developer would be able to view resources and cannot create or delete resources. Kubernetes offers the ability to administer roles and attach users to roles both at namespace and cluster level. I will be describing these in a bit, but let's look at various components in Kubernetes where access control is required:
Physical Security:
This involves ensuring only authorized users have access to physical infrastructure that hosts the Kubernetes cluster. This is a separate topic by itself.
Cluster/Node Security:
User management/security of nodes that hosts the cluster. From Kubernetes standpoint, this would essentially mean users with the ability to login to node (master node) and run Kubectl, kubeadm, etcdctl utilities.
Application Security:
Applications (PoDs or deployments) may implement their own security mechanisms for authenticating users. Single Sign On/OWAuth have become popular and do not have anything to do with Kubernetes.
Service Accounts:
Service account enables a PoD to authenticate with API server with an authentication token. A "default" service account is created when cluster is installed and unless overridden this account is used during the creation of objects like PoDs.
root@sathish-vm2:/home/sathish# kubectl get serviceaccount -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2020-10-15T05:06:26Z"
name: default
namespace: default
resourceVersion: "391"
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: ae3bea7c-1294-4c65-a54d-17b95bc55bc4
secrets:
- name: default-token-spkmz
kind: List
metadata:
resourceVersion: ""
selfLink: ""
root@sathish-vm2:/home/sathish# kubectl get secret default-token-spkmz
NAME TYPE DATA AGE
default-token-spkmz kubernetes.io/service-account-token 3 73d
root@sathish-vm2:/home/sathish# kubectl run mybusybox --image=busybox --command sleep 1000
pod/mybusybox created
#default-token-spkmz is mounted in the created PoD. This is used to authenticate with API server
root@sathish-vm2:/home/sathish# kubectl describe pods mybusybox
Name: mybusybox
Namespace: default
Priority: 0
Node: sathish-vm1/172.28.147.44
Start Time: Sun, 27 Dec 2020 07:11:46 +0000
Labels: run=mybusybox
Annotations: <none>
Status: Running
IP: 10.244.1.5
IPs:
IP: 10.244.1.5
Containers:
mybusybox:
Container ID: docker://3a5a93115f83d83bb8351120878d72fcce13b2fde7e0ec6bef3a6dfb29f56362
Image: busybox
Image ID: docker-pullable://busybox@sha256:bde48e1751173b709090c2539fdf12d6ba64e88ec7a4301591227ce925f3c678
Port: <none>
Host Port: <none>
Command:
sleep
1000
State: Running
Started: Sun, 27 Dec 2020 07:11:53 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-spkmz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-spkmz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-spkmz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m5s Successfully assigned default/mybusybox to sathish-vm1
Normal Pulling 9m4s kubelet, sathish-vm1 Pulling image "busybox"
Normal Pulled 8m58s kubelet, sathish-vm1 Successfully pulled image "busybox" in 5.59136436s
Normal Created 8m58s kubelet, sathish-vm1 Created container mybusybox
Normal Started 8m58s kubelet, sathish-vm1 Started container mybusybox
Let's create a new service account and try using it to create a PoD
#Create a new service account
root@sathish-vm2:/home/sathish# kubectl create sa sathish
serviceaccount/sathish created
#Create a new clusterrole
root@sathish-vm2:/home/sathish# kubectl create clusterrole pod-admin --verb=get,list,watch,create,delete --resource=podsclusterrole.rbac.authorization.k8s.io/pod-admin created
#Create a rolebinding
root@sathish-vm2:/home/sathish# kubectl create clusterrolebinding pod-admin-binding --clusterrole=pod-admin --serviceaccount=sathish
error: serviceaccount must be <namespace>:<name>
root@sathish-vm2:/home/sathish# kubectl create clusterrolebinding pod-admin-binding --clusterrole=pod-admin --serviceaccount=default:sathish
clusterrolebinding.rbac.authorization.k8s.io/pod-admin-binding created
#Create and view PoD with service account
root@sathish-vm2:/home/sathish# kubectl run customsabusybox --image=busybox --command sleep 1000 --serviceaccount=sathish
pod/customsabusybox created
#This PoD uses the sathish SA
root@sathish-vm2:/home/sathish# kubectl describe pod customsabusybox
Name: customsabusybox
Namespace: default
Priority: 0
Node: sathish-vm1/172.28.147.44
Start Time: Sun, 27 Dec 2020 07:36:04 +0000
Labels: run=customsabusybox
Annotations: <none>
Status: Running
IP: 10.244.1.6
IPs:
IP: 10.244.1.6
Containers:
customsabusybox:
Container ID: docker://a7cc5b57cc82d97c7c26da6354ef1178f1b1f2fe7ffe16f7c68cc64ba51c10a9
Image: busybox
Image ID: docker-pullable://busybox@sha256:bde48e1751173b709090c2539fdf12d6ba64e88ec7a4301591227ce925f3c678
Port: <none>
Host Port: <none>
Command:
sleep
1000
State: Running
Started: Sun, 27 Dec 2020 07:36:10 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from sathish-token-2jsvm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
sathish-token-2jsvm:
Type: Secret (a volume populated by a Secret)
SecretName: sathish-token-2jsvm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 68s Successfully assigned default/customsabusybox to sathish-vm1
Normal Pulling 68s kubelet, sathish-vm1 Pulling image "busybox"
Normal Pulled 63s kubelet, sathish-vm1 Successfully pulled image "busybox" in 5.015162728s
Normal Created 62s kubelet, sathish-vm1 Created container customsabusybox
Normal Started 62s kubelet, sathish-vm1 Started container customsabusyboxAs we can see this PoD uses the custom service account
Kubernetes role and rolebindings
Kubernetes implements role-based access control to manage users. A role determines level of access a particular user has. For example, it is possible to create roles to only view PoDs. RoleBindings "bind" user(s) to a role. Further, roles and RoleBindings are categorized as:
Namespace wide
Cluster Wide
Let's look at the steps involved in creating roles and rolebinding in default namespace
Note: ClusterRoles and ClusterRoleBindings can be created by following similar steps. Use ClusterRole and ClusterRolebinding keywords when creating roles and role bindings for cluster.Create a Linux user .
root@sathish-vm2:/home/sathish# useradd podadmin
root@sathish-vm2:/home/sathish# passwd podadmin
#Grant SUDO permission to user.
root@sathish-vm2:/home/sathish# usermod -aG sudo podadmin2. Generate csr and private keys. Change to podadmin user before running the below commands.
# Create Private Key
$ openssl genrsa -out podadmin.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
.....................+++++
...........................................+++++
e is 65537 (0x010001)
# Create CSR and sign with the key
$ openssl req -new -key podadmin.key -out podadmin.csr -subj "/CN=podadmin
# Generate selfsigned cert with CSR
sudo openssl x509 -req -in podadmin.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out podadmin.crt -days 365
3. Create a certificate signing request. CSR needs certificate request to be base64 encoded
Getting base64 encoded format of certificate
$ cat podadmin.csr | base64 | tr -d "\n"
<copy the complete content>
# Create a CSR
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: podadmin
spec:
groups:
- system:authenticated
request: LS0<pasted base64value>
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth4. At this point change back to the user that has full cluster access and create/approve csr
root@sathish-vm2:/home/k8admin# whoami
root
root@sathish-vm2:/home/sathish# kubectl apply -f /home/podadmin/csr.yaml
certificatesigningrequest.certificates.k8s.io/defaultpodadmin created
root@sathish-vm2:/home/sathish# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
podadmin 58s kubernetes.io/kube-apiserver-client kubernetes-admin Pending
# Appprove the CSR
root@sathish-vm2:/home/sathish# kubectl certificate approve podadmin
certificatesigningrequest.certificates.k8s.io/podadmin approved
root@sathish-vm2:/home/sathish# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
podadmin 72s kubernetes.io/kube-apiserver-client kubernetes-admin Approved,IssuedNow, podadmin can authenticate/talk to kube-api server. However, the user does not have the required Kubernetes permissions. This can be verified with kubectl auth
root@sathish-vm2:/home/k8admin# kubectl auth can-i create pods --as=podadmin
no5. Grant permission to podadmin to create PoDs by creating roles and rolebindings. We have the option of creating namespace wide or cluster-wide roles. For this example, we will choose namespace wide rolebinding.
root@sathish-vm2:/home/k8admin# kubectl create role podadmin --verb=create,delete,get,list --resource=pod
role.rbac.authorization.k8s.io/podadmin created
root@sathish-vm2:/home/k8admin# kubectl describe role podadmin
Name: podadmin
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
pods [] [] [create delete get list]
root@sathish-vm2:/home/sathish# kubectl create rolebinding podadmin --role=podadmin --user=podadmin
rolebinding.rbac.authorization.k8s.io/podadmin created
root@sathish-vm2:/home/sathish# kubectl describe rolebinding podadminbinding
Name: podadmin
Labels: <none>
Annotations: <none>
Role:
Kind: Role
Name: podadmin
Subjects:
Kind Name Namespace
---- ---- ---------
User podadmin6. Verify the role. podadmin can create pods in default namespace but not in other namespaces like kube-system
root@sathish-vm2:/home/defaultpodadmin# kubectl auth can-i create pods --as=podadmin
yes
root@sathish-vm2:/home/defaultpodadmin# kubectl auth can-i create pods --as=podadmin -n kube-system
no
#Checkdefaultpodadmin should not be able to create other objects
root@sathish-vm2:/home/defaultpodadmin# kubectl auth can-i create pv --as=podadmin
Warning: resource 'persistentvolumes' is not namespace scoped
no
root@sathish-vm2:/home/defaultpodadmin# kubectl auth can-i create deployments --as=podadmin
no
root@sathish-vm2:/home/defaultpodadmin# kubectl auth can-i create replicasets --as=podadmin
noNow, create the Kubeconfig file for the user- we can copy the existing kubeconfig file and make necessary changes. Before following the steps below, change to podadmin user
$ sudo cp /root/.kube/config /home/podadmin/.kube
[sudo] password for podadmin:
$ sudo chown -R podadmin /home/podadmin
# Add Credentials
$ kubectl config set-credentials podadmin --client-key=/home/podadmin/podadmin.key --client-certificate=/home/podadmin/podadmin.crt --embed-certs=true
User "podadmin" set.
# Add context
$ kubectl config set-context podadmin --cluster=kubernetes --user=podadmin
Context "podadmin" created.
# Change context to pod admin
$ kubectl config use-context podadmin
Switched to context "podadmin"
# Check Permissions
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mywebserver-6d6c47b597-jqtfc 1/1 Running 0 9d
$ kubectl get deployments
Error from server (Forbidden): deployments.apps is forbidden: User "podadmin" cannot list resource "deployments" in API group "apps" in the namespace "default"
$ kubectl run nginx1 --image=nginx
pod/nginx1 createdThe above user can administer pods but cannot create/delete other objects like deployments. This is RBAC in Kubernetes.
That's it for today folks !! Hope, this short intro to user administration in Kubernetes was useful. Until next time, have a great week ahead, and Happy New Year!
Comments