Unable to execute shell on K8S pod

php Editor Banana discovered that some users were unable to execute shell commands on Kubernetes (K8S) pods. This problem may prevent users from debugging and troubleshooting, which is a big trouble for development and operation and maintenance personnel. In this article, we will introduce the possible causes and solutions to this problem to help users successfully execute shell commands and improve work efficiency.

Question content

Use rancher-2.6.6 to install k8s (v1.23.7 ​​rke2r2). Related questions are here.

I cannot execute the shell on any container in k8s (v1.23.7 ​​rke2r2).

The log is a bit long, I don’t know if it will be helpful.

I'm not familiar with go, it took me a few hours and I don't know how to solve it. Any help would be very happy.

[root@123 ~]# /var/lib/rancher/rke2/bin/kubectl --kubeconfig rke2.yaml -n minio  --v=9 exec --stdin --tty console-656df75d45-bdb4m -- /bin/bash
I1117 17:59:25.557423 1522170 loader.go:372] Config loaded from file:  rke2.yaml
I1117 17:59:25.814339 1522170 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.23.7+rke2r2 (linux/amd64) kubernetes/42c05a5" 'https://127.0.0.1:6443/api/v1/namespaces/minio/pods/console-656df75d45-bdb4m'
I1117 17:59:25.814906 1522170 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:6443 succeed
I1117 17:59:25.824624 1522170 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 3 ms ServerProcessing 5 ms Duration 10 ms
I1117 17:59:25.824713 1522170 round_trippers.go:577] Response Headers:
I1117 17:59:25.824735 1522170 round_trippers.go:580]     Date: Thu, 17 Nov 2022 09:59:25 GMT
I1117 17:59:25.824808 1522170 round_trippers.go:580]     Audit-Id: 06174be1-bd2f-4500-a59b-58bf68ff79f9
I1117 17:59:25.824832 1522170 round_trippers.go:580]     Cache-Control: no-cache, private
I1117 17:59:25.824884 1522170 round_trippers.go:580]     Content-Type: application/json
I1117 17:59:25.824909 1522170 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: fb6d6698-bf27-4014-9f5c-9d7405a8ecc9
I1117 17:59:25.824927 1522170 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 95a48fbf-d3d7-44a2-aa35-f6b4ccc22845
I1117 17:59:25.825269 1522170 request.go:1181] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"console-656df75d45-bdb4m","generateName":"console-656df75d45-","namespace":"minio","uid":"aa46973d-debc-419d-8d52-338bba5e121f","resourceVersion":"61701433","creationTimestamp":"2022-11-17T06:07:05Z","labels":{"app.kubernetes.io/instance":"minio-operator-console","app.kubernetes.io/name":"operator","pod-template-hash":"656df75d45"},"annotations":{"cni.projectcalico.org/containerID":"6b52546c61cebbc4450c21e7bedaa1198aa176ee708cfa416dd570717e89d14d","cni.projectcalico.org/podIP":"10.42.48.137/32","cni.projectcalico.org/podIPs":"10.42.48.137/32","kubernetes.io/psp":"global-unrestricted-psp"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"console-656df75d45","uid":"5954db5d-1892-41af-82c9-8aad46116e48","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"v1","time":"2022-11-17T06:07:05Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}},"subresource":"status"},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2022-11-17T06:07:05Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/name":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"5954db5d-1892-41af-82c9-8aad46116e48\"}":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"operator\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"CONSOLE_OPERATOR_MODE\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2022-11-17T06:07:14Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.42.48.137\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]},"spec":{"volumes":[{"name":"kube-api-access-qb55d","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"operator","image":"mirrorhub.swdcmg.com/minio/console:v0.21.3","args":["server"],"ports":[{"name":"http","containerPort":9090,"protocol":"TCP"},{"name":"https","containerPort":9443,"protocol":"TCP"}],"env":[{"name":"CONSOLE_OPERATOR_MODE","value":"on"}],"resources":{},"volumeMounts":[{"name":"kube-api-access-qb55d","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"console-sa","serviceAccount":"console-sa","nodeName":"123.hdp.dc.xyz","securityContext":{"runAsUser":1000,"runAsNonRoot":true},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-11-17T06:07:05Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-11-17T06:07:14Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-11-17T06:07:14Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-11-17T06:07:05Z"}],"hostIP":"10.0.1.38","podIP":"10.42.48.137","podIPs":[{"ip":"10.42.48.137"}],"startTime":"2022-11-17T06:07:05Z","containerStatuses":[{"name":"operator","state":{"running":{"startedAt":"2022-11-17T06:07:13Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"mirrorhub.swdcmg.com/minio/console:v0.21.3","imageID":"mirrorhub.swdcmg.com/minio/console@sha256:c6d591a56c1592516b9814b930f6a466395093db64032cc1232e0e97d9b330cf","containerID":"containerd://c01cdf6b493bc85d9948c8e1330743b94640cd5fa9fb01c65423fbb50ea9e420","started":true}],"qosClass":"BestEffort"}}
I1117 17:59:25.830130 1522170 podcmd.go:88] Defaulting container name to operator
I1117 17:59:25.830901 1522170 round_trippers.go:466] curl -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.23.7+rke2r2 (linux/amd64) kubernetes/42c05a5" 'https://127.0.0.1:6443/api/v1/namespaces/minio/pods/console-656df75d45-bdb4m/exec?command=%2Fbin%2Fbash&container=operator&stdin=true&stdout=true&tty=true'
                                                                 I1117 17:59:25.861715 1522170 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 30 ms
                                                                                                                                                                                                I1117 17:59:25.861770 1522170 round_trippers.go:577] Response Headers:
                                               I1117 17:59:25.861788 1522170 round_trippers.go:580]     X-Content-Type-Options: nosniff
                                                                                                                                       I1117 17:59:25.861802 1522170 round_trippers.go:580]     Content-Length: 42
                                                                                                                                                                                                                  I1117 17:59:25.861815 1522170 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
                                                                                           I1117 17:59:25.861833 1522170 round_trippers.go:580]     Date: Thu, 17 Nov 2022 09:59:25 GMT
                                                                                                                                                                                       I1117 17:59:25.862631 1522170 helpers.go:219] server response object: [{
  "metadata": {}
}]
F1117 17:59:25.862770 1522170 helpers.go:118] Error from server:
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x331bd40, 0x3, 0x0, 0xc0001ced90, 0x2, {0x27bb393, 0x10}, 0xc000430800, 0x0)
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd
k8s.io/klog/v2.(*loggingT).printDepth(0xc0005a4318, 0x13, 0x0, {0x0, 0x0}, 0x0, {0xc00042d2c0, 0x1, 0x1})
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:735 +0x1ae
k8s.io/klog/v2.FatalDepth(...)
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1518
k8s.io/kubectl/pkg/cmd/util.fatal({0xc0005a4318, 0x13}, 0xc00049d220)
        /go/src/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:96 +0xc5
k8s.io/kubectl/pkg/cmd/util.checkErr({0x22b7c60, 0xc00049d220}, 0x212ec10)
        /go/src/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:191 +0x7d7
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /go/src/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:118
k8s.io/kubectl/pkg/cmd/exec.NewCmdExec.func1(0xc000628c80, {0xc0009ece60, 0x2, 0xa})
        /go/src/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/exec/exec.go:97 +0xc5
github.com/spf13/cobra.(*Command).execute(0xc000628c80, {0xc0009ecd20, 0xa, 0xa})
        /go/src/kubernetes/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc0007dfb80)
        /go/src/kubernetes/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/kubernetes/vendor/github.com/spf13/cobra/command.go:902
k8s.io/component-base/cli.run(0xc0007dfb80)
        /go/src/kubernetes/vendor/k8s.io/component-base/cli/run.go:146 +0x325
k8s.io/component-base/cli.RunNoErrOutput(...)
        /go/src/kubernetes/vendor/k8s.io/component-base/cli/run.go:84
main.main()
        cmd/kubectl/kubectl.go:30 +0x1e

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a
created by k8s.io/klog/v2.init.0
        /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:420 +0xfb

goroutine 35 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x22b7f60, 0xc000598000}, 0x1, 0xc0000fe360)
        /go/src/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x12a05f200, 0x0, 0x0, 0x0)
        /go/src/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x0, 0x0)
        /go/src/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x28
created by k8s.io/component-base/logs.InitLogs
        /go/src/kubernetes/vendor/k8s.io/component-base/logs/logs.go:179 +0x85

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7f2601536858, 0x72)
        /usr/local/go/src/runtime/netpoll.go:303 +0x85
internal/poll.(*pollDesc).wait(0xc000151f00, 0xc000800000, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000151f00, {0xc000800000, 0x2172, 0x2172})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000151f00, {0xc000800000, 0xc000801578, 0x1a})
        /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00011c0f8, {0xc000800000, 0x51399e, 0xc00055f7f0})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0004905d0, {0xc000800000, 0x0, 0x40c2ed})
        /usr/local/go/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc0006033f8, {0x22b5dc0, 0xc0004905d0})
        /usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000603180, {0x22b9220, 0xc00011c0f8}, 0xbff)
        /usr/local/go/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000603180, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc000603180, {0xc000231000, 0x1000, 0x9592c0})
        /usr/local/go/src/crypto/tls/conn.go:1277 +0x16f
bufio.(*Reader).Read(0xc0003e1da0, {0xc00021c820, 0x9, 0x967862})
        /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x22b5ba0, 0xc0003e1da0}, {0xc00021c820, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc00021c820, 0x9, 0xc0016df5f0}, {0x22b5ba0, 0xc0003e1da0})
        /go/src/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00021c7e0)
        /go/src/kubernetes/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00055ff98)
        /go/src/kubernetes/vendor/golang.org/x/net/http2/transport.go:2101 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00004e900)
        /go/src/kubernetes/vendor/golang.org/x/net/http2/transport.go:1997 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /go/src/kubernetes/vendor/golang.org/x/net/http2/transport.go:725 +0xac5

goroutine 27 [syscall]:
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2c

Solution

Resolved. We deployed an application that uses 10010. We changed 10010. Everything is fine now.

The above is the detailed content of Unable to execute shell on K8S pod. For more information, please follow other related articles on the PHP Chinese website!