Live Debugging K8s
K8s DeepDive
K8s D2
BOX_IMAGE = "bento/ubuntu-18.04"
HOST_NAME = "ubuntu1804"
GO_VERSION = "1.17.8"
# privileged: true (root)
$pre_install = <<-SCRIPT
export DEBIAN_FRONTEND=noninteractive
echo ">>>> pre-install <<<<<<"
apt-get -qq update
apt-get -qq -y install gcc make pkg-config libseccomp-dev tree jq
## CPU architecture - amd64|arm64
ARCH=$(dpkg --print-architecture)
echo ">>>> install go <<<<<<"
GO_PACKAGE="go${GO_VERSION}.linux-${ARCH}".tar.gz
curl --stderr /dev/null -OL https://storage.googleapis.com/golang/${GO_PACKAGE}
tar -C /usr/local -xzf ${GO_PACKAGE}
echo 'PATH=$PATH:/usr/local/go/bin' | tee -a /etc/profile
echo ">>>>> install docker <<<<<<"
apt-get -qq -y install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl --stderr /dev/null -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=${ARCH}] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get -qq update
apt-get -qq -y install docker-ce docker-ce-cli containerd.io
SCRIPT
Vagrant.configure("2") do |config|
config.vm.define HOST_NAME do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.hostname = HOST_NAME
subconfig.vm.network :private_network, ip: "192.168.100.2"
subconfig.vm.provider "virtualbox" do |v|
v.memory = 1536
v.cpus = 2
end
sub config.vm.provision "shell", privileged: true, env: {"GO_VERSION"=>GO_VERSION}, inline: $pre_install
end
end
[실습1] 쿠버네티스 디버깅
DELVE ?
DELVE 설치
go install github.com/go-delve/delve/cmd/dlv@latest
go: downloading github.com/go-delve/delve v1.8.2�…
sudo cp $HOME/go/bin/dlv /usr/local/bin
dlv
Delve is a source level debugger for Go programs.
Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.�…
git-clone K8s
cd $HOME
git clone https://github.com/kubernetes/kubernetes.git --branch v1.19.16 --single-branch
git-clone K8s
cd $HOME/kubernetes
ls CHANGELOG�CHANGELOG-1.19.md OWNERS README.md
kube-apiserver 코드 수정
vi staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go
...
// Line 417 ~ return 34 * time.Second
return 34 * time.Minute
원활한 디버깅을 위하여 “타임아웃"을 수정합니다
kube-apiserver 코드 수정
vi staging/src/k8s.io/apiserver/pkg/server/config.go
...
// Line 307 ~ time.Duration(60) * time.Second,
RequestTimeout: time.Duration(60) * time.Minute,
원활한 디버깅을 위하여 “타임아웃"을 수정합니다
kube-apiserver 빌드 해보기
sudo make all WHAT=cmd/kube-apiserver GOFLAGS=-v GOGCFLAGS="all=-N -l" GOLDFLAGS=""
* Pre-compile ~ 코드 최적화 옵션을 다 끕니다 for debugging
GOGCFLAGS=”all=-N -l”
-N : Disable optimization
-l : Disable inlining
GOLDFLAGS=”” # 기본값이 적용되지 않도록 명시적으로 빈문자열을 줍니다 �*기본값 -s -w : (strips debug information)
바이너리 확인: $HOME/kubernetes/_output/bin/kube-apiserver
Etcd 기동
mkdir $HOME/etcd-data
sudo docker container run -d \
--volume=$HOME/etcd-data:/default.etcd \
--net=host quay.io/coreos/etcd
kube-apiserver 기동
sudo $HOME/kubernetes/_output/bin/kube-apiserver \
--etcd-servers=http://127.0.0.1:2379
curl http://localhost:8080
터미널2
kube-apiserver 기동 /w DELVE
sudo dlv \
exec $HOME/kubernetes/_output/bin/kube-apiserver \
-- \
--etcd-servers=http://127.0.0.1:2379
kube-apiserver 기동 /w DELVE
(dlv) b k8s.io/kubernetes/cmd/kube-apiserver/app.Run
Breakpoint 1 set at 0x4910352 for k8s.io/kubernetes/cmd/kube-apiserver/app.Run() …
b: break
c: continue
n: next
s: step in
p: print <변수명>
help: help
exit: 종료
[실습2] 로컬 (Mac) 환경 갖추기
기준 폴더
mkdir $HOME/k8sd2
go1.17 다운로드 및 설치
cd $HOME/k8sd2
wget https://go.dev/dl/go1.17.8.darwin-amd64.tar.gz
tar -xzf go1.17.8.darwin-amd64.tar.gz
# 확인
go/bin/go version
go/bin/go env
git-clone ETCD
cd $HOME/k8sd2
git clone -b v3.5.0 https://github.com/etcd-io/etcd.git
ETCD 빌드 및 실행
cd etcd
./build.sh
bin/etcd
{"level":"info","ts":"2022-03-23T09:15:19.439+0900","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["etcd/bin/etcd"]}
git-clone K8s
cd $HOME/k8sd2
git clone https://github.com/kubernetes/kubernetes.git --branch v1.19.16 --single-branch
Cloning into 'kubernetes'...
remote: Enumerating objects: 1055621, done.
remote: Counting objects: 100% (493/493), done.
remote: Compressing objects: 100% (297/297), done.
Receiving objects: 41% (435717/1055621), 318.07 MiB | 10.22 MiB/s
git-clone K8s
ls kubernetes/CHANGELOG
CHANGELOG-1.19.md
GoLand > 프로젝트 등록
${HOME}/k8sd2/kubernetes
GoLand > 환경설정 > GOROOT
${HOME}/k8sd2/go
GoLand > 터미널
go version
go version go1.17.8 darwin/amd64
make
(3.22) go1.18 make실패
GoLand > 터미널
go mod tidy
go mod vendor
* GoLand 에서 Run/Debug 수행을 위해 필요. 단, 이후로는 make 실행이 안됨
go mod tidy ~ go.mod의 디펜던시 정리
go mod vendor ~ 빌드에 필요한 모든 패키지를 vendor 디렉토리에 복제
GoLand > break point 설정
>> shift + shift
1
2
GoLand > Run/Debug 설정
1
2
GoLand > Run/Debug 실행
[실습3] Remote Debugging
[VM] kube-apiserver 기동
sudo nohup dlv \
--headless \
--listen=:56268 \
--api-version=2 \
exec $HOME/kubernetes/_output/bin/kube-apiserver \
-- \
--etcd-servers=http://127.0.0.1:2379 > ./apiserver.log 2>&1 &
[로컬(Mac)] GoLand 디버거 설정
1
2
3
[로컬(Mac)] GoLand Debug 실행
END