目录
一、环境准备
二、基础镜像
三、源码编译
1.1下载源码
1.2编译kube-cross组件
1.3编译pause组件
1.4编译debian-base组件
1.5编译debian--hyperkube-base组件
1.6编译debian-iptables组件
1.7编译k8s组件
四、其他组件
1.1 Coredns编译
1.2 Flannel编译
1.3 Etcd编译
1.4 registry2.7.1镜像
附录1
一、环境准备
本次涉及到的相关组件及系统如下:
二、基础镜像
由k8s源码可知,源码是基于Debian系统构建的,因此准备好hub上龙芯的Debian源镜像
docker pull loongnix/debian:stretch
关于该镜像,没有集成golang,因此我们可以直接下载hub上龙芯已经集成的那个镜像,也可以自己基于此镜像手动集成,这里我们选择后者,集成方法如下:
docker run -it loongnix/debian:stretch /bin/bash
echo "deb http://mirrors./debian stable main contrib non-free" > /etc/apt/sources.list
apt-get udate
apt-get install golang-go
go version #查看golang版本
docker commit [CONTAINER ID] debian-golang:1.11.6
注意:docker官方说明,建议不要用commit生成镜像,这里可以自己写个Dockerfile。
三、源码编译
1.1下载源码
git clone -b release-1.13 /leechm/kubernetes.git
1.2编译kube-cross组件
目录:kubernetes/build/build-image/cross/
准备好etcd和protobuf,下载对应的版本,在Dockerfile里面可以看到。
git clone -b v3.2.24 /leechm/etcd.git
git clone -b v3.0.0 /leechm/protobuf.git
本地编译上述组件
首先编译etcd,进入etcd中,修改etcdmain/etcd.go,
执行./build,完成之后在当前bin目录下生成可执行程序。
编译protobuf,进入protobuf中,步骤如下:
./autogen.sh
./configure --prefix=/此安装目录自己命名
make
make install
cd /此安装目录自己命名
在这个目录下有三个文件夹 bin include lib 这个就是拿来写Dockerfile的
注意:上述两个组件都有可执行的二进制程序生成,在本地运行一下,未报错即可使用
接着准备好tools,xerrors和mod组件
git clone -b release-branch.go1.11 /leechm/golang-tools.git
git clone /leechm/xerrors.git
git clone /leechm/mod.git
完成上述两个组件的编译之后。进入kubernetes/build/build-image/cross/下,修改Makefile,删除--pull,修改Dockerfile,重点在于etcd和protobuf这两个组件的写法。还有golang-tools,xerrors和mod组件这三个组件在镜像中的位置:放到$GOPATH/src//x/下,例如:
最后执行make build即可,编译过程中报错什么,解决一下即可,若为未安装某包,替换源码包仓库即可,这里我们提供以下三个仓库
deb /debian stretch main
deb /ubuntu xenial main universe
deb http://mirrors./debian stable main contrib non-free
编译完成,docker images查看生成的镜像。
1.3编译pause组件
目录:kubernetes/build/pause
修改Makefile,删除--pull和增加mips64le相关代码,执行make pause即可生成pause可执行文件。
(若报错:/usr/bin/ld: cannot find -lc,装上glibc-static即可)
最后修改Dockerfile,执行docker build -t pause:tag . 即可。
1.4编译debian-base组件
目录:kubernetes/build/debian-base
修改Makefile,删除--pull选项,增加mips64le架构代码。
注意:在GitHub上,qemu-mips64-static.tar.gz 这个组件已经提供了。
我们只需要执行make build即可。
1.5编译debian--hyperkube-base组件
目录:kubernetes/build/debian-hyperkube-base/
修改Makefile,删除--pull选项,增加mips64le架构代码。由Makefile可知,需要network-plugins插件
git clone -b v0.7.5 /leechm/plugins.git
执行./build.sh
即可在./bin/生成二进制文件
cd /bin
tar czvf cni-plugins-mips64-v0.7.5.tgz *
最后执行make build 即可。
1.6编译debian-iptables组件
目录:kubernetes/build/debian-iptables/
修改Makefile,删除--pull选项,增加mips64le架构代码。执行make build即可。。
上述步骤完成之后,docker images查看相关base镜像的脚本安装包如下:
1.7编译k8s组件
依赖的镜像上面已经编译好了,接下里需要修改kubernetes/build/lib/release.sh这个文件,删除--pull这个参数选项。
修改hack/lib/golang.sh,在KUBE_SERVER_PLATFORMS,KUBE_CLIENT_PLATFORMS和KUBE_TEST_PLATFORMS中增加linux/mip64le,也可以去掉除linux/mips64le以外的其他平台,以此来减少编译所用时间。
上述就是增加对mips架构的支持,详细的修改见附录1
配置好mips支持架构之后,执行如下命令:
KUBE_BUILD_PLATFORMS=linux/mips64le KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_HYPERKUBE=n make release-images GOFLAGS=-v GOGCFLAGS="-N -l"
参数说明:
UBE_BUILD_CONFORMANCE=n 和 KUBE_BUILD_HYPERKUBE=n 参数配置是否构建 hyperkube-mips64le 和 conformance-mips64le 镜像,默认是 y 构建,这里设置为 n 暂时不需要构建了。
make release-images 表示执行编译并生成镜像 的tar 包。
GOFLAGS=-v开启verbose日志,GOGCFLAGS=”-N -l”禁止编译优化和内联,减小可执行程序大小。
分析:若出现错误,无法根据报错提示判断什么错误,可以使用docker logs [CONTAINER ID]查看容器报错信息。
最后编译好的kubernetes 组件 docker 镜像以 tar 包的形式存在kubernetes/_output/release-tars/mips64le目录中。执行docker load -i 即可导入本地。
生成的核心组件二进制可执行文件以及镜像,在 _output/release-stage/server/linux-mips64le/kubernetes/server/bin/
四、其他组件
1.1 Coredns编译
下载源码
git clone -b v1.2.2 /leechm/coredns.git
源码存放路径为:$GOPATH/src//coredns/
cd到该路径下,执行go build 命令,一般会报cannot find package的错误,这个只需要把相关缺少的组件放到对应的位置即可,此时我们开启golang代理模式,根据go.mod文件自动联网下载相关依赖
# 启用 Go Modules 功能export GO111MODULE=on# 配置 GOPROXY 环境变量export GOPROXY=https://goproxy.io
最后编译出来的二进制文件验证如下:
然后根据目录中的Dockerfile,即可生成coredns镜像。
1.2 Flannel编译
下载源码:
git clone -b v0.10.0 /leechm/flannel.git
源码存放路径为:$GOPATH/src//coreos/
官方提供的方案是基于alpine镜像做的,有兴趣的可以研究一下这个方法,在龙芯开源社区上也提供了mips架构的相关包,如:/web/#/78?page_id=293
这里我们基于中标麒麟base镜像
根据架构修改Makefile,注意CGO_ENABLED的值为1,cp一份Dockerfile.arm64为Dockerfile.mips64le,修改Dockerfile.mips64le.注意若是基于centos,则GOPACH路径需要修改,如下:
最后执行make image即可。
1.3 Etcd编译
下载源码:
git clone -b v3.2.24 /leechm/etcd.git
下载龙芯alpine镜像
docker pull loongnix/alpine:3.11
修改etcdmain/etcd.go
然后执行./build,即可在./bin/目录下生成可执行程序。
验证生成文件:
修改Dockerfile-release,接着执行:docker build -f Dockerfile-release -t etcd:v3.2.24 .
1.4 registry2.7.1镜像
mkdir -p $GOPATH/src//docker/
cd $GOPATH/src//docker/
git clone -b v2.7.1 /leechm/distribution.git
make binaries
Dockerfile如下:
执行docker build 即可生成镜像。
验证一下:
docker run -d -p 5000:5000 --restart always --name registry registry:2.7.1
附录1
diff -urN kubernetes-1.13/build/common.sh kubernetes/build/common.sh--- kubernetes-1.13/build/common.sh -07-06 19:56:02.000000000 +0800+++ kubernetes/build/common.sh-07-11 16:58:13.000000000 +0800@@ -59,9 +59,6 @@# limitations under the License.# Common utilities, variables and checks for all build scripts.- export PS4='+${BASH_SOURCE}:${LINENO}: '- set -xset -o errexitset -o nounsetset -o pipefail@@ -94,11 +96,11 @@ kube::build::get_docker_wrapped_binaries### If you change any of these lists, please also update DOCKERIZED_BINARIES### in build/BUILD. And kube::golang::server_image_targetslocal targets=(- cloud-controller-manager,"loongnixk8s/debian-base-${arch}:${debian_base_version}"- kube-apiserver,"loongnixk8s/debian-base-${arch}:${debian_base_version}"- kube-controller-manager,"loongnixk8s/debian-base-${arch}:${debian_base_version}"- kube-scheduler,"loongnixk8s/debian-base-${arch}:${debian_base_version}"- kube-proxy,"loongnixk8s/debian-iptables-${arch}:${debian_iptables_version}"+ cloud-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"+ kube-apiserver,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"+ kube-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"+ kube-scheduler,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"+ kube-proxy,"k8s.gcr.io/debian-iptables-${arch}:${debian_iptables_version}")echo "${targets[@]}"@@ -529,6 +537,7 @@ function kube::build::ensure_data_container() {--volume /usr/local/go/pkg/linux_arm_cgo--volume /usr/local/go/pkg/linux_arm64_cgo--volume /usr/local/go/pkg/linux_ppc64le_cgo- --volume /usr/local/go/pkg/linux_mips64le_cgo--volume /usr/local/go/pkg/darwin_amd64_cgo--volume /usr/local/go/pkg/darwin_386_cgo--volume /usr/local/go/pkg/windows_amd64_cgodiff -urN kubernetes-1.13/cluster/clientbin.sh kubernetes/cluster/clientbin.sh--- kubernetes-1.13/cluster/clientbin.sh -07-06 19:56:02.000000000 +0800+++ kubernetes/cluster/clientbin.sh -07-11 16:58:13.000000000 +0800@@ -59,9 +59,6 @@ppc64le*)host_arch=ppc64le;;- mips64*)- host_arch=mips64le- ;;*)echo "Unsupported host arch. Must be x86_64, 386, arm, s390x or ppc64le." >&2exit 1diff -urN kubernetes-1.13/cluster/images/conformance/Makefile kubernetes/cluster/images/conformance/Makefile--- kubernetes-1.13/cluster/images/conformance/Makefile -07-06 19:57:12.000000000 +0800+++ kubernetes/cluster/images/conformance/Makefile -07-11 16:58:13.000000000 +0800@@ -25,7 +25,7 @@E2E_TEST_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/e2e.testCLUSTER_DIR?=$(shell pwd)/../../../cluster/-BASEIMAGE=staging-k8s.gcr.io/debian-hyperkube-base-mips64:0.12.0+BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.12.0TEMP_DIR:=$(shell mktemp -d -t conformanceXXXXXX)all: build@@ -48,7 +48,7 @@cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile- docker build -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR}+ docker build --pull -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR}rm -rf "${TEMP_DIR}"push: builddiff -urN kubernetes-1.13/cluster/images/hyperkube/Makefile kubernetes/cluster/images/hyperkube/Makefile--- kubernetes-1.13/cluster/images/hyperkube/Makefile -07-06 19:58:00.000000000 +0800+++ kubernetes/cluster/images/hyperkube/Makefile -07-11 16:58:13.000000000 +0800@@ -22,7 +22,7 @@OUT_DIR?=_outputHYPERKUBE_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube-BASEIMAGE=staging-k8s.gcr.io/debian-hyperkube-base-mips64:0.12.0+BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.12.0TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX)all: build@@ -39,7 +39,7 @@cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile- docker build -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR}+ docker build --pull -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR}rm -rf "${TEMP_DIR}"push: builddiff -urN kubernetes-1.13/hack/lib/golang.sh kubernetes/hack/lib/golang.sh--- kubernetes-1.13/hack/lib/golang.sh -07-06 20:02:37.000000000 +0800+++ kubernetes/hack/lib/golang.sh -07-11 16:58:14.000000000 +0800@@ -118,7 +118,6 @@readonly KUBE_SERVER_PLATFORMS=(linux/amd64linux/arm- linux/mips64lelinux/arm64linux/s390xlinux/ppc64le@@ -129,7 +128,6 @@linux/amd64linux/armlinux/arm64- linux/mips64lelinux/s390xlinux/ppc64lewindows/amd64@@ -141,7 +139,6 @@linux/386linux/armlinux/arm64- linux/mips64lelinux/s390xlinux/ppc64ledarwin/amd64@@ -156,7 +153,6 @@linux/armlinux/arm64linux/s390x- linux/mips64lelinux/ppc64ledarwin/amd64windows/amd64@@ -334,10 +330,6 @@export CGO_ENABLED=1export CC=s390x-linux-gnu-gcc;;- "linux/mips64le")- export CGO_ENABLED=1- export CC=mips64el-linux-gnu-gcc- ;;esacfi}diff -urN kubernetes-1.13/hack/lib/util.sh kubernetes/hack/lib/util.sh--- kubernetes-1.13/hack/lib/util.sh -07-06 19:49:11.000000000 +0800+++ kubernetes/hack/lib/util.sh -07-11 16:58:14.000000000 +0800@@ -145,11 +145,8 @@ppc64le*)host_arch=ppc64le;;- mips64*)- host_arch=mips64le- ;;*)- kube::log::error "Unsupported host arch. Must be x86_64, mips64le, 386, arm, arm64, s390x or ppc64le."+ kube::log::error "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le."exit 1;;esacdiff -urN kubernetes-1.13/hack/local-up-cluster.sh kubernetes/hack/local-up-cluster.sh--- kubernetes-1.13/hack/local-up-cluster.sh -07-06 19:48:14.000000000 +0800+++ kubernetes/hack/local-up-cluster.sh -07-11 16:58:14.000000000 +0800@@ -326,11 +326,8 @@ppc64le*)host_arch=ppc64le;;- mips64*)- host_arch=mips64le- ;;*)- echo "Unsupported host arch. Must be x86_64, mips64el, 386, arm, arm64, s390x or ppc64le." >&2+ echo "Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le." >&2exit 1;;esacdiff -urN kubernetes-1.13/test/images/image-util.sh kubernetes/test/images/image-util.sh--- kubernetes-1.13/test/images/image-util.sh -07-06 20:03:19.000000000 +0800+++ kubernetes/test/images/image-util.sh -07-11 16:58:16.000000000 +0800@@ -25,7 +25,7 @@source "${KUBE_ROOT}/hack/lib/util.sh"# Mapping of go ARCH to actual architectures shipped part of multiarch/qemu-user-static project-declare -A QEMUARCHS=( ["amd64"]="x86_64" ["arm"]="arm" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" ["mips64le"]="mips64")+declare -A QEMUARCHS=( ["amd64"]="x86_64" ["arm"]="arm" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" )# Returns list of all supported architectures from BASEIMAGE filelistArchs() {diff -urN kubernetes-1.13/vendor//google/cadvisor/fs/fs.go kubernetes/vendor//google/cadvisor/fs/fs.go--- kubernetes-1.13/vendor//google/cadvisor/fs/fs.go -07-06 20:06:34.000000000 +0800+++ kubernetes/vendor//google/cadvisor/fs/fs.go -07-11 16:58:17.000000000 +0800@@ -524,8 +524,8 @@return nil, fmt.Errorf("stat failed on %s with error: %s", dir, err)}- major := major(uint64(buf.Dev))- minor := minor(uint64(buf.Dev))+ major := major(buf.Dev)+ minor := minor(buf.Dev)for device, partition := range self.partitions {if partition.major == major && partition.minor == minor {return &DeviceInfo{device, major, minor}, nildiff -urN kubernetes-1.13/vendor//vishvananda/netns/netns_linux.go kubernetes/vendor//vishvananda/netns/netns_linux.go--- kubernetes-1.13/vendor//vishvananda/netns/netns_linux.go -07-06 20:07:37.000000000 +0800+++ kubernetes/vendor//vishvananda/netns/netns_linux.go -07-11 16:58:17.000000000 +0800@@ -21,7 +21,6 @@"arm": 375,"mips": 4344,"mipsle": 4344,- "mips64le": 5303,"ppc64": 350,"ppc64le": 350,"s390x": 339,diff -urN kubernetes-1.13/vendor//x/sys/unix/zsyscall_linux_mips64le.go kubernetes/vendor//x/sys/unix/zsyscall_linux_mips64le.go--- kubernetes-1.13/vendor//x/sys/unix/zsyscall_linux_mips64le.go -07-06 20:08:36.000000000 +0800+++ kubernetes/vendor//x/sys/unix/zsyscall_linux_mips64le.go -07-11 16:58:17.000000000 +0800@@ -57,16 +57,6 @@return}- // THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDIT-func InotifyInit() (fd int, err error) {- r0, _, e1 := RawSyscall(SYS_INOTIFY_INIT, 0, 0, 0)- fd = int(r0)- if e1 != 0 {- err = errnoErr(e1)- }- return-}-// THIS FILE IS GENERATED BY THE COMMAND AT THE TOP; DO NOT EDITfunc openat(dirfd int, path string, flags int, mode uint32) (fd int, err error) {