騰訊云：一文讀懂Kubernetes APIServer原理（中）

kubeAPIServer代碼結(jié)構(gòu)整理如下：

1. apiserver整體啟動(dòng)邏輯 k8s.io/kubernetes/cmd/kube-apiserver

2. apiserver bootstrap-controller創(chuàng)建&運(yùn)行邏輯 k8s.io/kubernetes/pkg/master

3. API Resource對(duì)應(yīng)后端RESTStorage(based on genericregistry.Store)創(chuàng)建 k8s.io/kubernetes/pkg/registry

4. aggregated-apiserver創(chuàng)建&處理邏輯 k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator

5. extensions-apiserver創(chuàng)建&處理邏輯 k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver

6. apiserver創(chuàng)建&運(yùn)行 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server

7. 注冊(cè)API Resource資源處理handler(InstallREST&Install?isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints

8. 創(chuàng)建存儲(chǔ)后端(etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage

genericregistry.Store.CompleteWithOptions初始化 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

調(diào)用鏈整理如下：

更多代碼原理詳情，參考kubernetes-reading-notes[1]。

aggregatorServer

aggregatorServer主要用于處理擴(kuò)展Kubernetes API Resources的第二種方式Aggregated APIServer(AA)，將CR請(qǐng)求代理給AA：

這里結(jié)合Kubernetes官方給出的aggregated apiserver例子sample-apiserver，總結(jié)原理如下：

·aggregatorServer通過(guò)APIServices對(duì)象關(guān)聯(lián)到某個(gè)Service來(lái)進(jìn)行請(qǐng)求的轉(zhuǎn)發(fā)，其關(guān)聯(lián)的Service類(lèi)型進(jìn)一步?jīng)Q定了請(qǐng)求轉(zhuǎn)發(fā)的形式。aggregatorServer包括一個(gè)GenericAPIServer和維護(hù)自身狀態(tài)的Controller。其中GenericAPIServer主要處理apiregistration.k8s.io組下的APIService資源請(qǐng)求，而Controller包括：

apiserviceRegistrationController：負(fù)責(zé)根據(jù)APIService定義的aggregated server service構(gòu)建代理，將CR的請(qǐng)求轉(zhuǎn)發(fā)給后端的aggregated server

availableConditionController：維護(hù)APIServices的可用狀態(tài)，包括其引用Service是否可用等；

autoRegistrationController：用于保持API中存在的一組特定的APIServices；

crdRegistrationController：負(fù)責(zé)將CRD GroupVersions自動(dòng)注冊(cè)到APIServices中；

openAPIAggregationController：將APIServices資源的變化同步至提供的OpenAPI文檔；

·apiserviceRegistrationController負(fù)責(zé)根據(jù)APIService定義的aggregated server service構(gòu)建代理，將CR的請(qǐng)求轉(zhuǎn)發(fā)給后端的aggregated server。apiService有兩種類(lèi)型：Local(Service為空)以及Service(Service非空)。apiserviceRegistrationController負(fù)責(zé)對(duì)這兩種類(lèi)型apiService設(shè)置代理：Local類(lèi)型會(huì)直接路由給kube-apiserver進(jìn)行處理；而Service類(lèi)型則會(huì)設(shè)置代理并將請(qǐng)求轉(zhuǎn)化為對(duì)aggregated Service的請(qǐng)求(proxyPath:="/apis/"+apiService.Spec.Group+"/"+apiService.Spec.Version)，而請(qǐng)求的負(fù)載均衡策略則是優(yōu)先本地訪問(wèn)kube-apiserver(如果service為kubernetes default apiserver service:443)=>通過(guò)service ClusterIP:Port訪問(wèn)(默認(rèn))或者通過(guò)隨機(jī)選擇service endpoint backend進(jìn)行訪問(wèn)：

func (s *APIAggregator) AddAPIService(apiService *v1.APIService) error {

  ...

    proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version

    // v1. is a special case for the legacy API.  It proxies to a wider set of endpoints.

    if apiService.Name == legacyAPIServiceName {

        proxyPath = "/api"

    }

    // register the proxy handler

    proxyHandler := &proxyHandler{

        localDelegate:   s.delegateHandler,

        proxyClientCert: s.proxyClientCert,

        proxyClientKey:  s.proxyClientKey,

        proxyTransport:  s.proxyTransport,

        serviceResolver: s.serviceResolver,

        egressSelector:  s.egressSelector,

    }

  ...

    s.proxyHandlers[apiService.Name] = proxyHandler

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(proxyPath, proxyHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix(proxyPath+"/", proxyHandler)

  ...

    // it's time to register the group aggregation endpoint

    groupPath := "/apis/" + apiService.Spec.Group

    groupDiscoveryHandler := &apiGroupHandler{

        codecs:    aggregatorscheme.Codecs,

        groupName: apiService.Spec.Group,

        lister:    s.lister,

        delegate:  s.delegateHandler,

    }

    // aggregation is protected

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(groupPath, groupDiscoveryHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle(groupPath+"/", groupDiscoveryHandler)

    s.handledGroups.Insert(apiService.Spec.Group)

    return nil

}

// k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109

func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

    // 加載roxyHandlingInfo處理請(qǐng)求  

    value := r.handlingInfo.Load()

    if value == nil {

        r.localDelegate.ServeHTTP(w, req)

        return

    }

    handlingInfo := value.(proxyHandlingInfo)

  ...

    // 判斷APIService服務(wù)是否正常

    if !handlingInfo.serviceAvailable {

        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)

        return

    }

    // 將原始請(qǐng)求轉(zhuǎn)化為對(duì)APIService的請(qǐng)求

    // write a new location based on the existing request pointed at the target service

    location := &url.URL{}

    location.Scheme = "https"

    rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort)

    if err != nil {

        klog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err)

        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)

        return

    }

    location.Host = rloc.Host

    location.Path = req.URL.Path

    location.RawQuery = req.URL.Query().Encode()

    newReq, cancelFn := newRequestForProxy(location, req)

    defer cancelFn()

   ...

    proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper)

    handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})

    handler.ServeHTTP(w, newReq)

}

$ kubectl get APIService           

NAME                                           SERVICE                                    AVAILABLE     AGE

...

v1.apps                                         Local                                         True                50d

...

v1beta1.metrics.k8s.io                 kube-system/metrics-server     True                50d

...

# default APIServices

$ kubectl get -o yaml APIService/v1.apps

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

  labels:

    kube-aggregator.kubernetes.io/automanaged: onstart

  name: v1.apps

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.apps

spec:

  group: apps

  groupPriorityMinimum: 17800

  version: v1

  versionPriority: 15

status:

  conditions:

  - lastTransitionTime: "2020-10-20T10:39:48Z"

    message: Local APIServices are always available

    reason: Local

    status: "True"

    type: Available

# aggregated server    

$ kubectl get -o yaml APIService/v1beta1.metrics.k8s.io

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

  labels:

    addonmanager.kubernetes.io/mode: Reconcile

    kubernetes.io/cluster-service: "true"

  name: v1beta1.metrics.k8s.io

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io

spec:

  group: metrics.k8s.io

  groupPriorityMinimum: 100

  insecureSkipTLSVerify: true

  service:

    name: metrics-server

    namespace: kube-system

    port: 443

  version: v1beta1

  versionPriority: 100

status:

  conditions:

  - lastTransitionTime: "2020-12-05T00:50:48Z"

    message: all checks passed

    reason: Passed

    status: "True"

    type: Available

# CRD

$ kubectl get -o yaml APIService/v1.duyanghao.example.com

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

  labels:

    kube-aggregator.kubernetes.io/automanaged: "true"

  name: v1.duyanghao.example.com

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.duyanghao.example.com

spec:

  group: duyanghao.example.com

  groupPriorityMinimum: 1000

  version: v1

  versionPriority: 100

status:

  conditions:

  - lastTransitionTime: "2020-12-11T08:45:37Z"

    message: Local APIServices are always available

    reason: Local

    status: "True"

    type: Available

·aggregatorServer創(chuàng)建過(guò)程中會(huì)根據(jù)所有kube-apiserver定義的API資源創(chuàng)建默認(rèn)的APIService列表，名稱(chēng)即是$VERSION.$GROUP，這些APIService都會(huì)有標(biāo)簽kube-aggregator.kubernetes.io/automanaged:onstart，例如：v1.apps apiService。autoRegistrationController創(chuàng)建并維護(hù)這些列表中的APIService，也即我們看到的Local apiService；對(duì)于自定義的APIService(aggregated server)，則不會(huì)對(duì)其進(jìn)行處理

·aggregated server實(shí)現(xiàn)CR(自定義API資源)的CRUD API接口，并可以靈活選擇后端存儲(chǔ)，可以與core kube-apiserver一起公用etcd，也可自己獨(dú)立部署etcd數(shù)據(jù)庫(kù)或者其它數(shù)據(jù)庫(kù)。aggregated server實(shí)現(xiàn)的CR API路徑為：/apis/VERSION，具體到sample apiserver為：/apis/wardle.example.com/v1alpha1，下面的資源類(lèi)型有：flunders以及fischers

·aggregated server通過(guò)部署APIService類(lèi)型資源，service fields指向?qū)?yīng)的aggregated server service實(shí)現(xiàn)與core kube-apiserver的集成與交互

sample-apiserver目錄結(jié)構(gòu)如下，可參考編寫(xiě)自己的aggregated server：

staging/src/k8s.io/sample-apiserver

├── artifacts

│   ├── example

│   │   ├── apiservice.yaml

        ...

├── hack

├── main.go

└── pkg

  ├── admission

  ├── apis

  ├── apiserver

  ├── cmd

  ├── generated

  │   ├── clientset

  │   │   └── versioned

                ...

  │   │       └── typed

  │   │           └── wardle

  │   │               ├── v1alpha1

  │   │               └── v1beta1

  │   ├── informers

  │   │   └── externalversions

  │   │       └── wardle

  │   │           ├── v1alpha1

  │   │           └── v1beta1

  │   ├── listers

  │   │   └── wardle

  │   │       ├── v1alpha1

  │   │       └── v1beta1

  └── registry

·其中，artifacts用于部署yaml示例

·hack目錄存放自動(dòng)腳本(eg:update-codegen)

·main.go是aggregated server啟動(dòng)入口；pkg/cmd負(fù)責(zé)啟動(dòng)aggregated server具體邏輯；pkg/apiserver用于aggregated server初始化以及路由注冊(cè)

·pkg/apis負(fù)責(zé)相關(guān)CR的結(jié)構(gòu)體定義，自動(dòng)生成(update-codegen)

·pkg/admission負(fù)責(zé)準(zhǔn)入的相關(guān)代碼

·pkg/generated負(fù)責(zé)生成訪問(wèn)CR的clientset，informers，以及l(fā)isters

·pkg/registry目錄負(fù)責(zé)CR相關(guān)的RESTStorage實(shí)現(xiàn)

更多代碼原理詳情，參考kubernetes-reading-notes[1]。

apiExtensionsServer

apiExtensionsServer主要負(fù)責(zé)CustomResourceDefinition（CRD）apiResources以及apiVersions的注冊(cè)，同時(shí)處理CRD以及相應(yīng)CustomResource（CR）的REST請(qǐng)求(如果對(duì)應(yīng)CR不能被處理的話則會(huì)返回404)，也是apiserver Delegation的最后一環(huán)

原理總結(jié)如下：

·Custom Resource，簡(jiǎn)稱(chēng)CR，是Kubernetes自定義資源類(lèi)型，與之相對(duì)應(yīng)的就是Kubernetes內(nèi)置的各種資源類(lèi)型，例如Pod、Service等。利用CR我們可以定義任何想要的資源類(lèi)型

·CRD通過(guò)yaml文件的形式向Kubernetes注冊(cè)CR實(shí)現(xiàn)自定義api-resources，屬于第二種擴(kuò)展Kubernetes API資源的方式，也是普遍使用的一種