騰訊云：一文讀懂Kubernetes APIServer原理（下）

·APIExtensionServer負(fù)責(zé)CustomResourceDefinition（CRD）apiResources以及apiVersions的注冊，同時處理CRD以及相應(yīng)CustomResource（CR）的REST請求(如果對應(yīng)CR不能被處理的話則會返回404)，也是apiserver Delegation的最后一環(huán)

·crdRegistrationController負(fù)責(zé)將CRD GroupVersions自動注冊到APIServices中。具體邏輯為：枚舉所有CRDs，然后根據(jù)CRD定義的crd.Spec.Group以及crd.Spec.Versions字段構(gòu)建APIService，并添加到autoRegisterController.apiServicesToSync中，由autoRegisterController進行創(chuàng)建以及維護操作。這也是為什么創(chuàng)建完CRD后會產(chǎn)生對應(yīng)的APIService對象

·APIExtensionServer包含的controller以及功能如下所示：

openapiController：將crd資源的變化同步至提供的OpenAPI文檔，可通過訪問/openapi/v2進行查看；

crdController：負(fù)責(zé)將crd信息注冊到apiVersions和apiResources中，兩者的信息可通過kubectl api-versions和kubectl api-resources查看；

kubectl api-versions命令返回所有Kubernetes集群資源的版本信息（實際發(fā)出了兩個請求，分別是https://127.0.0.1:6443/api以及https://127.0.0.1:6443/apis，并在最后將兩個請求的返回結(jié)果進行了合并）

$ kubectl -v=8 api-versions 

I1211 11:44:50.276446   22493 loader.go:375] Config loaded from file:  /root/.kube/config

I1211 11:44:50.277005   22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

...

I1211 11:44:50.290265   22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

I1211 11:44:50.293673   22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

...

I1211 11:44:50.298360   22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

apiextensions.k8s.io/v1

apiextensions.k8s.io/v1beta1

apiregistration.k8s.io/v1

apiregistration.k8s.io/v1beta1

apps/v1

authentication.k8s.io/v1beta1

...

storage.k8s.io/v1

storage.k8s.io/v1beta1

v1

kubectl api-resources命令就是先獲取所有API版本信息，然后對每一個API版本調(diào)用接口獲取該版本下的所有API資源類型

$ kubectl -v=8 api-resources

 5077 loader.go:375] Config loaded from file:  /root/.kube/config

 I1211 15:19:47.593450   15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

 I1211 15:19:47.602273   15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

 I1211 15:19:47.606279   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

 I1211 15:19:47.610333   15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

 I1211 15:19:47.614700   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s

 I1211 15:19:47.614804   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s

 I1211 15:19:47.615687   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s

 https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s

 I1211 15:19:47.616794   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s

 I1211 15:19:47.616863   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s

 ...

 NAME                              SHORTNAMES   APIGROUP                  NAMESPACED       KIND

 bindings                                                                                         true                       Binding

 endpoints                         ep                                                          true                      Endpoints

 events                               ev                                                          true                      Event

 limitranges                       limits                                                      true                      LimitRange

 namespaces                     ns                                                           false                     Namespace

 nodes                               no                                                           false                    Node

 ...

namingController：檢查crd obj中是否有命名沖突，可在crd.status.conditions中查看；

establishingController：檢查crd是否處于正常狀態(tài)，可在crd.status.conditions中查看；

nonStructuralSchemaController：檢查crd obj結(jié)構(gòu)是否正常，可在crd.status.conditions中查看；

apiApprovalController：檢查crd是否遵循Kubernetes API聲明策略，可在crd.status.conditions中查看；

finalizingController：類似于finalizes的功能，與CRs的刪除有關(guān)；

總結(jié)CR CRUD APIServer處理邏輯如下：

// New returns a new instance of CustomResourceDefinitions from the given config.

func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error) {

  ...

    crdHandler, err := NewCustomResourceDefinitionHandler(

      versionDiscoveryHandler,

        groupDiscoveryHandler,

      s.Informers.Apiextensions().V1().CustomResourceDefinitions(),

        delegateHandler,

      c.ExtraConfig.CRDRESTOptionsGetter,

        c.GenericConfig.AdmissionControl,

      establishingController,

        c.ExtraConfig.ServiceResolver,

      c.ExtraConfig.AuthResolverWrapper,

        c.ExtraConfig.MasterCount,

        s.GenericAPIServer.Authorizer,

        c.GenericConfig.RequestTimeout,

        time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second,

        apiGroupInfo.StaticOpenAPISpec,

        c.GenericConfig.MaxRequestBodyBytes,

    )

    if err != nil {

        return nil, err

    }

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)

    ...

    return s, nil

}

·crdHandler處理邏輯如下：

解析req(GET/apis/duyanghao.example.com/v1/namespaces/default/students)，根據(jù)請求路徑中的group(duyanghao.example.com)，version(v1)，以及resource字段(students)獲取對應(yīng)CRD內(nèi)容(crd,err:=r.crdLister.Get(crdName))

通過crd.UID以及crd.Name獲取crdInfo，若不存在則創(chuàng)建對應(yīng)的crdInfo(crdInfo,err:=r.getOrCreateServingInfoFor(crd.UID,crd.Name))。crdInfo中包含了CRD定義以及該CRD對應(yīng)Custom Resource的customresource.REST storage

customresource.REST storage由CR對應(yīng)的Group(duyanghao.example.com)，Version(v1)，Kind(Student)，Resource(students)等創(chuàng)建完成，由于CR在Kubernetes代碼中并沒有具體結(jié)構(gòu)體定義，所以這里會先初始化一個范型結(jié)構(gòu)體Unstructured(用于保存所有類型的Custom Resource)，并對該結(jié)構(gòu)體進行SetGroupVersionKind操作(設(shè)置具體Custom Resource Type)

從customresource.REST storage獲取Unstructured結(jié)構(gòu)體后會對其進行相應(yīng)轉(zhuǎn)換然后返回

// k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223

func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

  ctx := req.Context()

  requestInfo, ok := apirequest.RequestInfoFrom(ctx)

  ...

  crdName := requestInfo.Resource + "." + requestInfo.APIGroup

  crd, err := r.crdLister.Get(crdName)

  ...

  crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)

  verb := strings.ToUpper(requestInfo.Verb)

  resource := requestInfo.Resource

  subresource := requestInfo.Subresource

  scope := metrics.CleanScope(requestInfo)

  ...

  switch {

  case subresource == "status" && subresources != nil && subresources.Status != nil:

      handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case subresource == "scale" && subresources != nil && subresources.Scale != nil:

      handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case len(subresource) == 0:

      handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  default:

      responsewriters.ErrorNegotiated(

          apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),

          Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,

      )

  }

  if handlerFunc != nil {

      handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)

      handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)

      handler.ServeHTTP(w, req)

      return

  }

}

更多代碼原理詳情，參考kubernetes-reading-notes[1]。

Conclusion

本文從源碼層面對Kubernetes apiserver進行了一個概覽性總結(jié)，包括：aggregatorServer，kubeAPIServer，apiExtensionsServer以及bootstrap-controller等。通過閱讀本文可以對apiserver內(nèi)部原理有一個大致的理解，另外也有助于后續(xù)深入研究。

參考資料

[1]kubernetes-reading-notes:https://github.com/duyanghao/kubernetes-reading-notes/tree/master/core/api-ser