腾讯云：一文读懂Kubernetes APIServer原理（下）

小提示：您能找到这篇{腾讯云：一文读懂Kubernetes APIServer原理（下）}绝对不是偶然，我们能帮您找到潜在客户，解决您的困扰。如果您对本页介绍的腾讯云：一文读懂Kubernetes APIServer原理（下）内容感兴趣，有相关需求意向欢迎拨打我们的服务热线，或留言咨询，我们将第一时间联系您！

< ">APIExtensionServer负责CustomResourceDefinition（CRD）apiResources以及apiVersions的注册，同时处理CRD以及相应CustomResource（CR）的REST请求(如果对应CR不能被处理的话则会返回404)，也是apiserver Delegation的最后一环

< ">crdRegistrationController负责将CRD GroupVersions自动注册到APIServices中。具体逻辑为：枚举所有CRDs，然后根据CRD定义的crd.Spec.Group以及crd.Spec.Versions字段构建APIService，并添加到autoRegisterController.apiServicesToSync中，由autoRegisterController进行创建以及维护操作。这也是为什么创建完CRD后会产生对应的APIService对象

< ">APIExtensionServer包含的controller以及功能如下所示：

< ">openapiController：将crd资源的变化同步至提供的OpenAPI文档，可通过访问/openapi/v2进行查看；

< ">crdController：负责将crd信息注册到apiVersions和apiResources中，两者的信息可通过kubectl api-versions和kubectl api-resources查看；

< ">kubectl api-versions命令返回所有Kubernetes集群资源的版本信息（实际发出了两个请求，分别是https://127.0.0.1:6443/api以及https://127.0.0.1:6443/apis，并在最后将两个请求的返回结果进行了合并）

$ kubectl -v=8 api-versions 

I1211 11:44:50.276446   22493 loader.go:375] Config loaded from file:  /root/.kube/config

I1211 11:44:50.277005   22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

...

I1211 11:44:50.290265   22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

I1211 11:44:50.293673   22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

...

I1211 11:44:50.298360   22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

apiextensions.k8s.io/v1

apiextensions.k8s.io/v1beta1

apiregistration.k8s.io/v1

apiregistration.k8s.io/v1beta1

apps/v1

authentication.k8s.io/v1beta1

...

storage.k8s.io/v1

storage.k8s.io/v1beta1

v1

< ">kubectl api-resources命令就是先获取所有API版本信息，然后对每一个API版本调用接口获取该版本下的所有API资源类型

$ kubectl -v=8 api-resources

 5077 loader.go:375] Config loaded from file:  /root/.kube/config

 I1211 15:19:47.593450   15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

 I1211 15:19:47.602273   15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

 I1211 15:19:47.606279   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

 I1211 15:19:47.610333   15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

 I1211 15:19:47.614700   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s

 I1211 15:19:47.614804   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s

 I1211 15:19:47.615687   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s

 https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s

 I1211 15:19:47.616794   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s

 I1211 15:19:47.616863   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s

 ...

 NAME                              SHORTNAMES   APIGROUP                  NAMESPACED       KIND

 bindings                                                                                         true                       Binding

 endpoints                         ep                                                          true                      Endpoints

 events                               ev                                                        &nb电商的网站sp; true                      Event

 limitranges                       limits                      &nb广告片广告sp;                               true                      LimitRange

 namespaces                     ns                                                           false                     Namespace

 nodes                               no                                                           false                    Node

 ...

< ">namingController：检查crd obj中是否有命名冲突，可在crd.status.conditions中查看；

< ">establishingController：检查crd是否处于正常状态，可在crd.status.conditions中查看；

< ">nonStructuralSchemaController：检查crd obj结构是否正常，可在crd.status.conditions中查看；

< ">apiApprovalController：检查crd是否遵循Kubernetes API声明策略，可在crd.status.conditions中查看；

< ">finalizingController：类似于finalizes的功能，与CRs的删除有关；

< ">总结CR CRUD APIServer处理逻辑如下：

// New returns a new instance of CustomResourceDefinitions from the given config.

func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error) {

  ...

    crdHandler, err := NewCustomResourceDefinitionHandler(

      versionDiscoveryHandler,

        groupDiscoveryHandler,

      s.Informers.Apiextensions().V1().CustomResourceDefinitions(),

        delegateHandler,

      c.ExtraConfig.CRDRESTOptionsGetter,

        c.GenericConfig.AdmissionControl,

      establishingController,

        c.ExtraConfig.ServiceResolver,

      c.ExtraConfig.AuthResolverWrapper,

        c.ExtraConfig.MasterCount,

        s.GenericAPIServer.Authorizer,

        c.GenericConfig.RequestTimeout,

        time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second,

        apiGroupInfo.StaticOpenAPISpec,

        c.GenericConfig.MaxRequestBodyBytes,

    )

    if err != nil {

        return nil, err

    }

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)

    ...

    return s, nil

}

< ">crdHandler处理逻辑如下：

< ">解析req(GET/apis/duyanghao.example.com/v1/namespaces/default/students)，根据请求路径中的group(duyanghao.example.com)，version(v1)，以及resource字段(students)获取对应CRD内容(crd,err:=r.crdLister.Get(crdName))

< ">通过crd.UID以及crd.Name获取crdInfo，若不存在则创建对应的crdInfo(crdInfo,err:=r.getOrCreateServingInfoFor(crd.UID,crd.Name))。crdInfo中包含了CRD定义以及该CRD对应Custom Resource的customresource.REST storage

< ">customresource.REST storage由CR对应的Group(duyanghao.example.com)，Version(v1)，Kind(Student)，Resource(students)等创建完成，由于CR在Kubernetes代码中并没有具体结构体定义，所以这里会先初始化一个范型结构体Unstructured(用于保存所有类型的Custom Resource)，并对该结构体进行SetGroupVersionKind操作(设置具体Custom Resource Type)

< ">从customresource.REST storage获取Unstructured结构体后会对其进行相应转换然后返回

// k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223

func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

  ctx := req.Context()

  requestInfo, ok := apirequest.RequestInfoFrom(ctx)

  ...

  crdName := requestInfo.Resource + "." + requestInfo.APIGroup

  crd, err := r.crdLister.Get(crdName)

  ...

  crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)

  verb := strings.ToUpper(requestInfo.Verb)

  resource := requestInfo.Resource

  subresource := requestInfo.Subresource

  scope := metrics.CleanScope(requestInfo)

  ...

  switch {

  case subresource == "status" && subresources != nil && subresources.Status != nil:

      handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case subresource == "scale" && subresources != nil && subresources.Scale != nil:

      handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case len(subresource) == 0:

      handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  default:

      responsewriters.ErrorNegotiated(

          apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),

          Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,

      )

  }

  if handlerFunc != nil {

      handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)

      handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)

      handler.ServeHTTP(w, req)

      return

  }

}

< ">更多代码原理详情，参考kubernetes-reading-notes[1]。

< ">< color: rgb(0, 112, 192);">Conclusion

< ">本文从源码层面对Kubernetes apiserver进行了一个概览性总结，包括：aggregatorServer，kubeAPIServer，apiExtensionsServer以及bootstrap-controller等。通过阅读本文可以对apiserver内部原理有一个大致的理解，另外也有助于后续深入研究。

参考资料

[1]kubernetes-reading-notes:https://github.com/duyanghao/kubernetes-reading-notes/tree/master/core/api-ser