腾讯云：一文读懂Kubernetes APIServer原理（中）

小提示：您能找到这篇{腾讯云：一文读懂Kubernetes APIServer原理（中）}绝对不是偶然，我们能帮您找到潜在客户，解决您的困扰。如果您对本页介绍的腾讯云：一文读懂Kubernetes APIServer原理（中）内容感兴趣，有相关需求意向欢迎拨打我们的服务热线，或留言咨询，我们将第一时间联系您！

< ">kubeAPIServer代码结构整理如下：

1. apiserver整体启动逻辑 k8s.io/kubernetes/cmd/kube-apiserver

2. apiserver bootstrap-controller创建&运行逻辑 k8s.io/kubernetes/pkg/master

3. API Resource对应后端RESTStorage(based on genericregistry.Store)创建 k8s.io/kubernetes/pkg/registry

4. aggregated-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator

5. extensions-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver

6. apiserver创建&运行 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server

7. 注册API Resource资源处理handler(InstallREST&Install?isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints

8. 创建存储后端(etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage

genericregistry.Store.CompleteWithOptions初始化 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

< ">调用链整理如下：

< ">更多代码原理详情，参考kubernetes-reading-notes[1]。

< ">< color: rgb(0, 112, 192);">aggregatorServer< color: rgb(0, 112, 192);">

< ">aggregatorServer主要用于处理扩展Kubernetes API Resources的第二种方式Aggregated APIServer(AA)，将CR请求代理给AA：

< ">这里结合Kubernetes官方给出的aggregated apiserver例子sample-apiserver，总结原理如下：

< ">aggregatorServer通过APIServices对象关联到某个Service来进行请求的转发，其关联的Service类型进一步决定了请求转发的形式。aggregatorServer包括一个GenericAPIServer和维护自身状态的Controller。其中GenericAPIServer主要处理apiregistration.k8s.io组下的APIService资源请求，而Controller包括：

< ">apiserviceRegistrationController：负责根据APIService定义的aggregated server service构建代理，将CR的请求转发给后端的aggregated server

< ">availableConditionController：维护APIServices的可用状态，包括其引用Service是否可用等；

< ">autoRegistrationController：用于保持API中存在的一组特定的APIServices；

< ">crdRegistrationController：负责将CRD GroupVersions自动注册到APIServices中；

< ">openAPIAggregationController：将APIServices资源的变化同步至提供的OpenAPI文档；

< ">apiserviceRegistrationController负责根据APIService定义的aggregated server service构建代理，将CR的请求转发给后端的aggregated server。apiService有两种类型：Local(Service为空)以及Service(Service非空)。apiserviceRegistrationController负责对这两种类型apiService设置代理：Local类型会直接路由给kube-apiserver进行处理；而Service类型则会设置代理并将请求转化为对aggregated Service的请求(proxyPath:="/apis/"+apiService.Spec.Group+"/"+apiService.Spec.Version)，而请求的负载均衡策略则是优先本地访问kube-apiserver(如果service为kubernetes default apiserver service:443)=>通过service ClusterIP:Port访问(默认)或者通过随机选择service endpoint backend进行访问：

func (s *APIAggregator) AddAPIService(apiService *v1.APIService) error {

  ...

    proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version

    // v1. is a special case for the legacy API.  It proxies to a wider set of endpoints.

    if apiService.Name == legacyAPIServiceName {

        proxyPath = "/api"

    }

    // register the proxy handler

    proxyHandler := &proxyHandler{

        localDelegate:   s.delegateHandler,

        proxyClientCert: s.proxyClientCert,

        proxyClientKey:  s.proxyClientKey,

        proxyTransport:  s.proxyTransport,

        serviceResolver: s.serviceResolver,

        egressSelector:  s.egressSelector,

    }

  ...

    s.proxyHandlers[apiService.Name] = proxyHandler

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(proxyPath, proxyHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix(proxyPath+"/", proxyHandler)

  ...

    // it's time to register the group aggregation endpoint

    groupPath := "/apis/" + apiService.Spec.Group

    groupDiscoveryHandler := &apiGroupHandler{

        codecs:    aggregatorscheme.Codecs,

        groupName: apiService.Spec.Group,

        lister:    s.lister,

        delegate:  s.delegateHandler,

    }

    // aggregation is protected

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(groupPath, groupDiscoveryHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle(groupPath+"/", groupDiscoveryHandler)

    s.handledGroups.Insert(apiService.Spec.Group)

    return nil

}

// k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109

func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

    // 加载roxyHandlingInfo处理请求  

    value := r.handlingInfo.Load()

    if value == nil {

        r.localDelegate.ServeHTTP(w, req)

        return

    }

    handlingInfo := value.(proxyHandlingInfo)

  ...

    // 判断APIService服务是否正常

    if !handlingInfo.serviceAvailable {

        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)

        return

    }

    // 将原始请求转化为对APIService的请求

    // write a new location based on the existing request pointed at the target service

    location := &url.URL{}

    location.Scheme = "https"

    rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort)

    if err != nil {

        klog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err)

        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)

        return

    }

    location.Host = rloc.Host

    location.Path = req.URL.Path

    location.RawQuery = req.URL.Query().Encode()

    newReq, cancelFn := newRequestForProxy(location, req)

    defer cancelFn()

   ...

    proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper)

    handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})

    handler.ServeHTTP(w, newReq)

}

$ kubectl get APIService           

NAME                                           SERVICE                                    AVAILABLE     AGE

...

v1.apps                                         Local                                         True                50d

...

v1beta1.metrics.k8s.io                 kube-system/metrics-server     True                50d

...

# default APIServices

$ kubectl get -o yaml APIService/v1.apps

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

  labels:

    kube-aggregator.kubernetes.io/automanaged: onstart

  name: v1.apps

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.apps

spec:

  group: apps

  groupPriorityMinimum: 17800

  version: v1

  versionPriority: 15

status:

  conditions:

  - last新开店铺怎么引流TransitionTime: "2020-10-20T10:39:48Z"

    message: Local APIServices are always available

    reason: Local

    status: "True"

    type: Available

# aggregated server    

$ kubectl get -o yaml APIService/v1beta1.metrics.k8s.io

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

  labels:

    addonmanager.kubernetes.io/mode: Reconcile

    kubernetes.io/cluster-service: "true"

  name: v1beta1.metrics.k8s.io

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io

spec:

  group: metrics.k8s.io

  groupPriorityMinimum: 100

  insecureSkipTLSVerify: true

  service:

    name: metrics-server

    namespace: kube-system

    port: 443

  version: v1beta1

  versionPriority: 100

status:

  conditions:

  - lastTransitionTime: "2020-12-05T00:50:48Z"

    message: all checks passed

    reason: Passed

    status: "True"

    type: Available

# CRD

$ kubectl get -o yaml APIService/v1.duyanghao.example.com

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metada营销策划方案范本ta:

  labels:

    kube-aggregator.kubernetes.io/automanaged: "true"

  name: v1.duyanghao.example.com

  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.duyanghao.example.com

spec:

  group: duyanghao.example.com

  groupPriorityMinimum: 1000

  version: v1

  versionPriority: 100

status:

  conditions:

  - lastTransitionTime: "2020-12-11T08:45:37Z"

    message: Local APIServices are always available

    reason: Local

    status: "True"

    type: Available

< ">aggregatorServer创建过程中会根据所有kube-apiserver定义的API资源创建默认的APIService列表，名称即是$VERSION.$GROUP，这些APIService都会有标签kube-aggregator.kubernetes.io/automanaged:onstart，例如：v1.apps apiService。autoRegistrationController创建并维护这些列表中的APIService，也即我们看到的Local apiService；对于自定义的APIService(aggregated server)，则不会对其进行处理

< ">aggregated server实现CR(自定义API资源)的CRUD API接口，并可以灵活选择后端存储，可以与core kube-apiserver一起公用etcd，也可自己独立部署etcd数据库或者其它数据库。aggregated server实现的CR API路径为：/apis/VERSION，具体到sample apiserver为：/apis/wardle.example.com/v1alpha1，下面的资源类型有：flunders以及fischers

< ">aggregated server通过部署APIService类型资源，service fields指向对应的aggregated server service实现与core kube-apiserver的集成与交互

< ">sample-apiserver目录结构如下，可参考编写自己的aggregated server：

staging/src/k8s.io/sample-apiserver

├── artifacts

│   ├── example

│   │   ├── apiservice.yaml

        ...

├── hack

├── main.go

└── pkg

  ├── admission

  ├── apis

  ├── apiserver

  ├── cmd

  ├── generated

  │   ├── clientset有哪些营销方式和渠道

  │   │   └── versioned

                ...

  │   │       └── typed

  │   │           └── wardle

  │   │               ├── v1alpha1

  │   │               └── v1beta1

  │   ├── informers

  │   │   └── externalversions

  │   │       └── wardle

  │   │           ├── v1alpha1

  │   │           └── v1beta1

  │   ├── listers

  │   │   └── wardle

  │   │       ├── v1alpha1

  │   │       └── v1beta1

  └── registry

< ">其中，artifacts用于部署yaml示例

< ">hack目录存放自动脚本(eg:update-codegen)

< ">main.go是aggregated server启动入口；pkg/cmd负责启动aggregated server具体逻辑；pkg/apiserver用于aggregated server初始化以及路由注册

< ">pkg/apis负责相关CR的结构体定义，自动生成(update-codegen)

< ">pkg/admission负责准入的相关代码

< ">pkg/generated负责生成访问CR的clientset，informers，以及listers

< ">pkg/registry目录负责CR相关的RESTStorage实现

< ">更多代码原理详情，参考kubernetes-reading-notes[1]。

< ">

apiExtensionsServer

< ">apiExtensionsServer主要负责CustomResourceDefinition（CRD）apiResources以及apiVersions的注册，同时处理CRD以及相应CustomResource（CR）的REST请求(如果对应CR不能被处理的话则会返回404)，也是apiserver Delegation的最后一环

< ">原理总结如下：

< ">Custom Resource，简称CR，是Kubernetes自定义资源类型，与之相对应的就是Kubernetes内置的各种资源类型，例如Pod、Service等。利用CR我们可以定义任何想要的资源类型

< ">CRD通过yaml文件的形式向Kubernetes注册CR实现自定义api-resources，属于第二种扩展Kubernetes API资源的方式，也是普遍使用的一种