Skip to main content

24 posts tagged with "windows"

View All Tags

Describe Kubelet Service Parameters on Azure Windows node

· 4 min read

Query Kubelet service

Managed by nssm

<code class="language-bash line-numbers">C:\k>sc qc kubelet
[SC] QueryServiceConfig SUCCESS

SERVICE_NAME: kubelet
TYPE : 10 WIN32_OWN_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\k\nssm.exe
LOAD_ORDER_GROUP :
TAG : 0
DISPLAY_NAME : Kubelet
DEPENDENCIES : docker
SERVICE_START_NAME : LocalSystem

Query kubelet AppParameters by nssm

<code class="language-bash line-numbers">C:\k>nssm get kubelet Application
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

C:\k>nssm get kubelet AppParameters
c:\k\kubeletstart.ps1

Powershell scripts to start kubelet

<code class="language-powershell line-numbers">$global:MasterIP = "q1game-q1game-6adca6-e3314a8c.hcp.westus2.azmk8s.io"
$global:KubeDnsSearchPath = "svc.cluster.local"
$global:KubeDnsServiceIp = "10.0.0.10"
$global:MasterSubnet = "10.240.0.0/16"
$global:KubeClusterCIDR = "10.240.0.0/16"
$global:KubeServiceCIDR = "10.0.0.0/16"
$global:KubeBinariesVersion = "1.17.3"
$global:CNIPath = "c:\k\cni"
$global:NetworkMode = "L2Bridge"
$global:ExternalNetwork = "ext"
$global:CNIConfig = "c:\k\cni\config\$global:NetworkMode.conf"
$global:HNSModule = "c:\k\hns.psm1"
$global:VolumePluginDir = "c:\k\volumeplugins"
$global:NetworkPlugin="azure"
$global:KubeletNodeLabels="kubernetes.azure.com/role=agent,agentpool=q1win,storageprofile=managed,storagetier=Premium_LRS,kubernetes.azure.com/cluster=MC_q1game_q1game_westus2"
Write-Host "NetworkPlugin azure, starting kubelet."

# Turn off Firewall to enable pods to talk to service endpoints. (Kubelet should eventually do this)
netsh advfirewall set allprofiles state off
# startup the service

# Find if network created by CNI exists, if yes, remove it
# This is required to keep the network non-persistent behavior
# Going forward, this would be done by HNS automatically during restart of the node

$hnsNetwork = Get-HnsNetwork | ? Name -EQ azure
if ($hnsNetwork)
{
# Cleanup all containers
docker ps -q | foreach {docker rm $_ -f}

Write-Host "Cleaning up old HNS network found"
Remove-HnsNetwork $hnsNetwork
# Kill all cni instances & stale data left by cni
# Cleanup all files related to cni
taskkill /IM azure-vnet.exe /f
taskkill /IM azure-vnet-ipam.exe /f
$cnijson = [io.path]::Combine("c:\k", "azure-vnet-ipam.json")
if ((Test-Path $cnijson))
{
Remove-Item $cnijson
}
$cnilock = [io.path]::Combine("c:\k", "azure-vnet-ipam.json.lock")
if ((Test-Path $cnilock))
{
Remove-Item $cnilock
}

$cnijson = [io.path]::Combine("c:\k", "azure-vnet.json")
if ((Test-Path $cnijson))
{
Remove-Item $cnijson
}
$cnilock = [io.path]::Combine("c:\k", "azure-vnet.json.lock")
if ((Test-Path $cnilock))
{
Remove-Item $cnilock
}
}

# Restart Kubeproxy, which would wait, until the network is created
# This was fixed in 1.15, workaround still needed for 1.14 https://github.com/kubernetes/kubernetes/pull/78612
Restart-Service Kubeproxy

$env:AZURE_ENVIRONMENT_FILEPATH="c:\k\azurestackcloud.json"

c:\k\kubelet.exe --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --azure-container-registry-config=c:\k\azure.json --cgroups-per-qos=false --client-ca-file=c:\k\ca.crt --cloud-config=c:\k\azure.json --cloud-provider=azure --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --dynamic-config-dir=/var/lib/kubelet --enforce-node-allocatable="" --event-qps=0 --eviction-hard="" --feature-gates=RotateKubeletServerCertificate=true --hairpin-mode=promiscuous-bridge --image-gc-high-threshold=85 --image-gc-low-threshold=80 --image-pull-progress-deadline=20m --keep-terminated-pod-volumes=false --kube-reserved=cpu=100m,memory=1843Mi --kubeconfig=c:\k\config --max-pods=30 --network-plugin=cni --node-status-update-frequency=10s --non-masquerade-cidr=0.0.0.0/0 --pod-infra-container-image=kubletwin/pause --pod-max-pids=-1 --protect-kernel-defaults=true --read-only-port=0 --resolv-conf="" --rotate-certificates=false --streaming-connection-idle-timeout=4h --system-reserved=memory=2Gi --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --node-labels=$global:KubeletNodeLabels --volume-plugin-dir=$global:VolumePluginDir --cni-bin-dir=c:\k\azurecni\bin --cni-conf-dir=c:\k\azurecni\netconf

Enable Hyper-V Isolation by modify kubelet parameters

1. Modify c:\k\kubeletstart.ps1 to add parameter to kubelet

--feature-gates="XXX=true,HyperVContainer=true"

2. Restart kubelet
<code class="language-bash line-numbers">C:\k>nssm restart kubelet
Kubelet: STOP: A stop control has been sent to a service that other running services are dependent on.

C:\k>sc queryex kubelet

SERVICE_NAME: kubelet
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
PID : 4044
FLAGS :

C:\k>taskkill /PID 4044 /F

C:\k>sc start kubelet

Restart the Windows node if necessary

Run Windows container with Hyper-V isolation mode in Kubernetes

· 3 min read

Windows Container 有两种隔离运行模式 Hyper-V 和 Process, 参见:Isolation Modes

两种模式下的 host 的 OS 版本与 containter 的 OS 版本存在兼容性又不相同,参见:Windows container version compatibility

很明显 Hyper-V 模式的兼容性要比 Process 模式要好,向下兼容,也就是高版本的 host OS 可以运行低版本的 container OS, 反之不行;

而 Process 模式下 Windows Server 中则要求 host OS 与 container OS 的版本完全相同, Windows 10 中则不支持 Process 模式.

某一天,我想在 Kubernetes Windows 节点中以 Hyper-V 模式运行 Container, 于是乎发现1.17 的文档中写道:

Note: In this document, when we talk about Windows containers we mean Windows containers with process isolation. Windows containers with Hyper-V isolation is planned for a future release.

不甘心又 google 了一下,发现:

1. 有人提了 bug, 已经被修复了: https://github.com/kubernetes/kubernetes/issues/58750 2. 代码也 merge 了: https://github.com/kubernetes/kubernetes/pull/58751 3. 有人在测试过程中遇到问题,也解决了: https://github.com/kubernetes/kubernetes/issues/62812

但我测试的过程中却提示:

Error response from daemon: CreateComputeSystem test: The container operating system does not match the host operating system.

我的环境:

Kubernetes Ver: 1.14.8

Kubernetes Node OS Ver: Windows Server Datacenter 10.0.17763.504, 属于 1809 的版本

Container Base Image: windowsservercore-1709

Deployment yaml:

apiVersion: <span style="color: #ff0000;">apps/v1beta2</span>
kind: Deployment
metadata:
labels:
app: test
name: test
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
annotations:
<span style="color: #ff0000;">experimental.windows.kubernetes.io/isolation-type: hyperv</span>
labels:
app: test
...

然后对比了下 github 别人试成功的 deployment yaml, 发现人家用的是 apps/v1

apiVersion: <span style="color: #ff0000;">apps/v1</span>
kind: Deployment
metadata:
name: whoami
labels:
app: whoami
spec:
...

目前在 k8s 中启用 hyperv isolation 的三个条件:

  1. kubelet 启用参数: --feature-gates=HyperVContainer=true
  2. Pod/Deployment apiVersion: apps/v1
  3. spec.template.metadata.annotations[].experimental.windows.kubernetes.io/isolation-type:hyperv

参见: https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#hyper-v-isolation

目前我的云提供商给的 kubernetes 1.14.8 又不支持 apps/v1 ...

于是乎,我要么等提供商升级 kubernetes,要么自己升级 container OS 跟 kubernetes node OS 一样...

ClustrMaps