Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: InitContainer support for ACI Connector #204

Merged
merged 66 commits into from
Nov 19, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
e979e99
ignore helm init and pod specs example
t-ysalazar Jun 7, 2022
7fc6c1b
add initContainers property
t-ysalazar Jun 7, 2022
16c676b
declaration of getInitContainers
t-ysalazar Jun 7, 2022
c991e03
getInitContainers
t-ysalazar Jun 7, 2022
cbe2be4
factorize getVolumeMounts() from getContainers()
t-ysalazar Jun 7, 2022
ff1a32f
fix type errors
t-ysalazar Jun 7, 2022
1b051a5
factorize getEnvironmentVariables
t-ysalazar Jun 7, 2022
b967171
basics of init containers on getInitContainer
t-ysalazar Jun 8, 2022
0ed160d
getBasicContainer
t-ysalazar Jun 8, 2022
179f9ce
getBasicContainer on initContainer
t-ysalazar Jun 8, 2022
c5fba52
implementation of InitContainerDefinition type
t-ysalazar Jun 9, 2022
4ac821a
factorize verifyContainer from getContainer
t-ysalazar Jun 9, 2022
4a63bbf
throw erros with unsupported properties in ACI initContainers
t-ysalazar Jun 9, 2022
c98c7c4
fix .gitignore for PR
t-ysalazar Jun 9, 2022
3d4bd30
unit test for InitContainers
t-ysalazar Jun 9, 2022
db4eef2
revert gitignore changes
t-ysalazar Jun 10, 2022
98686e2
comments on added methods
t-ysalazar Jun 10, 2022
767a445
complete unit test for ACI InitContainers
t-ysalazar Jun 10, 2022
fd52af9
Move the InitContainerDefinition closer to InitContainerProperties
t-ysalazar Jun 10, 2022
bdf2cc1
Revert "ignore helm init and pod specs example"
t-ysalazar Jun 13, 2022
a161e72
e2e for init containers
t-ysalazar Jul 5, 2022
bd4ec99
e2e for init containers
t-ysalazar Jul 5, 2022
443468c
e2e for init containers, move podName and podDir
t-ysalazar Jul 5, 2022
961171e
e2e comments
t-ysalazar Jul 5, 2022
d0d7228
Merge branch 'master' into t-ysalazar/init-containers
t-ysalazar Jul 6, 2022
5daedd1
fix pod specs for init containers in e2e
t-ysalazar Jul 6, 2022
df6eb45
Merge branch 't-ysalazar/init-containers' of https://github.com/t-ysa…
t-ysalazar Jul 6, 2022
5ef36bc
fix pod specs for init containers in e2e
t-ysalazar Jul 6, 2022
4f9b1e2
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 7, 2022
7ba7fb1
Merge branch 'virtual-kubelet:master' into t-ysalazar/init-containers
t-ysalazar Jul 7, 2022
2506197
Merge branch 'virtual-kubelet:master' into t-ysalazar/init-containers
t-ysalazar Jul 11, 2022
3c9d25a
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 11, 2022
fe5cbd1
Merge branch 'virtual-kubelet:master' into master
t-ysalazar Jul 20, 2022
05f87ad
Merge branch 'master' into t-ysalazar/init-containers
t-ysalazar Jul 20, 2022
65e77f5
fix merge conflict
t-ysalazar Jul 20, 2022
cd68b60
Merge branch 't-ysalazar/init-containers' of https://github.com/t-ysa…
t-ysalazar Jul 20, 2022
10f2b3c
Merge branch 'virtual-kubelet:master' into t-ysalazar/init-containers
suselva Jul 21, 2022
bcfe341
Merge branch 'virtual-kubelet:master' into t-ysalazar/init-containers
t-ysalazar Jul 27, 2022
f97e251
added init containers usage to README
fnuarnav Aug 17, 2022
d8600b8
Merge remote-tracking branch 'upstream/master'
fnuarnav Sep 9, 2022
fff9885
Merge branch 'master' into t-ysalazar/init-containers
fnuarnav Sep 9, 2022
1040b31
use separate test for init container for clean namespace
fnuarnav Sep 13, 2022
bfe957b
remove unused code
fnuarnav Sep 13, 2022
6f547ea
removed rg from helm install since role isn't present
fnuarnav Sep 14, 2022
aa31d28
merged changes from upstream
fnuarnav Oct 28, 2022
3d72f73
added initContainer using sdk
fnuarnav Oct 28, 2022
9e7fd19
added unit tests for init containers
fnuarnav Oct 31, 2022
0164951
renoved duplicate test
fnuarnav Oct 31, 2022
007f743
removed init containers from old code before sdk
fnuarnav Oct 31, 2022
b60194b
removed init container test from old code before sdk
fnuarnav Oct 31, 2022
186c220
removed unused import
fnuarnav Oct 31, 2022
7b79e29
Merge remote-tracking branch 'upstream/master' into t-ysalazar/init-c…
fnuarnav Oct 31, 2022
899e83b
fix error with pointer
fnuarnav Nov 1, 2022
2b53bf4
added initContainer to AzureFilesVolume test; log errors; error text …
fnuarnav Nov 2, 2022
0f214fc
removed init containers from limitations
fnuarnav Nov 2, 2022
79920d2
added volume to init container e2e test
fnuarnav Nov 2, 2022
4c1c6ac
Merge remote-tracking branch 'upstream/master' into t-ysalazar/init-c…
fnuarnav Nov 9, 2022
ee2d54d
create pod correctly in e2e test
fnuarnav Nov 9, 2022
0e59b10
Merge remote-tracking branch 'upstream/master' into t-ysalazar/init-c…
fnuarnav Nov 10, 2022
a75b217
added e2e test for init container order
fnuarnav Nov 14, 2022
7ca3314
merged changes from upstream/master
fnuarnav Nov 15, 2022
c28c349
merge with upstream/master
fnuarnav Nov 17, 2022
d69adb9
moved initcontainer e2e tests to a separate file
fnuarnav Nov 17, 2022
096f9b4
fix merge error
fnuarnav Nov 18, 2022
e6ca1f8
fix lint error
fnuarnav Nov 18, 2022
60e6aeb
combined initcontainer test; remove println; use index
fnuarnav Nov 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions client/aci/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,12 @@ type Container struct {
ContainerProperties `json:"properties,omitempty"`
}

// InitContainerDefinition is a initContainer instance.
type InitContainerDefinition struct {
t-ysalazar marked this conversation as resolved.
Show resolved Hide resolved
Name string `json:"name,omitempty"`
InitContainerProperties `json:"properties,omitempty"`
}

// ContainerGroup is a container group.
type ContainerGroup struct {
api.ResponseMetadata `json:"-"`
Expand All @@ -87,14 +93,15 @@ type ContainerGroup struct {
type ContainerGroupProperties struct {
ProvisioningState string `json:"provisioningState,omitempty"`
Containers []Container `json:"containers,omitempty"`
InitContainers []InitContainerDefinition `json:"initContainers,omitempty"`
ImageRegistryCredentials []ImageRegistryCredential `json:"imageRegistryCredentials,omitempty"`
RestartPolicy ContainerGroupRestartPolicy `json:"restartPolicy,omitempty"`
IPAddress *IPAddress `json:"ipAddress,omitempty"`
OsType OperatingSystemTypes `json:"osType,omitempty"`
Volumes []Volume `json:"volumes,omitempty"`
InstanceView ContainerGroupPropertiesInstanceView `json:"instanceView,omitempty"`
Diagnostics *ContainerGroupDiagnostics `json:"diagnostics,omitempty"`
SubnetIds []*SubnetIdDefinition `json:"subnetIds,omitempty"`
SubnetIds []*SubnetIdDefinition `json:"subnetIds,omitempty"`
Extensions []*Extension `json:"extensions,omitempty"`
DNSConfig *DNSConfig `json:"dnsConfig,omitempty"`
}
Expand All @@ -105,7 +112,7 @@ type ContainerGroupPropertiesInstanceView struct {
State string `json:"state,omitempty"`
}

// SubnetIdDefinition is the subnet ID, the format should be
// SubnetIdDefinition is the subnet ID, the format should be
// /subscriptions/{subscriptionID}/resourceGroups/{ResourceGroup}/providers/Microsoft.Network/virtualNetworks/{VNET}/subnets/{Subnet}
type SubnetIdDefinition struct {
ID string `json:"id,omitempty"`
Expand Down Expand Up @@ -137,6 +144,15 @@ type ContainerProperties struct {
ReadinessProbe *ContainerProbe `json:"readinessProbe,omitempty"`
}

// InitContainerProperties is the initContainer instance properties.
type InitContainerProperties struct {
Image string `json:"image,omitempty"`
Command []string `json:"command,omitempty"`
EnvironmentVariables []EnvironmentVariable `json:"environmentVariables,omitempty"`
InstanceView ContainerPropertiesInstanceView `json:"instanceView,omitempty"`
VolumeMounts []VolumeMount `json:"volumeMounts,omitempty"`
}

// ContainerPropertiesInstanceView is the instance view of the container instance. Only valid in response.
type ContainerPropertiesInstanceView struct {
RestartCount int32 `json:"restartCount,omitempty"`
Expand Down Expand Up @@ -242,7 +258,7 @@ type GPUSKU string

const (
// K80 specifies the K80 GPU SKU
K80 GPUSKU = "K80"
K80 GPUSKU = "K80"
// P100 specifies the P100 GPU SKU
P100 GPUSKU = "P100"
// V100 specifies the V100 GPU SKU
Expand Down Expand Up @@ -460,7 +476,7 @@ type ExtensionType string

// Supported extension types
const (
ExtensionTypeKubeProxy ExtensionType = "kube-proxy"
ExtensionTypeKubeProxy ExtensionType = "kube-proxy"
ExtensionTypeRealtimeMetrics ExtensionType = "realtime-metrics"
)

Expand Down
121 changes: 97 additions & 24 deletions provider/aci.go
Original file line number Diff line number Diff line change
Expand Up @@ -667,6 +667,11 @@ func (p *ACIProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
containerGroup.RestartPolicy = aci.ContainerGroupRestartPolicy(pod.Spec.RestartPolicy)
containerGroup.ContainerGroupProperties.OsType = aci.OperatingSystemTypes(p.operatingSystem)

// get initContainers
helayoty marked this conversation as resolved.
Show resolved Hide resolved
initContainers, err := p.getInitContainers(pod)
if err != nil {
return err
}
// get containers
containers, err := p.getContainers(pod)
if err != nil {
Expand All @@ -683,6 +688,7 @@ func (p *ACIProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
return err
}
// assign all the things
containerGroup.ContainerGroupProperties.InitContainers = initContainers
containerGroup.ContainerGroupProperties.Containers = containers
containerGroup.ContainerGroupProperties.Volumes = volumes
containerGroup.ContainerGroupProperties.ImageRegistryCredentials = creds
Expand Down Expand Up @@ -1443,45 +1449,112 @@ func readDockerConfigJSONSecret(secret *v1.Secret, ips []aci.ImageRegistryCreden
return ips, err
}

//verify if Container is properly declared for the use on ACI
//this method is used for both initConainers and containers
func (p *ACIProvider) verifyContainer(container *v1.Container) error {
if len(container.Command) == 0 && len(container.Args) > 0 {
return errdefs.InvalidInput("ACI does not support providing args without specifying the command. Please supply both command and args to the pod spec.")
}
return nil
}

//get the command declared on Container
//this method is used for both initConainers and containers
func (p *ACIProvider) getCommand(container *v1.Container) []string {
return append(container.Command, container.Args...)
}

//get VolumeMounts declared on Container as []aci.VolumeMount
//this method is used for both initConainers and containers
func (p *ACIProvider) getVolumeMounts(container *v1.Container) []aci.VolumeMount {
volumeMounts := make([]aci.VolumeMount, 0, len(container.VolumeMounts))
for _, v := range container.VolumeMounts {
volumeMounts = append(volumeMounts, aci.VolumeMount{
Name: v.Name,
MountPath: v.MountPath,
ReadOnly: v.ReadOnly,
})
}
return volumeMounts
}

//get EnvironmentVariables declared on Container as []aci.EnvironmentVariable
//this method is used for both initConainers and containers
func (p *ACIProvider) getEnvironmentVariables(container *v1.Container) []aci.EnvironmentVariable {
environmentVariable := make([]aci.EnvironmentVariable, 0, len(container.Env))
for _, e := range container.Env {
if e.Value != "" {
envVar := getACIEnvVar(e)
environmentVariable = append(environmentVariable, envVar)
}
}
return environmentVariable
}

//get InitContainers defined in Pod as []aci.InitContainerDefinition
func (p *ACIProvider) getInitContainers(pod *v1.Pod) ([]aci.InitContainerDefinition, error) {
initContainers := make([]aci.InitContainerDefinition, 0, len(pod.Spec.InitContainers))
for _, initContainer := range pod.Spec.InitContainers {
err := p.verifyContainer(&initContainer)
if err != nil {
return nil, err
}

if initContainer.Ports != nil {
return nil, errdefs.InvalidInput("ACI initContainers does not support ports.")
}
if initContainer.Resources.Requests != nil {
return nil, errdefs.InvalidInput("ACI initContainers does not support resources requests.")
}
if initContainer.Resources.Limits != nil {
return nil, errdefs.InvalidInput("ACI initContainers does not support resources limits.")
}
if initContainer.LivenessProbe != nil {
return nil, errdefs.InvalidInput("ACI initContainers does not support livenessProbe.")
}
if initContainer.ReadinessProbe != nil {
return nil, errdefs.InvalidInput("ACI initContainers does not support readinessProbe.")
}

newInitContainer := aci.InitContainerDefinition{
Name: initContainer.Name,
}
newInitContainer.Image = initContainer.Image
newInitContainer.Command = p.getCommand(&initContainer)

newInitContainer.VolumeMounts = p.getVolumeMounts(&initContainer)
newInitContainer.EnvironmentVariables = p.getEnvironmentVariables(&initContainer)

initContainers = append(initContainers, newInitContainer)
}
return initContainers, nil
}

//get Containers defined in Pod as []aci.Container type
func (p *ACIProvider) getContainers(pod *v1.Pod) ([]aci.Container, error) {
containers := make([]aci.Container, 0, len(pod.Spec.Containers))
for _, container := range pod.Spec.Containers {

if len(container.Command) == 0 && len(container.Args) > 0 {
return nil, errdefs.InvalidInput("ACI does not support providing args without specifying the command. Please supply both command and args to the pod spec.")
err := p.verifyContainer(&container)
if err != nil {
return nil, err
}

c := aci.Container{
Name: container.Name,
ContainerProperties: aci.ContainerProperties{
Image: container.Image,
Command: append(container.Command, container.Args...),
Ports: make([]aci.ContainerPort, 0, len(container.Ports)),
},
}
c.Image = container.Image
c.Command = p.getCommand(&container)

c.Ports = make([]aci.ContainerPort, 0, len(container.Ports))
for _, p := range container.Ports {
c.Ports = append(c.Ports, aci.ContainerPort{
Port: p.ContainerPort,
Protocol: getProtocol(p.Protocol),
})
}

c.VolumeMounts = make([]aci.VolumeMount, 0, len(container.VolumeMounts))
for _, v := range container.VolumeMounts {
c.VolumeMounts = append(c.VolumeMounts, aci.VolumeMount{
Name: v.Name,
MountPath: v.MountPath,
ReadOnly: v.ReadOnly,
})
}

c.EnvironmentVariables = make([]aci.EnvironmentVariable, 0, len(container.Env))
for _, e := range container.Env {
if e.Value != "" {
envVar := getACIEnvVar(e)
c.EnvironmentVariables = append(c.EnvironmentVariables, envVar)
}
}
c.VolumeMounts = p.getVolumeMounts(&container)
c.EnvironmentVariables = p.getEnvironmentVariables(&container)

// NOTE(robbiezhang): ACI CPU request must be times of 10m
cpuRequest := 1.00
Expand Down
Loading