[Development][Add] Added Index template support (#13)

* Updated lifecycle CRD Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated CRD for index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated CRD for index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated CRD for index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated Documentation Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated Documentation Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Added function to generate index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Added elastic connection details Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Added code for index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Added controller for index template Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated helm charts for latest code Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com> * Updated Documentation for latest code Signed-off-by: iamabhishek-dubey <abhishekbhardwaj510@gmail.com>
OT-CONTAINER-KIT · Sep 7, 2020 · 61e5121 · 61e5121
1 parent d31f564
commit 61e5121
Show file tree

Hide file tree

Showing 13 changed files with 375 additions and 98 deletions.
diff --git a/README.md b/README.md
@@ -34,24 +34,28 @@ A golang based CRD operator to setup and manage logging stack (Elasticsearch, Fl
 
 > The K8s API name is "logging.opstreelabs.in/v1alpha1"
 
-Our roadmap is present in [ROADMAP](ROADMAP.md)
+### Documentation
+
+[Documentation](https://docs.opstreelabs.in/logging-operator)
 
 ### Supported Features
 
 The "Logging Operator" includes these features:-
 
 - Elasticsearch different node types, like:-
-  - Master Node
-  - Data Node
-  - Ingestion Node
-  - Client/Coordinator Node
-- Elasticsearch setup with/without TLS
-- Customizable elasticsearch configuration and Heap size
-- Fluentd as a log-shipper which already has JSON logs support
+  - **Master Node** => A node that has the master role (default), which makes it eligible to be elected as the master node, which controls the cluster.
+  - **Data Node** => A node that has the data role (default). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.
+  - **Ingestion Node** => A node that has the ingest role (default). Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to not include the ingest role from nodes that have the master or data roles.
+  - **Client or Coordinator Node** => Requests like search requests or bulk-indexing requests may involve data held on different data nodes. A search request, for example, is executed in two phases which are coordinated by the node which receives the client request — the coordinating node.
+- Elasticsearch setup with or without TLS on Transport and HTTP Layer
+- Customizable elasticsearch configuration and configurable heap size
+- Fluentd as a lightweight log-shipper and JSON field seperation support
 - Kibana integration with elasticsearch for logs visualization
-- Seamless upgrade for Elasticsearch, Fluentd, and Kibana
+- Seamless upgrade for Elasticsearch, Fluentd, and Kibana stack
 - Inculcated best practices for Kubernetes setup like `SecurityContext` and `Privilege Control`
-- Loosely coupled setup, i.e. Elasticsearch, Fluentd, and Kibana can be setup individually as well.
+- Loosely coupled setup, i.e. Elasticsearch, Fluentd, and Kibana setup can be done individually as well
+- Index Lifecycle support to manage rollover and cleanup of indexes
+- Index template support for configuring index settings like:- policy, replicas, shards etc.
 
 ### Architecture
 
@@ -80,55 +84,7 @@ For the "Logging Operator" installation, we have categorized the steps in 3 part
 - RBAC setup for an operator to create resources in Kubernetes
 - Operator deployment and validation
 
-#### Namespace setup
-
-Since we are going to use pre-baked manifests of Kubernetes in that case we need to setup the namespace with a specific name called "logging-operator".
-
-```shell
-kubectl create ns logging-operator
-```
-
-#### CRD Setup
-
-So we have already pre-configured CRD in [config/crd](./config/crd) directory. We just have to run a magical `kubectl` commands.
-
-```shell
-kubectl apply -f config/crd/
-```
-
-#### RBAC setup
-
-Similar like CRD, we have pre-baked RBAC config files as well inside [config/crd](./config/rbac) which can be installed and configured by `kubectl`
-
-```shell
-kubectl apply -f config/rbac/
-```
-
-#### Operator Deployment and Validation
-
-Once all the initial steps are done, we can create the deployment for "Logging Operator". The deployment manifests for operator is present inside [config/manager/manager.yaml](./config/manager/manager.yaml) file.
-
-```shell
-kubectl apply -f config/manager/manager.yaml
-```
-
-### Deployment Using Helm
-
-For quick deployment, we have pre-baked helm charts for logging operator deployment and logging stack setup. In case you don't want to customize the manifests file and want to deploy the cluster with some minimal configuration change, in that case, "Helm" can be used.
-
-```shell
-helm upgrade logging-operator ./helm-charts/logging-operator/ \
-  -f ./helm-charts/logging-operator/values.yaml --namespace logging-operator --install
-```
-
-Once the logging operator setup is completed, we can create the logging stack for our requirement.
-
-```shell
-helm upgrade logging-stack ./helm-charts/logging-setup/ \
-  -f ./helm-charts/logging-setup/values.yaml --set elasticsearch.master.replicas=3 \
-  --set elasticsearch.data.replicas=3 --set elasticsearch.ingestion.replicas=1 \
-  --set elasticsearch.client.replicas=1 --namespace logging-operator --install
-```
+The detailed installation steps are present in [Documentation Guide](https://docs.opstreelabs.in/logging-operator)
 
 ### Examples
 

diff --git a/api/v1alpha1/indextemplate_types.go b/api/v1alpha1/indextemplate_types.go
@@ -22,14 +22,24 @@ import (
 
 // IndexTemplateSpec defines the desired state of IndexTemplate
 type IndexTemplateSpec struct {
-	Rollover Rollover `json:"rollover,omitempty"`
-	Delete   Delete   `json:"delete,omitempty"`
+	Enabled               *bool                   `json:"enabled,omitempty"`
+	IndexPatterns         []string                `json:"indexPatterns,omitempty"`
+	IndexTemplateSettings IndexTemplateSettings   `json:"settings,omitempty"`
+	Elasticsearch         ManagementElasticsearch `json:"elasticsearch,omitempty"`
+}
+
+// IndexTemplateSettings defines the desired state for settings of index
+type IndexTemplateSettings struct {
+	Shards             int32  `json:"shards,omitempty"`
+	Replicas           int32  `json:"replicas,omitempty"`
+	IndexLifecycleName string `json:"indexLifeCycleName,omitempty"`
+	RollOverAlias      string `json:"rolloverAlias,omitempty"`
 }
 
 // IndexTemplateStatus defines the observed state of IndexTemplate
 type IndexTemplateStatus struct {
-	Rollover Rollover `json:"rollover,omitempty"`
-	Delete   Delete   `json:"delete,omitempty"`
+	IndexPatterns         []string              `json:"indexPatterns,omitempty"`
+	IndexTemplateSettings IndexTemplateSettings `json:"settings,omitempty"`
 }
 
 // +kubebuilder:object:root=true

diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
diff --git a/config/crd/bases/logging.opstreelabs.in_indexlifecycles.yaml b/config/crd/bases/logging.opstreelabs.in_indexlifecycles.yaml
@@ -4,7 +4,7 @@ apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.4.0
+    controller-gen.kubebuilder.io/version: v0.3.0
   creationTimestamp: null
   name: indexlifecycles.logging.opstreelabs.in
 spec:
@@ -83,6 +83,16 @@ spec:
                 type: object
             type: object
         type: object
+    additionalPrinterColumns:
+    - jsonPath: .spec.rollover.maxSize
+      name: Rollover Size
+      type: string
+    - jsonPath: .spec.rollover.maxAge
+      name: Rollover Age
+      type: string
+    - jsonPath: .spec.delete.minAge
+      name: Deletion Age
+      type: string
     served: true
     storage: true
     subresources:

diff --git a/config/crd/bases/logging.opstreelabs.in_indextemplates.yaml b/config/crd/bases/logging.opstreelabs.in_indextemplates.yaml
@@ -4,7 +4,7 @@ apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
-    controller-gen.kubebuilder.io/version: v0.4.0
+    controller-gen.kubebuilder.io/version: v0.3.0
   creationTimestamp: null
   name: indextemplates.logging.opstreelabs.in
 spec:
@@ -36,40 +36,75 @@ spec:
           spec:
             description: IndexTemplateSpec defines the desired state of IndexTemplate
             properties:
-              delete:
-                description: Delete is the struct for index deletion
+              elasticsearch:
+                description: ManagementElasticsearch is the struct for elasticsearch
+                  configuration for fluentd
                 properties:
-                  minAge:
+                  host:
+                    type: string
+                  password:
+                    type: string
+                  username:
                     type: string
                 type: object
-              rollover:
-                description: Rollover is the struct for index roll overing
+              enabled:
+                type: boolean
+              indexPatterns:
+                items:
+                  type: string
+                type: array
+              settings:
+                description: IndexTemplateSettings defines the desired state for settings
+                  of index
                 properties:
-                  maxAge:
+                  indexLifeCycleName:
                     type: string
-                  maxSize:
+                  replicas:
+                    format: int32
+                    type: integer
+                  rolloverAlias:
                     type: string
+                  shards:
+                    format: int32
+                    type: integer
                 type: object
             type: object
           status:
             description: IndexTemplateStatus defines the observed state of IndexTemplate
             properties:
-              delete:
-                description: Delete is the struct for index deletion
-                properties:
-                  minAge:
-                    type: string
-                type: object
-              rollover:
-                description: Rollover is the struct for index roll overing
+              indexPatterns:
+                items:
+                  type: string
+                type: array
+              settings:
+                description: IndexTemplateSettings defines the desired state for settings
+                  of index
                 properties:
-                  maxAge:
+                  indexLifeCycleName:
                     type: string
-                  maxSize:
+                  replicas:
+                    format: int32
+                    type: integer
+                  rolloverAlias:
                     type: string
+                  shards:
+                    format: int32
+                    type: integer
                 type: object
             type: object
         type: object
+    additionalPrinterColumns:
+    - jsonPath: .spec.settings.shards
+      name: Shards
+      type: integer
+      format: int32
+    - jsonPath: .spec.settings.replicas
+      name: Replicas
+      type: integer
+      format: int32
+    - jsonPath: .spec.settings.indexLifeCycleName
+      name: Lifecycle Policy
+      type: string
     served: true
     storage: true
     subresources:

diff --git a/config/samples/index-template-example.yaml b/config/samples/index-template-example.yaml
@@ -0,0 +1,17 @@
+---
+apiVersion: logging.opstreelabs.in/v1alpha1
+kind: IndexTemplate
+metadata:
+  name: indextemplate-sample
+spec:
+  enabled: true
+  indexPatterns: ["kubernetes-default-*", "kubernetes-istio-system-*"]
+  settings:
+    shards: 1
+    replicas: 1
+    indexLifeCycleName: indexlifecycle-example
+    rolloverAlias: "kubernetes"
+  elasticsearch:
+    host: "https://elasticsearch-cluster-ingestion:9200"
+    username: elastic
+    password: "Opstree@1234"
diff --git a/config/samples/lifecycle-example.yaml b/config/samples/lifecycle-example.yaml
@@ -1,3 +1,4 @@
+---
 apiVersion: logging.opstreelabs.in/v1alpha1
 kind: IndexLifecycle
 metadata:

diff --git a/config/samples/logging_v1alpha1_indextemplate.yaml b/config/samples/logging_v1alpha1_indextemplate.yaml
diff --git a/controllers/indextemplate_controller.go b/controllers/indextemplate_controller.go
@@ -18,9 +18,14 @@ package controllers
 
 import (
 	"context"
+	"k8s.io/apimachinery/pkg/api/errors"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+	"time"
 
 	"github.com/go-logr/logr"
 	"k8s.io/apimachinery/pkg/runtime"
+
+	elasticutils "logging-operator/utils/elasticsearch"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
@@ -39,9 +44,28 @@ type IndexTemplateReconciler struct {
 
 func (r *IndexTemplateReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 	_ = context.Background()
-	_ = r.Log.WithValues("indextemplate", req.NamespacedName)
+	reqLogger := r.Log.WithValues("index-template", req.NamespacedName)
+
+	instance := &loggingv1alpha1.IndexTemplate{}
+	err := r.Get(context.TODO(), req.NamespacedName, instance)
+	if err != nil {
+		if errors.IsNotFound(err) {
+			return ctrl.Result{RequeueAfter: time.Second * 10}, nil
+		}
+		return ctrl.Result{RequeueAfter: time.Second * 10}, err
+	}
+
+	if err := controllerutil.SetControllerReference(instance, instance, r.Scheme); err != nil {
+		return ctrl.Result{RequeueAfter: time.Second * 10}, err
+	}
+
+	if instance.Spec.Enabled != nil && *instance.Spec.Enabled != false {
+		elasticutils.CompareandUpdateIndexTemplate(instance)
+	} else {
+		elasticutils.DeleteIndexTemplate(instance)
+	}
 
-	// your logic here
+	reqLogger.Info("Will reconcile after 10 seconds", "IndexTemplate.Namespace", instance.Namespace, "IndexTemplate.Name", instance.Name)
 
 	return ctrl.Result{}, nil
 }