Skip to content

Commit

Permalink
Merge pull request #20 from dasmeta/DMVP-6335-rds-aurora-integration
Browse files Browse the repository at this point in the history
feat(DMVP-6335): ability to create rds aurora clusters and configure rds-proxy
  • Loading branch information
mrdntgrn authored Dec 24, 2024
2 parents 46fe273 + aea00b3 commit a07263f
Show file tree
Hide file tree
Showing 17 changed files with 727 additions and 127 deletions.
85 changes: 46 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,28 @@
# please enable git hooks by running the following command
```sh
git config --global core.hooksPath ./githooks # enables git hooks globally
```
# How to use
## This terraform module allows to create aws rds cluster by using various engine types and configurations, it allows also to enable/create rds cluster attached rds proxy

Case 1. Create Security group and create RDS
## module upgrade guide
- from <1.4.0 versions to >=1.4.0 version upgrade
- make sure you moved the state of "db" underlying module by using command like following
```sh
terraform state mv module.<rds-module-name>.module.db module.<rds-module-name>.module.db[0]
```
- if you had no storage_type set explicitly then set it to "gp2"

```


## How to use (more examples/tests can be found in [./tests](./tests) folder)

### Case 1. Create Security group and create RDS

```terraform
data "aws_vpc" "main" {
id = "vpc-04c3b2abe39cd8a6a"
id = "vpc-xxxxxxx"
}
module "rds" {
source = "dasmeta/modules/aws//modules/rds"
source = "dasmeta/rds/aws"
version = "1.4.0"
allocated_storage = 20
storage_type = "gp2"
engine = "mysql"
Expand All @@ -24,33 +34,17 @@ module "rds" {
db_password = "some-password"
parameter_group_name = "default.mysql5.7"
vpc_id = "${data.aws_vpc.main.id}"
subnet_ids = ["subnet-04ad8ad2fdec889ec","subnet-0ea0a01c1bea0a0c9"]
create_security_group = true
ingress_with_cidr_blocks = [
{
description = "3306 from VPC"
from_port = 3306
to_port = 3306
protocol = "tcp"
cidr_blocks = "${data.aws_vpc.main.cidr_block}"
}]
egress_with_cidr_blocks = [
{
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks ="[0.0.0.0/0]"
}]
subnet_ids = ["subnet-xxxxxxxx","subnet-xxxxxx"]
}
```

Case 2. Create RDS
### Case 2. Create RDS and pass custom/external created security group ids

```
```terraform
module "rds" {
source = "dasmeta/modules/aws//modules/rds"
source = "dasmeta/rds/aws"
version = "1.4.0"
allocated_storage = 20
storage_type = "gp2"
engine = "mysql"
Expand All @@ -62,14 +56,20 @@ module "rds" {
db_password = "some-password"
parameter_group_name = "default.mysql5.7"
vpc_id = "vpc-04c3b2abe39cd8a6a"
subnet_ids = ["subnet-04ad8ad2fdec889ec","subnet-0ea0a01c1bea0a0c9"]
vpc_id = "vpc-xxxxxxxxxxxx"
subnet_ids = ["subnet-xxxxxxx","subnet-xxxxxxxx"]
create_security_group = false
// vpc_security_group_ids = ["sg-062742ac7a7f8c7a7"]
vpc_security_group_ids = ["sg-xxxxxxxxx"]
}
```

## contribution
### please enable git hooks by running the following command
```sh
git config --global core.hooksPath ./githooks # enables git hooks globally
```

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements

Expand All @@ -87,15 +87,18 @@ No requirements.
|------|--------|---------|
| <a name="module_cloudwatch_metric_filters"></a> [cloudwatch\_metric\_filters](#module\_cloudwatch\_metric\_filters) | dasmeta/monitoring/aws//modules/cloudwatch-log-based-metrics | 1.13.2 |
| <a name="module_cw_alerts"></a> [cw\_alerts](#module\_cw\_alerts) | dasmeta/monitoring/aws//modules/alerts | 1.3.5 |
| <a name="module_db"></a> [db](#module\_db) | terraform-aws-modules/rds/aws | ~> 6.1 |
| <a name="module_security_group"></a> [security\_group](#module\_security\_group) | terraform-aws-modules/security-group/aws | 4.7.0 |
| <a name="module_db"></a> [db](#module\_db) | terraform-aws-modules/rds/aws | 6.10.0 |
| <a name="module_db_aurora"></a> [db\_aurora](#module\_db\_aurora) | terraform-aws-modules/rds-aurora/aws | 9.11.0 |
| <a name="module_proxy"></a> [proxy](#module\_proxy) | ./modules/proxy | n/a |
| <a name="module_security_group"></a> [security\_group](#module\_security\_group) | terraform-aws-modules/security-group/aws | 5.2.0 |

## Resources

| Name | Type |
|------|------|
| [aws_db_instance.database](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/db_instance) | data source |
| [aws_ec2_instance_type.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type) | data source |
| [aws_vpc.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source |

## Inputs

Expand All @@ -104,6 +107,7 @@ No requirements.
| <a name="input_alarms"></a> [alarms](#input\_alarms) | n/a | <pre>object({<br> enabled = optional(bool, true)<br> sns_topic = string<br> custom_values = optional(any, {})<br> })</pre> | n/a | yes |
| <a name="input_allocated_storage"></a> [allocated\_storage](#input\_allocated\_storage) | The allocated storage in gigabytes | `number` | `20` | no |
| <a name="input_apply_immediately"></a> [apply\_immediately](#input\_apply\_immediately) | Specifies whether any database modifications are applied immediately, or during the next maintenance window | `bool` | `false` | no |
| <a name="input_aurora_configs"></a> [aurora\_configs](#input\_aurora\_configs) | The aws rd aurora specific configurations | <pre>object({<br> engine_mode = optional(string, "provisioned") # The database engine mode. Valid values: `global`, `multimaster`, `parallelquery`, `provisioned`, `serverless`(serverless is deprecated)<br> autoscaling_enabled = optional(bool, false) # Whether autoscaling enabled<br> autoscaling_min_capacity = optional(number, 0) # Min number of read replicas<br> autoscaling_max_capacity = optional(number, 2) # Max number of read replicas permitted<br> instances = optional(any, {}) # Cluster instances configs<br> serverlessv2_scaling_configuration = optional(any, {}) # for enabling serverless-2(the serverless-1(engine_mode=serverless, scaling_configuration is set) is deprecated), valid when `engine_mode` is set to `provisioned`<br> })</pre> | `{}` | no |
| <a name="input_backup_retention_period"></a> [backup\_retention\_period](#input\_backup\_retention\_period) | The days to retain backups for | `number` | `35` | no |
| <a name="input_backup_window"></a> [backup\_window](#input\_backup\_window) | The daily time range (in UTC) during which automated backups are created if they are enabled. Example: '09:46-10:16'. Must not overlap with maintenance\_window | `string` | `"03:00-06:00"` | no |
| <a name="input_cloudwatch_log_group_retention_in_days"></a> [cloudwatch\_log\_group\_retention\_in\_days](#input\_cloudwatch\_log\_group\_retention\_in\_days) | The number of days to retain CloudWatch logs for the DB instance | `number` | `30` | no |
Expand All @@ -112,7 +116,7 @@ No requirements.
| <a name="input_create_db_parameter_group"></a> [create\_db\_parameter\_group](#input\_create\_db\_parameter\_group) | Whether to create a database parameter group | `bool` | `false` | no |
| <a name="input_create_db_subnet_group"></a> [create\_db\_subnet\_group](#input\_create\_db\_subnet\_group) | Whether to create a database subnet group | `bool` | `true` | no |
| <a name="input_create_monitoring_role"></a> [create\_monitoring\_role](#input\_create\_monitoring\_role) | Create IAM role with a defined name that permits RDS to send enhanced monitoring metrics to CloudWatch Logs | `bool` | `false` | no |
| <a name="input_create_security_group"></a> [create\_security\_group](#input\_create\_security\_group) | n/a | `bool` | `false` | no |
| <a name="input_create_security_group"></a> [create\_security\_group](#input\_create\_security\_group) | Whether to create security group and attach ingress/egress rules which will be used for rds instances(and rds proxy if we enabled it), if you already have one and do not want to create new security group you can explicitly set this variable to false and pass group id by using var.vpc\_security\_group\_ids | `bool` | `true` | no |
| <a name="input_db_instance_tags"></a> [db\_instance\_tags](#input\_db\_instance\_tags) | Additional tags for the DB instance | `map(any)` | `{}` | no |
| <a name="input_db_name"></a> [db\_name](#input\_db\_name) | The DB name to create. If omitted, no database is created initially | `string` | n/a | yes |
| <a name="input_db_option_group_tags"></a> [db\_option\_group\_tags](#input\_db\_option\_group\_tags) | Additional tags for the DB option group | `map(any)` | `{}` | no |
Expand Down Expand Up @@ -140,14 +144,17 @@ No requirements.
| <a name="input_multi_az"></a> [multi\_az](#input\_multi\_az) | Specifies if the RDS instance is multi-AZ | `bool` | `true` | no |
| <a name="input_options"></a> [options](#input\_options) | A list of Options to apply | `list(any)` | <pre>[<br> {<br> "option_name": "MARIADB_AUDIT_PLUGIN",<br> "option_settings": [<br> {<br> "name": "SERVER_AUDIT_EVENTS",<br> "value": "CONNECT"<br> },<br> {<br> "name": "SERVER_AUDIT_FILE_ROTATIONS",<br> "value": "37"<br> }<br> ]<br> }<br>]</pre> | no |
| <a name="input_parameter_group_name"></a> [parameter\_group\_name](#input\_parameter\_group\_name) | Name of the DB parameter group to associate or create | `string` | `"default.mysql5.7"` | no |
| <a name="input_parameters"></a> [parameters](#input\_parameters) | A list of DB parameters (map) to apply | `list(map(any))` | `[]` | no |
| <a name="input_parameters"></a> [parameters](#input\_parameters) | A list of DB parameters (map) to apply | <pre>list(object({<br> name = string<br> value = string<br> context = optional(string, "instance") # The context where parameter will be used, supported values are "instance" and "cluster"<br> apply_method = optional(string, "immediate") # The apply method for parameter, supported values are "immediate" and "pending-reboot"<br> }))</pre> | `[]` | no |
| <a name="input_port"></a> [port](#input\_port) | The port on which the DB accepts connections | `number` | `null` | no |
| <a name="input_proxy"></a> [proxy](#input\_proxy) | The aws rds proxy specific configurations | <pre>object({<br> enabled = optional(bool, false) # whether rds proxy is enabled<br> endpoints = optional(any, {}) # map of {<name>: <configs>} additional proxy endpoints(by default we have already one read/write endpoint), for more info check resource doc https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_proxy_endpoint<br> client_auth_type = optional(string, "MYSQL_NATIVE_PASSWORD") # The type of authentication the proxy uses for connections from clients<br> iam_auth = optional(string, "DISABLED") # Whether IAM auth enabled<br> target_db_cluster = optional(bool, true) # Whether the target db is cluster<br> debug_logging = optional(bool, false) # Whether enhanced logging is enabled<br> idle_client_timeout = optional(number, 1800) # The timeout of idle connections, default is 30 minutes<br> })</pre> | `{}` | no |
| <a name="input_publicly_accessible"></a> [publicly\_accessible](#input\_publicly\_accessible) | Whether the database is accessible publicly. Note that if you need to enable this you have to place db on public subnets | `bool` | `false` | no |
| <a name="input_security_group_description"></a> [security\_group\_description](#input\_security\_group\_description) | n/a | `string` | `"MySQL security group"` | no |
| <a name="input_security_group_name"></a> [security\_group\_name](#input\_security\_group\_name) | n/a | `string` | `"db_security_group"` | no |
| <a name="input_set_vpc_security_group_rules"></a> [set\_vpc\_security\_group\_rules](#input\_set\_vpc\_security\_group\_rules) | Whether to automatically add security group rules allowing access to db from vpc network | `bool` | `true` | no |
| <a name="input_skip_final_snapshot"></a> [skip\_final\_snapshot](#input\_skip\_final\_snapshot) | Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted | `bool` | `false` | no |
| <a name="input_slow_queries"></a> [slow\_queries](#input\_slow\_queries) | n/a | <pre>object({<br> enabled = optional(bool, true)<br> query_duration = optional(number, 3)<br> })</pre> | <pre>{<br> "enabled": true,<br> "query_duration": 3<br>}</pre> | no |
| <a name="input_storage_encrypted"></a> [storage\_encrypted](#input\_storage\_encrypted) | Specifies whether the DB instance is encrypted | `bool` | `false` | no |
| <a name="input_storage_type"></a> [storage\_type](#input\_storage\_type) | One of 'standard' (magnetic), 'gp2' (general purpose SSD), or 'io1' (provisioned IOPS SSD). The default is 'io1' if iops is specified, 'gp2' if not | `string` | `"gp2"` | no |
| <a name="input_storage_type"></a> [storage\_type](#input\_storage\_type) | One of 'standard' (magnetic), 'gp2' (general purpose SSD), or 'io1' (provisioned IOPS SSD). The default is 'io1' if iops is specified, 'gp2' if not | `string` | `null` | no |
| <a name="input_subnet_ids"></a> [subnet\_ids](#input\_subnet\_ids) | A list of VPC subnet IDs | `list(string)` | n/a | yes |
| <a name="input_tags"></a> [tags](#input\_tags) | A mapping of tags to assign to all resources | `map(any)` | `{}` | no |
| <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | n/a | `string` | `""` | no |
Expand Down
19 changes: 1 addition & 18 deletions alerts.tf
Original file line number Diff line number Diff line change
@@ -1,20 +1,3 @@
data "aws_ec2_instance_type" "this" {
instance_type = trim(var.instance_class, "db.")
}

data "aws_db_instance" "database" {
db_instance_identifier = var.identifier

depends_on = [
module.db
]
}

locals {
// SampleCount statistic adds 2 to the real count in case the engine is postgres, so 7 means 5 + 2
slow_queries_alert_threshold = var.engine == "postgres" ? 7 : 5
}

module "cw_alerts" {
count = var.alarms.enabled ? 1 : 0

Expand Down Expand Up @@ -96,7 +79,7 @@ module "cw_alerts" {
DBInstanceIdentifier = var.identifier
}
period = try(var.alarms.custom_values.disk.period, "300")
threshold = try(var.alarms.custom_values.disk.threshold, data.aws_db_instance.database.allocated_storage * 0.08 * 1024 * 1024 * 1024) #8% of storage in Bytes
threshold = try(var.alarms.custom_values.disk.threshold, data.aws_db_instance.database[0].allocated_storage * 0.08 * 1024 * 1024 * 1024) #8% of storage in Bytes
equation = try(var.alarms.custom_values.disk.equation, "lte")
statistic = try(var.alarms.custom_values.disk.statistic, "avg")
},
Expand Down
20 changes: 20 additions & 0 deletions data.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
data "aws_ec2_instance_type" "this" {
instance_type = trim(var.instance_class, "db.")
}

data "aws_db_instance" "database" {
db_instance_identifier = var.identifier

count = var.alarms.enabled ? 1 : 0

depends_on = [
module.db,
module.db_aurora
]
}

data "aws_vpc" "this" {
count = var.create_security_group ? 1 : 0

id = var.vpc_id
}
98 changes: 98 additions & 0 deletions locals.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
locals {
vpc_security_group_ids = var.create_security_group ? [module.security_group[0].security_group_id] : var.vpc_security_group_ids
enabled_cloudwatch_logs_exports = ((var.engine == "mysql" || var.engine == "mariadb") && var.slow_queries.enabled) ? ["slowquery"] : (var.engine == "postgres" && var.slow_queries.enabled) ? ["postgresql"] : var.enabled_cloudwatch_logs_exports
# Cloudwatch log groups from which log based metrics are created in case slow queries are enabled
cloudwatch_log_groups = var.slow_queries.enabled ? { for type in local.enabled_cloudwatch_logs_exports : type => "/aws/rds/instance/${var.identifier}/${type}" } : {}
create_db_parameter_group = var.slow_queries.enabled ? true : var.create_db_parameter_group
parameter_group_name = local.create_db_parameter_group ? "${var.identifier}-${var.engine}" : null
postgres_slow_queries_duration = var.slow_queries.query_duration * 1000
port = (endswith(var.engine, "mysql") || endswith(var.engine, "mariadb")) ? 3306 : endswith(var.engine, "postgres") ? 5432 : var.port
default_params_mysql = [
{
name = "slow_query_log"
value = "1"
},
{
name = "log_output"
value = "FILE"
},
{
name = "long_query_time"
value = var.slow_queries.query_duration
},
]
default_params_postgres = [
{
name = "log_min_duration_statement" //This setting causes PostgreSQL to log any query that takes longer than `local.slow_queries_duration` seconds to execute. It includes both the query text and its duration.
value = local.postgres_slow_queries_duration
},
{
name = "log_statement" //This setting prevents the logging of every single SQL statement and logs those ones which correspond to parameter group's configuration.
value = "none"
},
{
name = "log_duration" //When enabled, this logs the duration of every completed statement.
value = "1"
},
]

# Maps from the default parameters for easier merging
params_mysql = { for p in local.default_params_mysql : p.name => p.value }
params_postgres = { for p in local.default_params_postgres : p.name => p.value }

# Create a map from the user parameters
user_params_map = { for p in var.parameters : p.name => p.value if p.context == "instance" }
cluster_params_map = [for p in var.parameters : p if p.context == "cluster"]

# Merge the two maps, with user parameters overriding defaults
merged_params_map = merge(
((var.engine == "mysql" || var.engine == "mariadb") && var.slow_queries.enabled) ? local.params_mysql : {},
(var.engine == "postgres" && var.slow_queries.enabled) ? local.params_postgres : {},
local.user_params_map
)

# Convert the merged map back to a list of maps
combined_parameters = [for name, value in local.merged_params_map : { name = name, value = value }]
is_aurora = startswith(var.engine, "aurora")
engine_family = (endswith(var.engine, "mysql") || endswith(var.engine, "mariadb")) ? "MYSQL" : (endswith(var.engine, "postgres") ? "POSTGRESQL" : "")

// SampleCount statistic adds 2 to the real count in case the engine is postgres, so 7 means 5 + 2
slow_queries_alert_threshold = var.engine == "postgres" ? 7 : 5

ingress_with_cidr_blocks = concat(
var.ingress_with_cidr_blocks,
var.create_security_group && var.set_vpc_security_group_rules ? [ # make cluster available within vpc private network
{
description = "${local.port} from VPC"
from_port = local.port
to_port = local.port
protocol = "tcp"
cidr_blocks = data.aws_vpc.this[0].cidr_block
}
] : [],
var.create_security_group && var.publicly_accessible ? [ # expose rds to public, NOTE: you need also to place instances on public subnets
{
description = "Accessible from everywhere"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = "0.0.0.0/0"
}
] : []
)

egress_with_cidr_blocks = concat(
var.egress_with_cidr_blocks,
var.create_security_group && var.set_vpc_security_group_rules && var.proxy.enabled ? [ # this egress rule needed for rds proxy
{
description = "${local.port} to VPC"
from_port = local.port
to_port = local.port
protocol = "tcp"
cidr_blocks = data.aws_vpc.this[0].cidr_block
}
] : [],
)

credentials_secret_arn = try(module.db[0].db_instance_master_user_secret_arn, module.db_aurora[0].cluster_master_user_secret.secret_arn, null)
}
Loading

0 comments on commit a07263f

Please sign in to comment.