AWS: CloudFormation — шаблон для RDS PostgreSQL

Автор: | 27/10/2017
Нет комментариев
 

Задача — написать шаблон для развёртывания стека, в котором будут два AWS RDS PostgreSQL инстанса (Oscar и Grover).

К инстансам необходимо обеспечить доступ из другого AWS региона, т.к. EC2 интанс с Tableau сервером находится (пока) в us-west-1 (N. California), а новые RDS сервера — в Ирландии (eu-west-1). Поэтому — подсети в VPC будут публичными, ограничение доступа будет реализовано на уровне DB Security Group (подробнее — в документации), а для шифрования трафика между Tableau и RDS будет применяться TLS/SSL.

Подготовка

IAM пользователь

Создаём пользователя, подключаем ему роли.

Одна есть в AWS IAM по умолчанию — arn:aws:iam::aws:policy/AmazonRDSFullAccess.

Вторую — создаём руками. Можно просто скопировать из существующей роли AWSCloudFormationReadOnlyAccess, и отредактировать:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:*"
            ],
            "Resource": "*"
        }
    ]
}

Третья — тоже есть в IAM, AmazonVPCFullAccess.

Именованный профиль AWS CLI

Настраиваем AWS CLI:

[simterm]

$ aws configure --profile ads-rds-migration
AWS Access Key ID [None]: AKI***56Q
AWS Secret Access Key [None]: +RT***PhW
Default region name [None]: eu-west-1
Default output format [None]: json

[/simterm]

Проверяем:

[simterm]

$ aws cloudformation --profile ads-rds-migration describe-stacks                                                                                                                                      
{                                                                                                                                                                                                                                             
    "Stacks": [                                                                                                                                                                                                                               
        {                                                                                                                                                                                                                                     
            "StackId": "arn:aws:cloudformation:eu-west-1:159524815787:stack/***",
...

[/simterm]

Шаблон CloudFormation

Параметры

Добавляем необходимый минимум параметров:

...
  "Parameters" : {

    "VPCCIDRBlock": {
      "Description": "VPC CIDR block",
      "Type": "String",
      "Default": "10.0.0.0/16"
    },

    "PublicSubnet1aCIDR": {
      "Description": "Public Subnet CIDR, zone A",
      "Type": "String",
      "Default": "10.0.1.0/28"
    },

    "PublicSubnet1bCIDR": {
      "Description": "Public Subnet CIDR, zone B",
      "Type": "String",
      "Default": "10.0.16.0/28"
    },

    "ENV": {
      "Description": "Environment type. tbl-pg-rds-dev or tbl-pg-rds-production.",
      "Type": "String",
      "Default": "tbl-pg-rds-dev"
    },

    "KievAllowLocation": {
      "Description": "The IP address range that can be used to access RDS instances",
      "Type": "String",
      "MinLength": "9",
      "MaxLength": "18",
      "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
      "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x.",
      "Default": "194.***.***.45/32"
    }

  },
...

Mappings

Добавляем список Avalability Zones, т.к. RDS требует минимум две подсети в разных AZ:

...
  "Mappings" : {
    "SubnetsAZs" : {
      "public-subnets" : {
        "PublicSubnet1a" : "eu-west-1a",
        "PublicSubnet1b" : "eu-west-1b"
      }
    }
  },
...

VPC, подсети

Начинаем с сети — VPC и подсети.

Добавляем три ресурса, в ресурсе VPC включаем EnableDnsSupport и EnableDnsHostnames для RDS интансов:

...
  "Resources" : {

    "VPC" : {
      "Type" : "AWS::EC2::VPC",
      "Properties" : {
        "CidrBlock" : {"Ref" : "VPCCIDRBlock"},
        "EnableDnsSupport" : "true",
        "EnableDnsHostnames" : "true",
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "vpc"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },

    "PublicSubnet1a" : {
      "Type" : "AWS::EC2::Subnet",
      "Properties" : {
        "VpcId" : { "Ref" : "VPC" },
        "CidrBlock" : {"Ref" : "PublicSubnet1aCIDR"},
        "AvailabilityZone" : { "Fn::FindInMap" : [ "SubnetsAZs", "public-subnets", "PublicSubnet1a" ] },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "public-net-a"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },

    "PublicSubnet1b" : {
      "Type" : "AWS::EC2::Subnet",
      "Properties" : {
        "VpcId" : { "Ref" : "VPC" },
        "CidrBlock" : {"Ref" : "PublicSubnet1bCIDR"},
        "AvailabilityZone" : { "Fn::FindInMap" : [ "SubnetsAZs", "public-subnets", "PublicSubnet1b" ] },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "public-net-b"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    }
  }
...

Проверяем шаблон:

[simterm]

$ aws cloudformation --profile ads-rds-migration validate-template --template-body file://ads-pg-rds-stack.json

[/simterm]

Всё ОК — продолжаем.

Internet Gateway

Добавляем ресурс IGW:

...
    "InternetGateway" : {
      "Type" : "AWS::EC2::InternetGateway",
      "Properties" : {
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "igw"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },
...

Добавляем подключение IGW к VPC:

...
    "AttachGateway" : {
       "Type" : "AWS::EC2::VPCGatewayAttachment",
       "Properties" : {
         "VpcId" : { "Ref" : "VPC" },
         "InternetGatewayId" : { "Ref" : "InternetGateway" }
       }
    },
...
RouteTable

Добавляем таблицу маршрутизации:

...
    "PublicRouteTable": {
      "Type": "AWS::EC2::RouteTable",
      "Properties": {
        "VpcId": { "Ref": "VPC" },
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "public-route"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },
...

Маршрут для подсети PublicSubnet в 0.0.0.0 через созданный Internet Gateway:

...
    "PublicRoute": {
      "Type": "AWS::EC2::Route",
      "Properties": {
        "RouteTableId": {
          "Ref": "PublicRouteTable"
        },
        "DestinationCidrBlock": "0.0.0.0/0",
        "GatewayId": {
          "Ref": "InternetGateway"
        }
      }
    },
...

И подключаем таблицу маршрутизации к подсетям:

...
    "PublicSubnetRouteTableAssociation1A": {
      "Type": "AWS::EC2::SubnetRouteTableAssociation",
      "Properties": {
        "SubnetId": {
          "Ref": "PublicSubnet1a"
        },
        "RouteTableId": {
          "Ref": "PublicRouteTable"
        }
      }
    },
    
    "PublicSubnetRouteTableAssociation1B": {
      "Type": "AWS::EC2::SubnetRouteTableAssociation",
      "Properties": {
        "SubnetId": {
          "Ref": "PublicSubnet1b"
        },
        "RouteTableId": {
          "Ref": "PublicRouteTable"
        }
      }
    }
...

Создаём стек:

[simterm]

$ aws cloudformation --profile ads-rds-migration validate-template --template-body file://ads-pg-rds-stack.json
$ aws cloudformation --profile ads-rds-migration create-stack --stack-name ads-pg-rds-dev --template-body file://ads-pg-rds-stack.json

[/simterm]

RDS

Parameters

Для серверов баз данных нам потребуются параметры:

  • имя пользователя
  • пароль
  • имена серверов — два штука
  • тип интансов
  • размер дисков

Добавляем в Parameters:

...
    "OscarDBInstanceName": {
      "Default": "oscar-rds-pg",
      "Description" : "Oscar DB instance name",
      "Type": "String",
      "MinLength": "1",
      "MaxLength": "15",
      "AllowedPattern" : "[a-zA-Z-][a-zA-Z0-9-]*"
    },

    "GroverDBInstanceName": {
      "Default": "grover-rds-pg",
      "Description" : "Grover DB instance name",
      "Type": "String",
      "MinLength": "1",
      "MaxLength": "15",
      "AllowedPattern" : "[a-zA-Z-][a-zA-Z0-9-]*"
    },

    "DBUsername": {
      "Default": "pgadmin",
      "NoEcho": "true",
      "Description" : "The database admin account username",
      "Type": "String",
      "MinLength": "1",
      "MaxLength": "16",
      "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
      "ConstraintDescription" : "Must begin with a letter and contain only alphanumeric characters."
    },

    "DBPassword": {
      "Default": "pass",
      "NoEcho": "true",
      "Description" : "The database admin account password",
      "Type": "String",
      "MinLength": "8"
    },

    "DBClass" : {
      "Default" : "db.t2.micro",
      "Description" : "Database instance class",
      "Type" : "String",
      "AllowedValues" : [ "db.t2.micro", "db.m1.small", "db.m1.large", "db.m1.xlarge", "db.m2.xlarge" ],
      "ConstraintDescription" : "Must select a valid database instance type."
    },

    "DBAllocatedStorage" : {
      "Default": "5",
      "Description" : "The size of the database (Gb)",
      "Type": "Number",
      "MinValue": "5",
      "MaxValue": "6144",
      "ConstraintDescription" : "Must be between 5 - 1024 GB. Use at least 1024 for Production."
    }
...
Resources

Далее — добавляем ресурсы.

Потребуются:

Для AWS::RDS::DBParameterGroup надо будет указать параметр Family.

Находим актуальную версию для PostgreSQL RDS:

[simterm]

$ aws rds --profile ads-rds-migration describe-db-engine-versions --engine postgres --default-only --output text
DBENGINEVERSIONS        PostgreSQL      PostgreSQL 9.6.3-R1     postgres9.6     postgres        9.6.3
VALIDUPGRADETARGET      False   PostgreSQL 9.6.5-R1     postgres        9.6.5   False

[/simterm]

Добавляем ресурсы:

...
    "DBSecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties" : {
        "GroupDescription" : "PostgreSQL server Access",
        "VpcId"            : { "Ref": "VPC" },
        "SecurityGroupIngress" : [
          {
            "IpProtocol" : "tcp",
            "FromPort"   : 5432,
            "ToPort"     : 5432,
            "CidrIp"     : { "Ref": "KievAllowLocation" }
          },
          {
            "IpProtocol" : "tcp",
            "FromPort"   : 5432,
            "ToPort"     : 5432,
            "CidrIp"     : { "Ref": "KievVentiAllowLocation" }
          }
        ],
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "db-sg"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },

    "DBParamGroup": {
        "Type": "AWS::RDS::DBParameterGroup",
        "Properties": {
          "Description": "Database Parameter Group + pg_stat_statements",
          "Family": "postgres9.6",
          "Tags" : [
            {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "db-params"] ] } },
            {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
          ]
        }
    },

    "DBSubnetGroup" : {
      "Type" : "AWS::RDS::DBSubnetGroup",
      "Properties" : {
        "DBSubnetGroupDescription" : "DB Private Subnet",
        "SubnetIds" : [
          { "Ref": "PublicSubnet1a" },
          { "Ref": "PublicSubnet1b" }
        ],
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "db-subnet-group"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },

    "OscarRDSDB" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        "DBInstanceIdentifier" : { "Ref" : "OscarDBInstanceName" },
        "AllocatedStorage" : { "Ref" : "DBAllocatedStorage" },
        "DBInstanceClass" : { "Ref" : "DBClass" },
        "Engine" : "postgres",
        "MasterUsername" : { "Ref" : "DBUsername" } ,
        "MasterUserPassword" : { "Ref" : "DBPassword" },
        "DBSubnetGroupName" : { "Ref" : "DBSubnetGroup" },
        "DBParameterGroupName" : {"Ref" : "DBParamGroup" },
        "VPCSecurityGroups" : [ { "Fn::GetAtt" : [ "DBSecurityGroup", "GroupId" ] } ],
        "MultiAZ": "true",
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "oscar-rds-pg"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    },

    "GroverRDSDB" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        "DBInstanceIdentifier" : { "Ref" : "GroverDBInstanceName" },
        "AllocatedStorage" : { "Ref" : "DBAllocatedStorage" },
        "DBInstanceClass" : { "Ref" : "DBClass" },
        "Engine" : "postgres",
        "MasterUsername" : { "Ref" : "DBUsername" } ,
        "MasterUserPassword" : { "Ref" : "DBPassword" },
        "DBSubnetGroupName" : { "Ref" : "DBSubnetGroup" },
        "DBParameterGroupName" : {"Ref" : "DBParamGroup" },
        "VPCSecurityGroups" : [ { "Fn::GetAtt" : [ "DBSecurityGroup", "GroupId" ] } ],
        "MultiAZ": "true",
        "Tags" : [
          {"Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ {"Ref" : "AWS::StackName"}, "grover-rds-pg"] ] } },
          {"Key" : "Env", "Value" : {"Ref" : "ENV"} }
        ]
      }
    }
...

Сохраняем, выполняем validate-template и stack-update (или stack-create):

[simterm]

$ aws cloudformation --profile ads-rds-migration validate-template --template-body file://ads-pg-rds-stack.json
$ aws cloudformation --profile ads-rds-migration update-stack --stack-name ads-pg-rds-dev --template-body file://ads-pg-rds-stack.json

[/simterm]

Ждём обновления — проверяем:

[simterm]

$ aws rds --profile ads-rds-migration describe-db-instances | grep 'oscar-\|grover-'
            "DBInstanceIdentifier": "grover-rds-pg",
                "Address": "grover-rds-pg.***.eu-west-1.rds.amazonaws.com",
            "DBInstanceArn": "arn:aws:rds:eu-west-1:159524815787:db:grover-rds-pg"
            "DBInstanceIdentifier": "oscar-rds-pg",
                "Address": "oscar-rds-pg.***.eu-west-1.rds.amazonaws.com",
            "DBInstanceArn": "arn:aws:rds:eu-west-1:159524815787:db:oscar-rds-pg"

[/simterm]

Outputs

Последнее, что надо добавить — Outputs:

...
  "Outputs" : {
    "OscarEndpoint": {
      "Description" : "Oscar PostgreSQL servers endpoint",
      "Value" : { "Fn::Join": [ "", [
                                { "Fn::GetAtt": [ "OscarRDSDB", "Endpoint.Address" ] },
                                ":",
                                { "Fn::GetAtt": [ "OscarRDSDB", "Endpoint.Port" ] }
                              ]]}
    },
    "GroverEndpoint": {
      "Description" : "Grover PostgreSQL servers endpoint",
      "Value" : { "Fn::Join": [ "", [
                                { "Fn::GetAtt": [ "GroverRDSDB", "Endpoint.Address" ] },
                                ":",
                                { "Fn::GetAtt": [ "GroverRDSDB", "Endpoint.Port" ] }
                              ]]}
    }
  }
...

Обновляем стек, теперь с тегом Env (для Resource Group):

$ aws cloudformation --profile ads-rds-migration update-stack --stack-name ads-pg-rds-dev --tags Key=Env,Value=tbl-pg-rds-dev --template-body file://ads-pg-rds-stack.json

Проверяем:

Группа ресурсов:

Готово.