22 Feb Multi-Account AWS Terraform Setup with Encrypted Remote State on S3 Backend
Wow… that’s a mouthful, isn’t it? Well, here is what we wanted to do, in plain English:
For AWS billing and restricted environments security access purposes, we keep the different AWS-Hosted Restcomm environments in different sub-accounts, using AWS Organizations. We do this so that we know what each environment is costing us (or at least as close as Amazon allows us to know… AWS billing reports are notoriously bad at this) and restrict specific team members or automated CI environments to given AWS environments to avoid unauthorized access to them.
In any case, we are now in the process of migrating all this infrastructure, to code, using Terraform.
This should explain the “Multi-Account AWS Terraform Setup” part of the title. Now, let’s talk about remote state.
One of the reasons we picked Terraform is because, as a tool, it has been specifically designed to solve the problem of mapping cloud infrastructure to code. It was not built to do Configuration Management and then extended to also handle cloud resources, like some other solutions in the space (e.g. see Chef, Ansible).
In particular, one of the problems it solves really well is the support for a team of infrastructure engineers to manage the same set of Cloud resources. As you may imagine, in a team setup, there is a high chance two people might try to make changes to the same infrastructure at the same time. Typically, this is handled off-band: team members tell each other they’re about to make some changes before they do.
Terraform offers a better solution to this problem through its use of Remote State, and its support for locks on this state. Each engineer acquires a lock before making any changes, preventing others in the team to touch the same resources at the same time. And to top it all off, acquiring and releasing locks is not a manual process; it’s handled automatically for you, behind the scenes!
All you have to do is choose the appropriate Terraform Backend for your Remote State.
So, this is how we finally, get to the “Encrypted” part of the title.
Since the internal Terraform state is no longer stored on one of our infrastructure engineers’ laptops, and some sensitive data may find its way in that state at some point, we would like that state to be encrypted at-rest.
This is supported in the S3 Backend, through the use of the AWS Key Management Service.
Now, that the article title makes sense, let’s move on to our setup:
1. Choose an Administrative Account
As per the official terraform guidelines, it is useful to have one account that is just used for administrative purposes.
This should be a top-level account, in which you basically won’t have any other Cloud resources. For those resources, you should use AWS Organizations to create sub-accounts where they will reside.
A good approach can be to have a separate AWS account per `environment` (e.g. separate account for staging, preprod, production, etc.).
In this account, we will only setup IAM users for our admins to connect to. Then they will use the AssumeRole feature to switch to other accounts.
2. IAM Setup in Administrative Account
First, let’s set up the IAM admin users.
Create an IAM group for admins that will concentrate all permissions for your admins. This way, all your admins will have the same permissions, and therefore a common way of working.
Now, it’s time to create IAM users for each of your admins. Make sure to provide them with programmatic access and share the access and secret keys with them, through a secure channel (ideally, your team’s Password Manager).
Each IAM user should then simply be added to the `admins` group.
Now, it is time to create the IAM Roles for each of the sub-accounts representing your different environments, so you can set up Access Delegation using IAM Roles. Our process is based https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html[on this guide].
The process seems more complicated than it really is:
- You login to each of your sub-accounts and *create an IAM role*. This role is for admins of the particular environment. It represents what they’re allowed to do. IAM admins from the top-level account, will then “assume” this role when managing this account.
- In creating the role, when choosing type, select `Another AWS account`.
- For Account ID, type the top-level Administrative Account ID.
- In Permissions, you assign to this IAM Role whatever permissions / IAM Policies you would like admins to have in this env. For us, this is how it looks like:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"elasticfilesystem:*",
"route53domains:*",
"rds:*",
"s3:*",
"cloudwatch:*",
"route53:*",
"lambda:*",
"ec2:*",
"elasticloadbalancing:*",
"autoscaling:*"
],
"Resource": "*"
}
]
}
----
- Now, go back to top-level account and grant access to top-level IAM admins, by adding the following Custom
Policy to your `admins` group:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID:role/UpdateApp"
}
}
NOTE: Depending on your ops team size, you might want to further split your admin users into multiple IAM groups, each with different permissions for different AWS resources. This kind of separation lies beyond the scope of this document.
3. Select AWS Region
This region is for where your terraform state will be stored. Not for the actual resources. You should choose the region where (most of) your ops team is. For us, it is eu-west-1.
4. AWS KMS Key
With your admins set up, it is time to move on to the resources that will be required by Terraform. The first one is the AWS KMS key.
This key will be used to encrypt / decrypt terraform state in the S3 buckets, that we will use as our remote state backend.
AWS KMS is accessible from https://console.aws.amazon.com/iam/home?region=eu-west-1#/encryptionKeys/eu-west-1[here].
Please follow the steps below to create the key:
- Set alias: `terraform` and Description: `Used by terraform to store remote state`
- In the `Advanced` section select: `KMS`
- Next, you are prompted to select the users that have admin rights over this key. This should be a super-admin (probably the same person creating this key).
- Next, you are prompted to select the users that can use this key. In this context, this means the users that will be able to use the key to encrypt/decrypt the terraform state stored in the S3 bucket. You should probably insert the IAM users you created in step [2] above.
- You are prompted to Confirm policy. Proceed.
- Finally, please make sure you note down the ARN — you’ll need it below.
5. S3 Bucket
Remember, Terraform state is saved using remote state, so that it’s not just accessible on one computer, on a local file. We are using S3 as our terraform backend, to store this state, so we need an S3 bucket.
First, go to the region you decided in step [3] above.
Then:
- Create s3 bucket: Use `YourBucket` as the name.
- Make absolutely sure you enable versioning on bucket (or you’ll regret it)
- Enable default encryption with the KMS key you created in step [4].
- Add s3 bucket policy to ensure state is encrypted:
{
"Version":"2012-10-17",
"Id":"PutObjPolicy",
"Statement":[
{
"Sid":"DenyIncorrectEncryptionHeader",
"Effect":"Deny",
"Principal":"*",
"Action":"s3:PutObject",
"Resource":"arn:aws:s3:::YourBucket/*",
"Condition":{
"StringNotEquals":{
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
},
{
"Sid":"DenyUnEncryptedObjectUploads",
"Effect":"Deny",
"Principal":"*",
"Action":"s3:PutObject",
"Resource":"arn:aws:s3:::YourBucket/*",
"Condition":{
"Null":{
"s3:x-amz-server-side-encryption":"true"
}
}
}
]
}
IMPORTANT: Make sure to substitute `YourBucket` in the policy above with your bucket name.
Finally, you need to make sure your IAM admins from step [2], also have access to this S3 bucket for reading writing state. The following policy should cover it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::YourBucket/*"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::YourBucket/path/to/my/key"
},
{
"Effect": "Allow",
"Action": "dynamodb:*",
"Resource": "arn:aws:dynamodb:eu-west-1:<account_id>:table/terraform_locks"
}
]
}
6. DynamoDB Table
Create a new DynamoDB table, like so, or through the web console:
aws dynamodb create-table \
--region eu-west-1 \
--table-name terraform_locks \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
7. Terraform setup
With all the above in place, it’s time to start using the S3 backend for terraform.
Define S3 Backend
terraform {
backend "s3" {
bucket = "YourBucket" <1>
key = "common/production" <2>
region = "eu-west-1" <3>
dynamodb_table = "terraform_locks" <4>
kms_key_id = "arn:aws:kms:eu-west-1:<top-level-account-id>:key/<key-id>" <5>
}
}
<1> We are using *a single bucket* across all our terraform-backed projects. <2> The key is essentially the path in the bucket where you will store remote state. This should be UNIQUE for each of your projects. (By "projects", we mean for each part of your cloud infrastructure for which you would like to keep a separate maintenance lifecycle. In our example, we use `common/production` to store some common cloud resources for our production environment, e.g. a VPC, subnets, some security groups, etc.) <3> The `region` here is the bucket region, NOT the region in which you will be storing resources. Set this to the region where you created your bucket. <4> Like note (1) above, again, we are using a single DynamoDB table for all of our projects. Not one for each. <5> The AWS KMS key *ARN* (not ID !!!). You should have gotten this ARN from [Step 4] above.
Define AWS Provider for Multi-Account setup
Now that we have the remote state backend configured, let’s move on to how we will manage a multi-account setup. We use AWS Organizations and as per Step 1 & 2 above, we have a top-level account which is only used for administrative purposes.
This means we need to set up our AWS terraform provider in such a way that each of our admin users running terraform will:
- login as the IAM top-level account user
- switch to the respective sub-account in which the cloud resources will be created.
.AWS Provider Definition
provider "aws" {
region = "${var.aws_region}"
profile = "admin_iam" <1>
assume_role = {
role_arn = "arn:aws:iam::<aws-sub-account-id>:role/<role-name>" <2>
}
}
<1> For point (1) above, we rely on authentication through the `~/.aws/credentials` file and the use of AWS Profiles. In regular AWS terraform provider setups, each admin user can simply set the `AWS_PROFILE` environment variable and terraform picks up the respective credentials automatically. However, due to a terraform bug, you need BOTH the `profile` attribute in the AWS provider AND the environment variable. <2> For switching to the respective environment, you should have set up Delegate Access via IAM Roles, as described in Step 2. There is an IAM Role for each of your AWS Organizations Subaccounts where you will keep your resources. This is the ARN of the IAM Role of the particular environment for which you are now creating your resources.
.AWS Credentials File
[admin_iam] aws_access_key_id=... aws_secret_access_key=...
8. Let’s test this setup!
Head on over to the directory where your terraform files reside and run:
$ terraform init Initializing the backend... Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.8.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.8" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
9. Woohoo!!
Congrats! You should now be all set up to start using terraform with S3 as the remote state backend.
Troubleshooting:
No valid credential sources
If you see:
Error configuring the backend "s3": No valid credential sources found for AWS Provider. Please see https://terraform.io/docs/providers/aws/index.html for more information on providing credentials for the AWS Provider Please update the configuration in your Terraform files to fix this error then run this command again.
Check that:
- your `~/.aws/credentials` file exists
- there is a valid profile there
- you have a `profile` attribute in your AWS terraform provider
- the profile name in credentials file matches the name in your AWS terraform provider configuration.
- you have set the `AWS_PROFILE` env var (*as well as* having set the attribute, because of https://github.com/hashicorp/terraform/issues/6474[this issue].
If trying to fix that, you come across:
Error inspecting states in the "s3" backend: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' status code: 400, request id: 91F11B60E33DD51A, host id: Dl2TRE87ABvUrgDOngu370uxr1t4rIkh9BEdbeDLN2ComnaeGscgspYHoFpoprQ4x+EeYaEf/fs= Prior to changing backends, Terraform inspects the source and destination states to determine what kind of migration steps need to be taken, if any. Terraform failed to load the states. The data in both the source and the destination remain unmodified. Please resolve the above error and try again.
… then *as long as you’re sure you are still on the early setup and the following command is safe* you’ll probably need to `rm -rf .terraform` so that it gets
IAM Users Permissions
If you are struggling with `403` and permission denied errors, here’s a checklist:
- S3 bucket access permissions: Ensure IAM user have been granted access to read & write from/to S3 bucket.
- KMS Key Permissions: Ensure IAM user have been added as *users* of the KMS key.
- Permissions to Assume role: Ensure IAM user have permission to assume the sub-account role (either directly or through an IAM group).
- Permissions to Create AWS Resources: Ensure IAM Role created in sub-account has sufficient permissions to create the actual resources in your `.tf` files.
Sorry, the comment form is closed at this time.