Let the heavens rejoice; we can now share encrypted RDS snapshots between accounts within an AWS region.
Why am I so excited about this? Mainly because it allows us to more readily adopt a microservices approach to AWS account management – something I plan to write about in a future post.
For now, just take take this simple piece of advice: Don’t run your prod resources in the same AWS account as your dev/test (non-prod) resources. And since everything but your data is defined by code (it is, right?), it’s easy to deploy your application stack into any AWS account.
But how do you copy your database schema and data from one account to another? In the past, you had to 1) Export the schema manually, 2) Export the data with something like BCP, 3) Transfer the data, 4) Apply the schema to your new instance, 5) Import the data 6) Hope and pray that you didn’t miss any database settings, schema objects, or rows of data in the process.
Each time you wanted to copy data down from prod to dev, this complicated, fragile process had to be repeated. Why? Because restoring from a snapshot wasn’t supported between accounts. The only way around it was to keep your prod and non-prod resources in the same account (DANGER!).
Apparently I wasn’t the only one complaining to Amazon about this shortcoming, and they took a major step forward late last year by announcing support for sharing unencrypted database snapshots between accounts. This solved half the problem, but one major issue remained: What about my encrypted RDS instances? If you’re entrusted with sensitive, personally identification information (PII), then end-to-end encryption should be a top priority in the public cloud, including data at rest (yet another reason I argue you should treat your data as pets, not cattle).
Alas, there was still no viable option for a simple snapshot-based data overlay from one account to another.
Just when we (I) had all lost hope, Amazon announced four short months later that they now support sharing encrypted RDS snapshots between accounts!
At long last, we can use an AWS-native method to achieve a database overlay, even with encrypted volumes.
While this is great news, the process isn’t exactly straightforward. There are several moving pieces involved, which is why I decided to provide a walkthrough of the process from start to finish.
I’m going to show you how to accomplish the job through the console because it helps illustrate the concepts, but you should never use the console in production for anything but to verify your code worked. Get comfortable with the steps, then use your favorite AWS SDK or Command Line Tools to complete the same tasks in your projects.
And Now for the Main Event
There are 4 major tasks involved in sharing an encrypted RDS snapshot with another AWS account. They are:
- Create and share a custom KMS encryption key.
- Create an encrypted RDS instance using the KMS key you created.
- Create and share a snapshot of the encrypted RDS instance.
- Copy the shared snapshot to the target account.
(Note that you can only share encrypted snapshots within the same region. If you want to share a snapshot across regions, you cannot use encrypted storage.)
Without further ado, let’s get started.
Step 1: Create and Share a Custom KMS Encryption Key
Before you can share an encrypted RDS snapshot, you must create a new KMS key. You can’t use the default key because you can’t share access to it. If you want to share an existing snapshot of an encrypted instance that uses the default aws/rds KMS key for encryption, you’ll have to make a copy of it using your new custom KMS key first (similar to what you’ll see in step 4).
- Go to Identity and Access Management (IAM).
- Select “Encryption Keys” from the menu on the left.
- Click “Create Key”.
- Provide an alias and description for the new key and select “next step”.
- Choose the IAM users and roles that can administer the new KMS key through the KMS API.
- Choose whether the key administrators you chose in the last step can delete the key.
- Choose the IAM users and roles that can use the new KMS key.
- IMPORTANT: Click “Add an External Account” and enter the target account number. This is what allows the target account to decrypt the encrypted snapshot later on.
- Preview the key policy JSON document, and click finish.
Step 2: Create an Encrypted RDS Instance Using the New KMS Key
Now that your KMS key is created, you can create an RDS instance with encrypted storage using this key.
- From the AWS Console, select the RDS service.
- Click Launch a DB Instance.
- Select your engine and version. Here, I’ll select SQL Server Standard Edition.
- If you’re deploying the instance to PROD, select “Production” (which gives you Multi-AZ high availability and Provisioned IOPS). Otherwise select DEV/Test, as I am doing here.
- On the “Instance Specifications” page, select all the options you typically would with one exception: The instance class has to be at least db.m3.medium to support encryption (current as of the time of this writing).
- On the next page, I suggest using a non-default VPC, a private subnet group, a non-default port, and don’t allow the instance to be publicly accessible (but choose your own adventure).
- IMPORTANT: Select “Yes” for enable encryption, and select the new KMS key you created earlier under “Master Key”.
- Finalize your parameter choices, and click “Launch DB Instance”.
Step 3: Create and Share a Snapshot of Your Encrypted RDS Instance
Now the moment we’ve all been waiting for. Let’s create a snapshot and share it with another account.
- From the RDS Dashboard, select “Snapshots”.
- Click “Create Snapshot”.
- Choose your DB Instance and give the snapshot a name, then click “Create”.
- Wait for the snapshot to complete. Mine took just over a minute (You might have to hit refresh a few times before it shows as complete).
- Go back to the Snapshots tab in the RDS Dashboard and select the snapshot you just created. Click “Share Snapshot”.
- Notice that the public option for snapshot visibility is not available for encrypted snapshots. This is because, 1) The public at large does not have access to your KMS key (that would defeat the purpose), and 2) Why would you want to encrypt data at rest if you want to make it publicly available?
- Add the account number of the account you want to share the snapshot with (the same one you granted permission to access the KMS key earlier in this tutorial).
- Log out of the current AWS account, and log back in to the target account (or sign in to the target account using another browser – you can stay logged into both accounts that way).
- Navigate to the RDS Dashboard and select the “Snapshots” tab.
- Click the drop down filter list. It should say “Owned by Me” by default. Select “Shared with Me”.
- You should now see the snapshot you shared earlier in this list.
So you would think at this point that you can just select the snapshot and click “Restore Snapshot”, right?
Well, not quite. We have one more step.
Step 4: Create a Copy of the Shared Snapshot on Your Target Instance
Before restoring a shared, encrypted snapshot, you first have to make a copy of the snapshot in the target account.
- Select the snapshot and click “Copy Snapshot”.
- Note that you cannot share encrypted snapshots across regions, so the Destination Region is pre-selected for you.
- Enter in a value for the New DB Snapshot Identifier field, and select an encryption key (you’ll only see keys that are local to the current account).
- Select Copy Snapshot
- Switch your filter back to “Owned by Me”.
- You should now see the snapshot copy in progress. Wait for the copy to complete (took about 2 minutes for me).
- Once the copy is complete, you can select the copy of the snapshot and select “Restore Snapshot” to create a new instance from this snapshot.
Bonus Tip: If you already have a snapshot that was encrypted using the default KMS key, you can still share it. Just create a new KMS key like we did in step 1, make a copy of the snapshot in the source account using the new KMS key like we did in step 4, then follow steps 3 and 4 above.
And there you have it, four
simple steps to sharing an encrypted RDS snapshot. Happy overlaying!