Hashicorp Vault – Enabling HA with AWS RDS and PostgreSQL

Vault offers an array of flexible storage backends with a view to providing a highly available storage location to store secrets, this is a great baked-in design choice as if you make Vault an integral part of your infrastructure you can ill afford a sudden outage, a perfect platform for storing structured data is, of course, a RDBMS (Relational Database Management System), as many of the mainstays are scalable and support clustering out of the box and in the age of Cloud providers RDBMS‘ are often provided as a PaaS or SaaS offering. In this example we’ll be looking at using AWS RDS (Amazon Relational Database) running PostgreSQL as a viable HA backend for Vault.

Creating A PostgreSQL Instance on AWS RDS

First things first, we can’t use a database that doesn’t exist, so lets get one stood up, for the sake of understanding the process, lets use the AWS GUI. Within the RDS console we’ll Create a new RDS Instance:

Next we have the option to define Storage settings, these can be tailored to your requirements, for this setup I’ve stuck with the default settings, we can always come back and scale them later.

Next we define Connectivity to the Instance and this has a couple of gotchas:

1. First, unless you’re sure the rest of your VPC is properly configured (and you’re sure that you’re 100% actually need to and are comfortable making the Instance publicly available then don’t do it). I would still err on the side of caution as this database is going to contain all of your secrets and do you really want that to be on the internet?
2. Your VPC must have at least two Subnets created in at least two different Availability Zones ahead of time (your service wouldn’t be very highly available if it was all in one place after all)
3. Even if you do make your Instance publicly available, you will still need to permit TCP port 5432 (PostgreSQL) through your VPC’s Security Group(s) or create a new one and then allow it
4. The VPC that you select MUST be configured for both DNS Resoltion and DNS Hostnames. Failure to have these enabled will simply deny the creation of an Instance

Once this is all completed we will see our Instance is in a “creating” state, this takes a few minutes to complete:

Creating and Configuring the PostgreSQL Database

Now we have an Instance but it doesn’t actually contain a database that we can use, RDS isn’t psychic after all and doesn’t know that we intend to use this platform to host our Vault backend.

Before we can connect to the Instance, we need to know it’s address, once the creation is complete we can click in to the Instance and see the endpoint details under Connectivity & Security, these are what we will use to allow any downstream system to connect:

The best way to reach in to the Instance from here and do some damage is using pgAdmin, a free, cross-platform GUI for managing PostgreSQL which can be downloaded for free from https://www.pgadmin.org/.

Once downloaded, you will need to set up your connection back to your Instance:

All going well, you will be able to connect to your instance, if you can’t Amazon provide a very helpful guide for troubleshooting connections here, but there’s a very high chance it comes down to security rules in your VPC.

Once connected, we’ll need to complete a few tasks:

1. Create a new Database for Vault
2. Create the KV Store, Schema and Indexes
3. Create the HAEnabled backend
4. Create a Service Account so we aren’t doing everything with the master password

All of these are done with SQL statements which we can execute directly against the Instance within pgAdmin:

Create New Database Named vault:

CREATE DATABASE vault;

Create KV Store, Schema and Indexes:

CREATE TABLE vault_kv_store (
	parent_path TEXT COLLATE "C" NOT NULL,
	path        TEXT COLLATE "C",
	key         TEXT COLLATE "C",
	value       BYTEA,
	CONSTRAINT pkey PRIMARY KEY (path, key)
);

CREATE INDEX parent_path_idx ON vault_kv_store (parent_path);

Create the HAEnabled backend:

CREATE TABLE vault_ha_locks (
	ha_key                                      TEXT COLLATE "C" NOT NULL,
	ha_identity                                 TEXT COLLATE "C" NOT NULL,
	ha_value                                    TEXT COLLATE "C",
	valid_until                                 TIMESTAMP WITH TIME ZONE NOT NULL,
	CONSTRAINT ha_key PRIMARY KEY (ha_key)
);

Create a Database Service Account and grant Full Access to vault Database (substitute your own password as appropriate):

CREATE USER vaultadmin WITH PASSWORD 'supersecretpassword';
GRANT ALL PRIVILEGES ON DATABASE vault TO vaultadmin;

We can now see our new Database and Service Account are configured if we refresh pgAdmin:

Configuring Vault

Now that our database is created and configured, we need to configured Vault to look at our new backend.

On the host where Vault is installed (in our case mc-vault) we will need to edit /etc/vault.hcl and change the storage stanza to reflect that we wish to use postgres and provide a URL comprising of the Service Account Credentials and the Database Endpoint, a functional configuration is as below:

storage "postgresql" {
        connection_url = "postgres://vaultadmin:supersecretpassword@vaultbackend.crfdefx7ec1z.eu-west-2.rds.amazonaws.com:5432/vault"
}

ui = true

listener "tcp" {
        address = "192.168.1.47:8200"
        tls_disable = 0
}

max_lease_ttl = "10h"
default_lease_ttl = "10h"
api_addr = "https://192.168.1.47:8200"

Finally we can simply restart the Vault service with systemctl restart vault. If we now browse to the Vault UI we can see a new backend has been loaded and is waiting for first time initialisation: