Manage your secrets with HashiCorp Vault

How to run HashiCorp Vault in production

Security is important like never before. Keeping your secrets safe should be a top priority. This blog shows you how to get started in production.

In a software delivery pipeline, there are several environments involved and thus many types of secrets. We create our pipelines as code, but of course we cannot store our secrets in version control. Here at Praqma, we had a customer who raised the question of securing credentials for a container cluster used for building and testing.

Enter HashiCorp Vault, an encrypted key store with mechanisms to control access to the stored values. Values can be everything from passwords, certificates, URLs to other sensitive data. Vault supports a variety of backends to authenticate users, like GitHub, AWS, LDAP and Radius.

It is really easy to try out Vault, using what they call dev-mode. Things get a bit more tricky when you want to put it into production. We will show you how to setup Vault in production mode and control access to the content.

Let’s get started

We love containers at Praqma, so it would be nothing but strange to not run Vault in a container. Our setup is based on the official Docker image for Vault maintained by HashiCorp, and a free web UI called djenriquez/vault-ui. Both can be found on the Docker Hub.

A simple Dockerfile for Vault would look like this:

FROM vault:0.7.0
MAINTAINER Praqma

And for the UI:

FROM djenriquez/vault-ui:2.1.0
MAINTAINER Praqma

Place these in separate folders called vault and vault-ui, and we are ready to build and push to the Docker Hub:

docker build -t praqma/vault:0.7.0 vault/.
docker build -t praqma/vault-ui:2.0.2 vault-ui/.

docker push praqma/vault:0.7.0
docker push praqma/vault-ui:2.0.2

When done, we have two images in the Docker Hub, ready to use in our organization.

Prepare the network and configuration

The Vault UI needs to connect to the container running Vault, so let’s create a network:

docker network create vault-network

You can verify the network is created by running:

docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
7b332ca38df5        bridge              bridge              local
60f40b5e80f7        vault-network       bridge              local

Next we create a configuration file. We need to specify a location for the backend storage, in this case a file, which address range Vault should listen to and whether or not TLS should be disabled.

Create a file called vault.conf with the following content:

backend "file" {
  path = "/vault/file/vaultsecrets"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

We choose to put the storage file at /vault/file inside the container, as this is one of the volumes defined by the base container image. This way we can mount it on the host if we want to.

We tell Vault to listen on all network interfaces on port 8200, and to turn off TLS as we don’t have any certificates. When you do, add two lines like this to the listener section:

tls_cert_file = "/root/server.crt"
tls_key_file = "/root/server.key"

Remember to copy your certificates into the container at the location you specify.

Start the Vault container

Now we are ready to start our Vault container:

docker run -d \
  --hostname vault \
  --network vault-network \
  --privileged \
  --name vault \
  --cap-add=IPC_LOCK \
  -v ./vault.conf:/vault/config/vault.conf \
  -v /opt/persistant_storage/vault/:/vault/file/ \
  praqma/vault:0.7.0 server -config /vault/config/vault.conf

Vault should be running, and you can confirm this by running:

docker logs vault

You should get output like this:

==> Vault server configuration:
                     Cgo: disabled
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
                 Storage: file
                 Version: Vault v0.7.0
             Version Sha: 614deacfca3f3b7162bbf30a36d6fc7362cd47f0

==> Vault server started! Log data will stream in below:

Hooray, Vault is running. Now let’s initialize it, so we can start using it.

Initialization

Lock

Start a terminal session inside the Vault container:

docker exec -it vault /bin/sh

Set an environment variable to tell the Vault client where to connect, and then initialize Vault:

export VAULT_ADDR=http://127.0.0.1:8200
vault init

From this you will get five keys and a root token. When Vault starts up, it is sealed, preventing access to any data. The keys generated are used to unseal Vault and you need at least three keys to do so.

Unseal Key 1: z+LgaNbDbejE+hxqq9Q87vjBBfQH6VII1Ey/vmiH+Q4B
Unseal Key 2: Zmy5jsn/+LQCMKzaM97imgLOQ2XIBHrmms6Oyxdz5eoC
Unseal Key 3: xFEtXOUM6w4JMG+4tf4VaXu/bjoMbQx68/YKzMxO5G4D
Unseal Key 4: MKsgyCt935cPcKzvNRPE+0MKOveD0GYvCh7ktsTJvp0E
Unseal Key 5: kpa0GgeOzC0EcG+NszMzCDp7F6hHuRCzYyZgsR/0vxkF
Initial Root Token: 9cfb29e7-0fde-cc24-df72-76ee59f5c712

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.

You unseal Vault by using three of the keys like this:

/ # vault unseal z+LgaNbDbejE+hxqq9Q87vjBBfQH6VII1Ey/vmiH+Q4B
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: 88d38704-415f-b945-e00e-747a51aa8f14
/ # vault unseal Zmy5jsn/+LQCMKzaM97imgLOQ2XIBHrmms6Oyxdz5eoC
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
Unseal Nonce: 88d38704-415f-b945-e00e-747a51aa8f14
/ # vault unseal xFEtXOUM6w4JMG+4tf4VaXu/bjoMbQx68/YKzMxO5G4D
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:

Now that Vault is unsealed, we can authenticate by using the root token:

/ # vault auth 9cfb29e7-0fde-cc24-df72-76ee59f5c712
Successfully authenticated! You are now logged in.
token: 9cfb29e7-0fde-cc24-df72-76ee59f5c712
token_duration: 0
token_policies: [root]

And we are in.

We need to create a policy for our team to use. This will give them the permissions they need when fetching data from Vault.

In the container, create a policy file in HashiCorp’s HCL format:

vi /vault/praqma-policy.hcl

With the following content:

path "*"{
    capabilities = [ "create", "read", "update", "list" ]
}

This will give people access to everything related to these operations, but for example not deleting secrets. This is for demonstration purposes and you should adjust according to your security standards.

Then create the policy:

/ # vault policy-write praqma /vault/praqma-policy.hcl
Policy 'praqma' written.

Now we will enable a GitHub backend so our employees to use their GitHub account to login to the Vault UI.

/ # vault auth-enable github
Successfully enabled 'github' at 'github'!

Set the organization that a GitHub user must be member of:

/ # vault write auth/github/config organization=Praqma
Success! Data written to: auth/github/config

Tie the login from GitHub to the policy we created:

/ # vault write auth/github/map/teams/default value=praqma
Success! Data written to: auth/github/map/teams/default

In this example we used the team default. You can have multiple teams with different policies, like ops and dev, to get more fine grained access level control.

Start the Vault UI container

Vault UI

Exit the Vault container, so we can start the UI container.

We need to specify two environment variables: VAULT_URL_DEFAULT points to the Vault server and VAULT_AUTH_DEFAULT specifies the default authentication method, in this case GitHub:

docker run -d \
  -p 8000:8000 \
  --network vault-network \
  --hostname vault-ui \
  -e VAULT_URL_DEFAULT="http://vault:8200" \
  -e VAULT_AUTH_DEFAULT="GITHUB" \
  --name vault-ui \
  praqma/vault-ui:2.0.2

Running docker ps should show you two running containers: vault and vault-ui:

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                    NAMES
5941ad1aaeb8        praqma/vault-ui:2.0.2   "/entrypoint.sh np..."   5 seconds ago       Up 4 seconds        0.0.0.0:8000->8000/tcp   vault-ui
d0d126ac150e        praqma/vault:0.7.0      "docker-entrypoint..."   19 minutes ago      Up 19 minutes       8200/tcp                 vault

We see that Vault listens on port 8200 and Vault-UI on port 8000. Go to http://localhost:8000 and you should get the Vault-UI login screen.

Now go to github.com, click settings under your profile picture and then Personal access tokens in the left menu. Click on the button “Generate new token”, give the token a name and choose the permissions you want to give this token (You need at least read permissions).

Insert the token in the Vault-UI to login. You should now see the UI.

Click on the “secret” menu item in the left menu and click the “new secret” button. Give the secret a name and a secret value (by clicking on the square next to “(empty object)”) and then click Save. The new item should appear. Try and click the trash can in the right. You will be told “Error, Access denied”, as defined in our policy from earlier.

Summary

We got started with Vault, beyond dev-mode. We saw how to run and configure the Vault server and a UI where we can get an overview.