Ephemeral resources and write-only arguments appeared in Terraform a long time ago, back in version 1.10, but there was no opportunity to write about them in detail.

The main idea behind them is not to leave “traces” in the state file, which is especially useful for passwords or tokens, because the data only exists during the execution of Terraform itself in its memory.

However, there are certain limitations to their use – we’ll look at those later, but first, let’s see everything in action.

Example without ephemeral values and write-only arguments

Let’s start with the old scheme, without using ephemeral resources and write-only arguments – we will create a random password, the resource aws_secretsmanager_secret, store this password in it, and get it from data:

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      component   = "devops"
      created-by  = "terraform"
      environment = "test"
    }
  }
}

### RESOURCES ###

# generate a random password
resource "random_password" "test_random_password" {
   length  = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name                    = "db_password"
  description             = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
  secret_string = random_password.test_random_password.result
}

### DATA SOURCES ###

# retrieve the AWS Secret value
data "aws_secretsmanager_secret_version" "test_aws_secret_data" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id

  depends_on = [aws_secretsmanager_secret_version.test_aws_secret_version]
}

### OUTPUTS ###

# get the random password value
output test_random_password {
  value       = random_password.test_random_password.result
  sensitive   = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = data.aws_secretsmanager_secret_version.test_aws_secret_data.secret_string
  sensitive   = true
}

Here we are:

• resource “random_password”: generate the password itself
• resource “aws_secretsmanager_secret”: create a new entry in AWS Secrets Manager
• resource ‘aws_secretsmanager_secret_version’: write the value from resource “random_password” to this Secret
• data “aws_secretsmanager_secret_version”: get the value from AWS Secrets Manager
• output “test_random_password”: output the value from resource ‘random_password’
• output “test_aws_secret”: output the value obtained from AWS Secrets Manager

Execute terraform init and terraform apply:

...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

test_aws_secret = <sensitive>
test_random_password = <sensitive>

Looks OK – in outputs, thanks to sensitive = true, nothing is displayed.

But the password is in the state file:

$ cat terraform.tfstate
{
  ...
  "outputs": {
    "test_aws_secret": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    },
    "test_random_password": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    }
  },
...
  "resources": [
    {
      "mode": "data",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_data",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "random_password",
      "name": "test_random_password",
      ...
            "result": "1atcZYGR",

Now let’s start hiding this data from the state.

Using Write-Only Attributes

Resource attributes with the suffix _wo are “write-only” data, meaning that Terraform keeps them in memory during operations but does not store them anywhere.

However, not all resources support these attributes. For example, in AWS RDS, you can pass a password via the password_wo attribute through the aws_db_instance resource, but in aws_opensearch_domain and its master_user_password attribute to create a root user in the internal user database – not yet.

Official documentation – Use write-only arguments.

aws_secretsmanager_secret_version also supports write-only attributes – secret_string_wo instead of secret_string, and secret_string_wo_version instead of secret_string_version.

The use of secret_string_wo_version is mandatory for secret_string_wo, because since Terraform does not store password information, it will not know when to update it. To do this, we set a version that we increment each time we want to update the password.

Edit the code, change the only resource “aws_secretsmanager_secret_version” – set secret_string_wo and secret_string_wo_version, leaving the rest unchanged:

...
# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = random_password.test_random_password.result
  secret_string_wo_version = 1
}
...

Execute terraform apply, and check the state now:

$ cat terraform.tfstate
{
  ...
  "outputs": {
    "test_aws_secret": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    },
    "test_random_password": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    }
  },
...
  "resources": [
    {
      "mode": "data",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_data",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "",
            "secret_string_wo": null,
            "secret_string_wo_version": 1,

...
    {
      "mode": "managed",
      "type": "random_password",
      "name": "test_random_password",
      ...
            "result": "1atcZYGR",

Now we have managed.aws_secretsmanager_secret_version.test_aws_secret_version with no values for secret_string and secret_string_wo.

Using Ephemeral Resources

The idea behind “ephemeral” resources is the same as with write-only arguments – these resources only exist in Terraform’s memory during the execution of terraform apply and are not stored in the state file.

However, the use of such resources is limited:

• you can refer to them in write-only arguments
• in other ephemeral resources
• in locals
• in ephemeral variables
• in providers, provisioners, and connections

Documentation – Ephemeral block reference.

Let’s edit our code and change resource “random_password” to ephemeral ‘random_password’, leave resource “aws_secretsmanager_secret_version” – it will write the password to AWS Secrets Manager but will not store the value in state, and add a new resource – ephemeral “aws_secretsmanager_secret_version”, through which we will get this password back in Terraform.

At the same time, in the secret_string_wo and in output “test_random_password” we now refer to the password through ephemeral – ephemeral.random_password.test_random_password.result.

And in the output “test_aws_secret” we also use ephemeral.aws_secretsmanager_secret_version.test_aws_secret_data.secret_string.

The data ‘aws_secretsmanager_secret_version’ can be removed, because we will now get the password from the ephemeral “aws_secretsmanager_secret_version”:

...

### RESOURCES ###

# generate a random password
ephemeral "random_password" "test_random_password" {
   length  = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name                    = "db_password"
  description             = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = ephemeral.random_password.test_random_password.result
  secret_string_wo_version = 1
}

### DATA SOURCES ###

# Retrieve the password from Secrets Manager (ephemeral)
ephemeral "aws_secretsmanager_secret_version" "test_aws_secret_version_ephemeral" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
}

# retrieve the AWS Secret value
# data "aws_secretsmanager_secret_version" "test_aws_secret_data" {
#   secret_id = aws_secretsmanager_secret.test_aws_secret.id

#   depends_on = [aws_secretsmanager_secret_version.test_aws_secret_version]
# }

### OUTPUTS ###

# get the random password value
output test_random_password {
  value       = ephemeral.random_password.test_random_password.result
  sensitive   = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sensitive   = true
}

The “This output value is not declared as returning an ephemeral value” error

Execute terraform applyand catch the first error:

...
│ Error: Ephemeral value not allowed
│ 
│   on main.tf line 53, in output "test_random_password":
│   53:   value       = ephemeral.random_password.test_random_password.result
│ 
│ This output value is not declared as returning an ephemeral value, so it cannot be set to a result derived from an ephemeral value.
╵
╷
│ Error: Ephemeral value not allowed
│ 
│   on main.tf line 59, in output "test_aws_secret":
│   59:   value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
│ 
│ This output value is not declared as returning an ephemeral value, so it cannot be set to a result derived from an ephemeral value.

But even if we add the parameter ephemeral = true:

...
### OUTPUTS ###

# get the random password value
output test_random_password {
  value       = ephemeral.random_password.test_random_password.result
  sensitive   = true
  ephemeral = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sensitive   = true
  ephemeral = true
}

It still won’t work.

The “Ephemeral outputs are not allowed in context of a root module” error

Now the error will look like this:

...
╷
│ Error: Ephemeral output not allowed
│ 
│   on main.tf line 52:
│   52: output test_random_password {
│ 
│ Ephemeral outputs are not allowed in context of a root module
╵
╷
│ Error: Ephemeral output not allowed
│ 
│   on main.tf line 59:
│   59: output "test_aws_secret" {
│ 
│ Ephemeral outputs are not allowed in context of a root module

Because Ephemeral outputs can only be used in modules – we’ll see how later.

OK – for now, let’s just remove Outputs, and now terraform apply runs without any problems:

$ terraform apply
...
random_password.test_random_password: Refreshing state... [id=none]
ephemeral.random_password.test_random_password: Opening...
ephemeral.random_password.test_random_password: Opening complete after 0s
...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
...
ephemeral.random_password.test_random_password: Closing...
ephemeral.random_password.test_random_password: Closing complete after 0s
...

Please note that for ephemeral resources, Terraform now performs Opening and Closing operations instead of Reading and Refreshing state. That is, it simply creates an object in memory, reads the resource into it, and then “closes” and removes it from memory.

Let’s check the state file now:

...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "",
            "secret_string_wo": null,
            "secret_string_wo_version": 1,

...

Now we have:

• resources ephemeral “random_password” and ephemeral “aws_secretsmanager_secret_version” are not in the state at all,
• and managed.aws_secretsmanager_secret_version.test_aws_secret_version still has an empty field in secret_string_wo because we made it write-only earlier

OK, but how do we use the password now? Because we removed data “aws_secretsmanager_secret_version”.

Using values from Ephemeral resources

We have already seen an example of referencing Ephemeral resources above when we did secret_string_wo = ephemeral.random_password.test_random_password.result.

Similarly, we can use ephemeral.aws_secretsmanager_secret_version.db_password_wo_ephemeral.secret_string.

As mentioned above, we cannot do this everywhere, but it is allowed in providers.

To verify this, let’s run PostgreSQL with our password (we’ll take it directly from AWS Console > AWS Secrets Manager):

Launch a container, to which we pass the variable POSTGRES_PASSWORD="1atcZYGR":

$ docker run --rm --name some-postgres -e POSTGRES_PASSWORD="1atcZYGR" -p 5432:5432 postgres

Add the provider to our code and use it to connect to the container, where we will create a test database.

In the provider’s password field we will use a value from the ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string:

...

### PostgreSQL Configuration

terraform {
  required_providers {
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "~> 1.20"
    }
  }
}

provider "postgresql" {
  host     = "localhost"
  port     = 5432
  username = "postgres"
  password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name              = "demo_db"
  template          = "template0"
  connection_limit  = -1
  allow_connections = true
}

Run terraform init and terraform apply:

$ terraform init && terraform apply
...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening complete after 1s
postgresql_database.demo_db: Creating...
postgresql_database.demo_db: Creation complete after 0s [id=demo_db]
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing complete after 0s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Check the database:

$ export PGPASSWORD="1atcZYGR"
$ psql -h localhost -U postgres -c "\l"
                                                    List of databases
   Name    |  Owner   | Encoding | Locale Provider |  Collate   |   Ctype    | Locale | ICU Rules |   Access privileges   
-----------+----------+----------+-----------------+------------+------------+--------+-----------+-----------------------
 demo_db   | postgres | UTF8     | libc            | en_US.utf8 | en_US.utf8 |        |           | 
...

In the same way, we could use an ephemeral resource via locals:

...
locals {
  db_password_local = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
}

provider "postgresql" {
  host     = "localhost"
  port     = 5432
  username = "postgres"
  password = local.db_password_local
  #password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name              = "demo_db_via_local"
  template          = "template0"
  connection_limit  = -1
  allow_connections = true
}

Check:

$ terraform apply
...
  # postgresql_database.demo_db will be updated in-place
  ~ resource "postgresql_database" "demo_db" {
        id                     = "demo_db"
      ~ name                   = "demo_db" -> "demo_db_via_local"
        # (10 unchanged attributes hidden)
    }
...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

And in the state file, the password is not visible anywhere:

$ cat terraform.tfstate | grep 1atcZYGR | echo $?
127

Using Ephemeral Outputs

Above, we tried to use output “test_aws_secret” with ephemeral = true, but got the error “Ephemeral outputs are not allowed in context of a root module”.

Let’s try using it in our own module.

Documentation – ephemeral – Avoid storing values in state or plan files.

Let’s create a module modules/secret_ephemeral, in which we will generate a password and save it in AWS Secrets Manager, and add Ephemeral Output.

And in the root module, we will use outputs of this module to get ephemeral “aws_secretsmanager_secret_version”, as we did above.

Let’s write the file modules/secret_ephemeral/secret.tf:

### RESOURCES ###

# generate a random password
ephemeral "random_password" "test_random_password" {
   length  = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name                    = "db_password_via_module"
  description             = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = ephemeral.random_password.test_random_password.result
  secret_string_wo_version = 1
}

# Retrieve the password from Secrets Manager (ephemeral)
ephemeral "aws_secretsmanager_secret_version" "test_aws_secret_version_ephemeral" {
  secret_id     = aws_secretsmanager_secret.test_aws_secret.id
}

output "password_ephemeral" {
  value     = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  ephemeral = true
}

In the main file main.tf, remove everything related to the password, add a module call, and in locals use its output:

...

### PostgreSQL Configuration

terraform {
  required_providers {
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "~> 1.20"
    }
  }
}

module "secret_ephemeral" {
  source = "./modules/secret_ephemeral"
}

locals {
  db_password_local = module.secret_ephemeral.password_ephemeral
}

provider "postgresql" {
  host     = "localhost"
  port     = 5432
  username = "postgres"
  password = local.db_password_local
  #password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name              = "demo_db_via"
  template          = "template0"
  connection_limit  = -1
  allow_connections = true
}

First, you need to create a password – run terraform apply without resource “postgresql_database”, and update the container launch with the new password:

$ docker run --rm --name some-postgres -e POSTGRES_PASSWORD="PHsfzcIx" -p 5432:5432 postgres

Now our provider uses a password from the Ephemeral Output module modules/secret_ephemeral:

...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening complete after 1s
postgresql_database.demo_db: Creating...
postgresql_database.demo_db: Creation complete after 0s [id=demo_db_via]
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing complete after 0s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

In the state, we still don’t have a password:

$ cat terraform.tfstate | grep PHsfzcIx | echo $?
127

That’s basically it.

It’s a shame that aws_opensearch_domain doesn’t support write-only. I wanted to use it for the root password 🙁

But there is already an issue on GitHub Support ephemeral “write-only” argument for aws_opensearch_domain, and even a comment saying “I have started working on this issue, and will submit a PR shortly”.

And in the pull request itself, you can even see how it’s implemented.

Useful links

• Securely storing credentials in Terraform with ‘Ephemeral Blocks’ and ‘Write-Only’ attributes
• Understanding ephemerality in Terraform
• Ensuring Terraform State Security with Ephemeral Values and Write-Only Outputs
• Terraform 1.10: Secure Secrets with Ephemeral Values