We have an automation for AWS IAM that creates EKS Access Entries to give AWS IAM Users access to an EKS cluster.

I don’t remember if I wrote it myself or if some LLM generated it (although judging by the code, I did 🙂 ), but later I discovered an unpleasant feature of how this automation works: when a user is deleted from variables, Terraform starts “re-mapping” other users.

Actually, today we’ll look at how I did it, recall Terraform’s data types, and see how that should have been done it to avoid such problems.

Although the error is described about aws_eks_access_entry resource (here in the examples it will be local_file instead of aws_eks_access_entry), it actually concerns the general approach to using indexes and loops in Terraform.

Contents

Current implementation

Simplified, it looks like this:

variable "eks_clusters" {
  description = "List of EKS clusters to create records"
  type        = set(string)
  default = [
    "cluster-1",
    "cluster-2"
  ]
}

variable "eks_users" {
  description = "IAM Users to be added to EKS with aws_eks_access_entry, one item in the set() per each IAM User"
  type        = map(list(string))
  default = {
    backend = [
      "user1",
      "user2",
      "user3",
    ]
  }
}

locals {
  eks_users_access_entries_backend = flatten([
    for cluster in var.eks_clusters : [
      for user_arn in var.eks_users.backend : {
        cluster_name  = cluster
        principal_arn = user_arn
      }
    ]
  ])
}

resource "local_file" "backend" {
  for_each = { for cluster, user in local.eks_users_access_entries_backend : cluster => user }

   filename = "${each.value.cluster_name}@${each.value.principal_arn}.txt"
   content  = <<EOF
    cluster_name=${each.value.cluster_name}
    principal_arn=${each.value.principal_arn}
  EOF
}

Only in the original, instead of the resource "local_file", the resource "aws_eks_access_entry" is used.

Actually, in this code:

variable "eks_clusters": contains a list of our EKS clusters to which users should be attached
variable "eks_users": contains lists of groups (backend in this example) and users in this group – user1, user2, user3
locals.eks_users_access_entries_backend: creates a list for each unique combination of EKS cluster + IAM user
resource "local_file": for each cluster and each user, creates a file with a name like [email protected]

Now let’s take a closer look at the variables and data types – since I did this a long time ago, it’s useful to recall.

Variables and data types

`variable "eks_clusters"`

The type is simply set(string) with two elements –“cluster-1” and“cluster-2“:

variable "eks_clusters" {
  description = "List of EKS clusters to create records"
  type        = set(string)
  default = [
    "cluster-1",
    "cluster-2"
  ]
}

The set[] type has no indexes, and objects are accessed in any order.

`variable "eks_users"`

variable "eks_users" {
  description = "IAM Users to be added to EKS with aws_eks_access_entry, one item in the set() per each IAM User"
  type        = map(list(string))
  default = {
    backend = [
      "user1",
      "user2",
      "user3",
    ]
  }
}

Here we have a variable with type map(list(string)).

map{} is a set of key => value data, where key is the name of a user group (devops, backend, qa), and in the value we have a nested list[] with objects of type string, where each object is a username.

A list, unlike a set, has indices for each element, and therefore the order of access to objects in the list will be in turn.

That is, we can access them by indexes 0-1-2:

eks_users["backend"].0: will be user1
eks_users["backend"].1: will be user2
eks_users["backend"].2: will be user3

You can check this using outputs:

output "eks_users_all" {
  value = var.eks_users
}

output "eks_users_backend" {
  value = var.eks_users["backend"]
}

output "eks_users_backend_user_1" {
  value = var.eks_users["backend"].0
}

output "eks_users_backend_user_2" {
  value = var.eks_users["backend"].1
}
...

And as a result of terraform apply we get:

$ terraform apply
...
eks_users_all = tomap({
  "backend" = tolist([
    "user1",
    "user2",
  ])
})
eks_users_backend = tolist([
  "user1",
  "user2",
])
eks_users_backend_user_1 = "user1"
eks_users_backend_user_2 = "user2"

Or just look in the terraform console:

> var.eks_users["backend"].0
"user1"
> var.eks_users["backend"].1
"user2"
> var.eks_users["backend"].2
"user3"

But the problem arises not because of the indexes themselves, but because of the way they change if an item in the list is deleted or moved, especially if these indexes are used as keys for for_each. Actually, we’ll get to that soon.

`local.eks_users_access_entries_backend`

locals {
  eks_users_access_entries_backend = flatten([
    for cluster in var.eks_clusters : [
      for user_arn in var.eks_users.backend : {
        cluster_name  = cluster
        principal_arn = user_arn
      }
    ]
  ])
}

Here we’re using a double for that iterates over each cluster from set(string) in the var.eks_clusters, and then for each user from list(string) in the var.eks_users.backend.

Let’s remove flatten() for now:

...
  eks_users_access_entries_backend_unflatten = [
    for cluster in var.eks_clusters : [
      for user_arn in var.eks_users.backend : {
        cluster_name  = cluster
        principal_arn = user_arn
      }
    ]
  ]
...

Now in the eks_users_access_entries_backend_unflatten = [ ... ] we get a nested list – list(list(object)):

external name = [ ... ] – this is the first level of the list, where each element is a cluster from var.eks_clusters
then with the for cluster in var.eks_clusters : [ ... ] we generate a separate nested list of objects for each cluster
and then with the for user_arn in var.eks_users.backend : { ... } objects with the fields cluster_name and principal_arn are created – one object for each pair of cluster_name and principal_arn

Let’s also look at it with the terraform console again:

> local.eks_users_access_entries_backend_unflatten
tolist([
  [
    {
      "cluster_name" = "cluster-1"
      "principal_arn" = "user1"
    },
    {
      "cluster_name" = "cluster-1"
      "principal_arn" = "user2"
    },
    {
      "cluster_name" = "cluster-1"
      "principal_arn" = "user3"
    },
  ],
  [
    {
      "cluster_name" = "cluster-2"
      "principal_arn" = "user1"
    },
    {
      "cluster_name" = "cluster-2"
      "principal_arn" = "user2"
    },
    {
      "cluster_name" = "cluster-2"
      "principal_arn" = "user3"
    },
  ],
])

And flatten() simply removes this nesting of list(list(object)), and turns the result into a flat list(object), where each object is a unique cluster + user pair:

...
eks_users_access_entries_backend = [
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user2"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user2"
  },
]
...

Okay – we’ve figured it out, let’s move on – let’s see how this list will be used in for_each, and what mistakes I made there.

`resource "local_file" "backend"`

Finally, the main thing is that using this list the code creates users for each cluster:

...
resource "local_file" "backend" {
  for_each = { for cluster, user in local.eks_users_access_entries_backend : cluster => user }

   filename = "${each.value.cluster_name}@${each.value.principal_arn}.txt"
   content  = <<EOF
    cluster_name=${each.value.cluster_name}
    principal_arn=${each.value.principal_arn}
  EOF
}

In the original, it looks like this:

resource "aws_eks_access_entry" "backend" {
  for_each = { for cluser, user in local.eks_users_access_entries_backend : cluser => user }

  cluster_name  = each.value.cluster_name
  principal_arn = each.value.principal_arn

  kubernetes_groups = [
    "backend-team"
  ]
}

And again, loops and lists 🙂

What do we have here: from { for cluster, user in local.eks_users_access_entries_backend : cluster => user }, a map{} is formed, where the key (cluster) is the index of each element from the local.eks_users_access_entries_backend list, and the value (user) is an object from this list at this index, and this object contains the cluster_name and principal_arn fields.

That is, in the cluster we will have the values 0, 1, 2, and in the user – the values { cluster_name = "cluster-1", principal_arn = "user1" }, { cluster_name = "cluster-1", principal_arn = "user2" }, { cluster_name = "cluster-1", principal_arn = "user3" } respectively.

So, my first mistake is the names cluster and user in the for loop itself: it would be more correct to call them simply for index, entry in ..., or index (or idx) and user – because in each user we have a combination that identifies the user – cluster+user.

One more thing: since the key in this for_each is an index of type number, and the value is an object of type object, we get not the map{} type but object{}, because in map the key and value must be of the same type, and Terraform cannot create map(number => object):

> type({ for cluster, user in local.eks_users_access_entries_backend : cluster => user })
object({
    0: object({
        cluster_name: string,
        principal_arn: string,
    }),
    1: object({
        cluster_name: string,
        principal_arn: string,
    }),
...

Although this is not essential now.

The Issue

Now let’s move on to the main problem: if we delete a user in the list of users in the variable "eks_users", i.e. instead of:

variable "eks_users" {
  description = "IAM Users to be added to EKS with aws_eks_access_entry, one item in the set() per each IAM User"
  type        = map(list(string))
  default = {
    backend = [
      "user1",
      "user2",
      "user3",
    ]
  }
}

Will do in that way:

variable "eks_users" {
  description = "IAM Users to be added to EKS with aws_eks_access_entry, one item in the set() per each IAM User"
  type        = map(list(string))
  default = {
    backend = [
      "user1",
      "user3",
    ]
  }
}

Then this will lead to the fact that, firstly, local.eks_users_access_entries_backend will change, because instead of six objects:

> local.eks_users_access_entries_backend
[
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user2"
  },
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user2"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  },
]

We get a new list with four objects:

> local.eks_users_access_entries_backend
[
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  },
  {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  },
]

And since for_each is formed based on the indexes of the local.eks_users_access_entries_backend list:

for_each = {
  for cluster, user in local.eks_users_access_entries_backend :
  cluster => user
}

Then, when the number of elements in local.eks_users_access_entries_backend changes, the map{} (which is still the object) in the condition for the for_each will change too, because it is created based on the indices of the local.eks_users_access_entries_backend list.

That is, instead of 0, 1, … 5:

> { for cluster, user in local.eks_users_access_entries_backend : cluster => user }
{
  "0" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  }
  "1" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user2"
  }
  "2" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  }
  "3" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  }
  "4" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user2"
  }
  "5" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  }
}

We now have 0, 1, … 3:

> { for cluster, user in local.eks_users_access_entries_backend : cluster => user }
{
  "0" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  }
  "1" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  }
  "2" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  }
  "3" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  }
}

And if earlier for_each created a map with the key “3” and the value {“cluster_name” = “cluster-2” “principal_arn” = “user1“}, then now the key “3” will have the {“cluster_name” = “cluster-2” “principal_arn” = “user3“} value.

And for Terraform, it looks like the value of the object with the same key has changed, and therefore it should delete the old resource local_file.backend["3"], and create a new one at the same index, but with the new content:

Terraform will perform the following actions:

  # local_file.backend["1"] must be replaced
-/+ resource "local_file" "backend" {
      ~ content              = <<-EOT # forces replacement
            cluster_name=cluster-1
          -     principal_arn=user2
          +     principal_arn=user3
        EOT
...

  # local_file.backend["2"] must be replaced
-/+ resource "local_file" "backend" {
      ~ content              = <<-EOT # forces replacement
          - cluster_name=cluster-1
          + cluster_name=cluster-2
          -     principal_arn=user3
          +     principal_arn=user1
        EOT
...

  # local_file.backend["3"] must be replaced
-/+ resource "local_file" "backend" {
      ~ content              = <<-EOT # forces replacement
            cluster_name=cluster-2
          -     principal_arn=user1
          +     principal_arn=user3
        EOT
...

And all this is because for_each is based on an unstable index that can change.

The Fix

So, how can we prevent this from happening?

Simply change the way keys are created for the for_each.

Instead of creating the key from an index as a key and a value as an object, as it is done now:

> { for cluster, user in local.eks_users_access_entries_backend : cluster => user }
{
  "0" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  }
  ...
  }
  "5" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  }
}

We can create a unique key for each cluster+user pair, and iterate on that key:

> { for entry in local.eks_users_access_entries_backend : "${entry.cluster_name}-${entry.principal_arn}" => entry }
{
  "cluster-1-user1" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  }
  "cluster-1-user2" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user2"
  }
  "cluster-1-user3" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  }
  "cluster-2-user1" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  }
  "cluster-2-user2" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user2"
  }
  "cluster-2-user3" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  }
}

And then when you delete “user2“, all the other keys in the condition for the for_each will not change, and Terraform will not change the files:

> { for entry in local.eks_users_access_entries_backend : "${entry.cluster_name}-${entry.principal_arn}" => entry }
{
  "cluster-1-user1" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user1"
  }
  "cluster-1-user3" = {
    "cluster_name" = "cluster-1"
    "principal_arn" = "user3"
  }
  "cluster-2-user1" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user1"
  }
  "cluster-2-user3" = {
    "cluster_name" = "cluster-2"
    "principal_arn" = "user3"
  }
}

In the code, it will look like this:

resource "local_file" "backend" {
  for_each = { for entry in local.eks_users_access_entries_backend : "${entry.cluster_name}-${entry.principal_arn}" => entry }

   filename = "${each.value.cluster_name}@${each.value.principal_arn}.txt"
   content  = <<EOF
    cluster_name=${each.value.cluster_name}
    principal_arn=${each.value.principal_arn}
  EOF
}

At first, Terraform will still recreate all the resources because the keys have changed, but you can safely add/remove users later.

M	T	W	T	F	S	S
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30