Assigning broad IAM permissions to Amazon EKS worker nodes is a significant security risk. When you attach a policy to the node's IAM role, every pod running on that node inherits those permissions, violating the principle of least privilege. This creates a massive attack surface where a single compromised pod could potentially access your entire S3 data lake or delete DynamoDB tables.
IAM Roles for Service Accounts (IRSA) solves this by using OpenID Connect (OIDC) to provide fine-grained permissions. Instead of the node holding the keys, the individual Kubernetes service account is mapped to a specific AWS IAM role. This ensures that your applications only access the specific AWS resources they need to function.
TL;DR — Use IRSA to map IAM roles directly to Kubernetes ServiceAccounts via an OIDC provider. This eliminates the need for broad node-level permissions and ensures your EKS cluster adheres to strict security compliance standards.
The Architecture of IRSA and OIDC
IRSA relies on a trust relationship between AWS IAM and your EKS cluster's OIDC identity provider. When you enable the OIDC provider for your cluster, AWS generates a public OIDC endpoint. Kubernetes then uses this endpoint to issue "Web Identity Tokens" to pods. These tokens are short-lived and cryptographically signed, making them much more secure than static AWS credentials or long-term node roles.
The core benefit is isolation. In a multi-tenant cluster, Team A's pods cannot use the IAM role assigned to Team B's pods, even if they share the same physical worker node. This architecture is essential for organizations that must meet SOC2, HIPAA, or PCI-DSS compliance, as it provides a clear audit trail of which service accessed which resource via CloudTrail.
When to Use IRSA
You should use IRSA for any application running in EKS that needs to interact with AWS APIs. Common scenarios include an application uploading logs to S3, a controller like external-dns updating Route53 records, or a microservice querying a DynamoDB table. If the application uses an AWS SDK (like Boto3, AWS SDK for Go, or Java), IRSA is the standard way to provide credentials.
However, avoid over-engineering your security. You do not need IRSA for internal Kubernetes communication, such as a frontend talking to a backend service over ClusterIP. IRSA is strictly for "out-of-cluster" AWS resource access. For clusters with extremely high pod churn or thousands of service accounts, you might encounter OIDC throttling, though this is rare in standard production environments (typically those running EKS 1.27+).
The IRSA Authentication Flow
Understanding the data flow is critical for debugging why a pod might receive an "Access Denied" error despite having a role assigned. The process involves the EKS Admission Controller and the AWS Security Token Service (STS).
1. Pod starts -> EKS Admission Controller injects: - AWS_WEB_IDENTITY_TOKEN_FILE (path to token) - AWS_ROLE_ARN (the IAM role to assume) 2. Application SDK reads the token file. 3. SDK calls AWS STS: AssumeRoleWithWebIdentity. 4. STS validates the token against the Cluster OIDC Provider. 5. STS returns temporary credentials to the Pod. 6. Pod accesses AWS resources (S3, RDS, etc.).
This flow ensures that credentials are never stored as secrets in Kubernetes. They are generated on-the-fly and expire automatically. In my experience testing this on EKS 1.29, the token rotation handled by the kubelet ensures zero downtime for long-running processes.
Implementation Steps
Step 1: Create the IAM OIDC Provider
First, verify if your cluster has an existing OIDC provider. You can do this via the AWS CLI or Terraform. Each cluster has a unique OIDC URL that must be registered with IAM to establish trust.
# Using eksctl to associate the OIDC provider
eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve
Step 2: Create an IAM Role with a Trust Policy
The IAM role needs a trust policy that specifically allows the OIDC provider to assume the role, restricted to a specific namespace and service account name. This prevents other namespaces from "stealing" the role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:my-namespace:my-service-account"
}
}
}
]
}
Step 3: Annotate the Kubernetes ServiceAccount
Finally, create the ServiceAccount in Kubernetes and add the eks.amazonaws.com/role-arn annotation. The EKS pod identity webhook will detect this and inject the necessary environment variables into any pod using this ServiceAccount.
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: my-namespace
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role
IRSA vs. EKS Pod Identity
In late 2023, AWS introduced EKS Pod Identity as an alternative to IRSA. While IRSA is still widely used and highly resilient, it's important to understand the differences when designing a new architecture.
| Feature | IRSA (OIDC) | EKS Pod Identity |
|---|---|---|
| Complexity | High (Requires OIDC setup) | Low (Direct mapping) |
| Trust Policy | Per-cluster OIDC URL | Simplified pods.eks.amazonaws.com |
| Cross-cluster | Difficult (OIDC is cluster-specific) | Easier (Role reuse across clusters) |
| Scaling | Excellent | Excellent |
If you are managing hundreds of clusters, EKS Pod Identity simplifies IAM role management because you don't need to update the Trust Policy for every new cluster's OIDC URL. However, IRSA remains the choice for legacy clusters or environments where you need the most granular control over the OIDC federation itself.
Operational Tips for Security Teams
StringLike with wildcards (*) in your trust policy unless absolutely necessary. A wildcard like system:serviceaccount:default:* allows any pod in the default namespace to assume the role, which can lead to privilege escalation.
- Use Infrastructure as Code (IaC): Always manage your IRSA roles via Terraform or AWS CDK. Manual edits in the IAM console are prone to errors that are difficult to debug in Kubernetes logs.
- Monitor with CloudTrail: Use the
requestParameters.roleSessionNamein CloudTrail to identify exactly which pod performed an action. By default, EKS sets this to include the pod name. - Minimize Permissions: Use tools like
iam-policy-generatorto create the smallest possible JSON policy for your specific application needs.
By implementing IRSA, you move your EKS cluster toward a Zero Trust model. Your nodes become "dumb" execution environments, and your security logic lives where it belongs: at the identity layer. This approach significantly reduces the blast radius of any application-level vulnerability.
Frequently Asked Questions
Q. Why is my pod still getting 403 Forbidden even with IRSA?
A. This usually happens for three reasons: the ServiceAccount name in the IAM Trust Policy has a typo, the pod was not restarted after the ServiceAccount was annotated, or the IAM policy attached to the role does not have the necessary permissions for the specific AWS action.
Q. Can one Kubernetes ServiceAccount have multiple IAM roles?
A. No, a single ServiceAccount can only be annotated with one IAM role ARN. If your application needs permissions from multiple roles, you must combine those permissions into a single IAM policy or use multiple ServiceAccounts for different components of your app.
Q. Does IRSA work with Fargate?
A. Yes, IRSA is the recommended way to provide AWS permissions to pods running on AWS Fargate. Since Fargate doesn't allow you to manage the underlying node, IRSA is the only native way to grant granular IAM access to Fargate pods.
- IRSA uses OIDC to map IAM roles to Kubernetes ServiceAccounts.
- It enforces the principle of least privilege by removing permissions from the worker node.
- Always restrict trust policies to specific namespaces and service account names.
- Consider EKS Pod Identity for newer, multi-cluster environments to reduce IAM maintenance.
Post a Comment