Hardcoded credentials in application code, environment variables, and configuration files are one of the most common sources of AWS security incidents. A developer commits a database password to a Git repository. It gets indexed by a code scanner. The database gets compromised. The company spends a week rotating every credential in their environment and auditing what was accessed.
This is not a theoretical scenario. It happens regularly to teams that know better. The fix is removing credentials from code entirely and storing them in a purpose-built secret management service.
AWS Secrets Manager gives you a central, auditable, automatically rotating store for every secret your applications need. In this article I will walk you through setting it up properly, integrating it with your applications and CI/CD pipelines, and configuring automatic rotation so credentials update without any human involvement.
Secrets Manager versus Parameter Store
AWS has two services that store sensitive values. Teams often ask which one to use. The answer is straightforward.
Use Parameter Store for non-secret configuration values like feature flags, application settings, and environment-specific URLs. It is free for standard parameters and integrates cleanly with Lambda environment variables and ECS task definitions. Use SecureString parameters when you want encryption but do not need rotation.
Use Secrets Manager for actual secrets: database passwords, API keys, OAuth tokens, TLS certificates, and anything else that needs automatic rotation or fine-grained access auditing. It costs $0.40 per secret per month plus $0.05 per 10,000 API calls, which is negligible for the security guarantees it provides.
Creating Secrets with Terraform
resource "aws_kms_key" "secrets" {
description = "KMS key for Secrets Manager encryption"
deletion_window_in_days = 30
enable_key_rotation = true
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "Enable IAM User Permissions"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
}
Action = "kms:*"
Resource = "*"
},
{
Sid = "Allow Secrets Manager"
Effect = "Allow"
Principal = {
Service = "secretsmanager.amazonaws.com"
}
Action = [
"kms:GenerateDataKey",
"kms:Decrypt"
]
Resource = "*"
}
]
})
}
resource "aws_kms_alias" "secrets" {
name = "alias/secrets-manager"
target_key_id = aws_kms_key.secrets.key_id
}
resource "aws_secretsmanager_secret" "database" {
name = "production/app/database"
description = "Production application database credentials"
kms_key_id = aws_kms_key.secrets.arn
recovery_window_in_days = 30
tags = {
Environment = "production"
Application = "app"
ManagedBy = "terraform"
}
}
resource "aws_secretsmanager_secret_version" "database" {
secret_id = aws_secretsmanager_secret.database.id
secret_string = jsonencode({
username = "app_user"
password = var.initial_db_password
host = aws_db_instance.main.address
port = 5432
dbname = "appdb"
engine = "postgres"
})
lifecycle {
ignore_changes = [secret_string]
}
}
resource "aws_secretsmanager_secret" "third_party_api" {
name = "production/app/stripe-api-key"
description = "Stripe API secret key for payment processing"
kms_key_id = aws_kms_key.secrets.arn
recovery_window_in_days = 7
}
resource "aws_secretsmanager_secret_version" "third_party_api" {
secret_id = aws_secretsmanager_secret.third_party_api.id
secret_string = jsonencode({
secret_key = var.stripe_secret_key
})
lifecycle {
ignore_changes = [secret_string]
}
}The lifecycle ignore_changes block on secret_string is important. Once Secrets Manager starts rotating a secret automatically, Terraform should not overwrite the rotated value on the next apply. Without this, every terraform apply resets the password to the initial value, breaking rotation.
Using a customer-managed KMS key gives you control over who can decrypt secrets. The default AWS-managed key works but you cannot restrict access to it independently of the secret itself. A customer-managed key lets you add a condition that only specific services or roles can use it for decryption.
IAM Policies for Secret Access
The principle of least privilege applies directly here. Each application should only be able to read the secrets it needs, nothing else.
resource "aws_iam_policy" "app_secrets_read" {
name = "app-secrets-read-policy"
description = "Allows the application to read its required secrets"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "ReadAppSecrets"
Effect = "Allow"
Action = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
]
Resource = [
aws_secretsmanager_secret.database.arn,
aws_secretsmanager_secret.third_party_api.arn
]
},
{
Sid = "DecryptSecrets"
Effect = "Allow"
Action = [
"kms:Decrypt",
"kms:GenerateDataKey"
]
Resource = aws_kms_key.secrets.arn
Condition = {
StringEquals = {
"kms:ViaService" = "secretsmanager.${var.region}.amazonaws.com"
}
}
}
]
})
}
resource "aws_iam_role_policy_attachment" "app_secrets" {
role = aws_iam_role.app_execution_role.name
policy_arn = aws_iam_policy.app_secrets_read.arn
}The kms:ViaService condition on the KMS policy ensures the key can only be used when the request comes through Secrets Manager, not by calling KMS directly. This is a defense-in-depth measure that prevents someone from bypassing Secrets Manager’s access controls by going straight to KMS.
Reading Secrets in Your Application
The recommended pattern is to fetch secrets once at application startup and cache them in memory. Fetching on every request adds latency and unnecessary API calls.
import boto3
import json
import os
from functools import lru_cache
@lru_cache(maxsize=None)
def get_secret(secret_name: str) -> dict:
client = boto3.client(
"secretsmanager",
region_name=os.environ.get("AWS_REGION", "us-east-1")
)
response = client.get_secret_value(SecretId=secret_name)
if "SecretString" in response:
return json.loads(response["SecretString"])
else:
raise ValueError(f"Secret {secret_name} contains binary data, expected string")
class DatabaseConfig:
def __init__(self):
secret = get_secret("production/app/database")
self.host = secret["host"]
self.port = secret["port"]
self.database = secret["dbname"]
self.username = secret["username"]
self.password = secret["password"]
@property
def connection_string(self) -> str:
return f"postgresql://{self.username}:{self.password}@{self.host}:{self.port}/{self.database}"
db_config = DatabaseConfig()Using lru_cache caches the secret value in memory for the lifetime of the process. For Lambda functions, the cache persists across invocations within the same container instance. For long-running services, implement a TTL-based cache that refreshes periodically so rotated credentials are picked up without a restart.
Automatic Rotation with Lambda
Automatic rotation is the feature that separates Secrets Manager from a simple encrypted key-value store. You configure a rotation schedule and a Lambda function. Secrets Manager calls the Lambda on schedule, which generates a new credential, tests it, and updates the secret. No human involvement required.
resource "aws_secretsmanager_secret_rotation" "database" {
secret_id = aws_secretsmanager_secret.database.id
rotation_lambda_arn = aws_lambda_function.secret_rotation.arn
rotation_rules {
automatically_after_days = 30
}
}
resource "aws_lambda_permission" "secrets_manager_rotation" {
statement_id = "AllowSecretsManagerRotation"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.secret_rotation.function_name
principal = "secretsmanager.amazonaws.com"
source_arn = aws_secretsmanager_secret.database.arn
}The rotation Lambda follows a four-step lifecycle that Secrets Manager calls in sequence: createSecret, setSecret, testSecret, and finishSecret. Here is the structure of that function:
import boto3
import json
import string
import secrets
import psycopg2
def lambda_handler(event, context):
secret_arn = event["SecretId"]
token = event["ClientRequestToken"]
step = event["Step"]
client = boto3.client("secretsmanager")
if step == "createSecret":
create_secret(client, secret_arn, token)
elif step == "setSecret":
set_secret(client, secret_arn, token)
elif step == "testSecret":
test_secret(client, secret_arn, token)
elif step == "finishSecret":
finish_secret(client, secret_arn, token)
def generate_password(length=32) -> str:
alphabet = string.ascii_letters + string.digits + "!#$%&*+-=?"
return "".join(secrets.choice(alphabet) for _ in range(length))
def create_secret(client, secret_arn, token):
try:
client.get_secret_value(
SecretId=secret_arn,
VersionId=token,
VersionStage="AWSPENDING"
)
return
except client.exceptions.ResourceNotFoundException:
pass
current = json.loads(
client.get_secret_value(
SecretId=secret_arn,
VersionStage="AWSCURRENT"
)["SecretString"]
)
new_password = generate_password()
pending = {**current, "password": new_password}
client.put_secret_value(
SecretId=secret_arn,
ClientRequestToken=token,
SecretString=json.dumps(pending),
VersionStages=["AWSPENDING"]
)
def set_secret(client, secret_arn, token):
pending = json.loads(
client.get_secret_value(
SecretId=secret_arn,
VersionId=token,
VersionStage="AWSPENDING"
)["SecretString"]
)
current = json.loads(
client.get_secret_value(
SecretId=secret_arn,
VersionStage="AWSCURRENT"
)["SecretString"]
)
conn = psycopg2.connect(
host=current["host"],
port=current["port"],
database=current["dbname"],
user=current["username"],
password=current["password"]
)
conn.autocommit = True
with conn.cursor() as cur:
cur.execute(
"ALTER USER %s WITH PASSWORD %s",
(pending["username"], pending["password"])
)
conn.close()
def test_secret(client, secret_arn, token):
pending = json.loads(
client.get_secret_value(
SecretId=secret_arn,
VersionId=token,
VersionStage="AWSPENDING"
)["SecretString"]
)
conn = psycopg2.connect(
host=pending["host"],
port=pending["port"],
database=pending["dbname"],
user=pending["username"],
password=pending["password"]
)
conn.close()
def finish_secret(client, secret_arn, token):
metadata = client.describe_secret(SecretId=secret_arn)
current_version = next(
version for version, stages in metadata["VersionIdsToStages"].items()
if "AWSCURRENT" in stages
)
client.update_secret_version_stage(
SecretId=secret_arn,
VersionStage="AWSCURRENT",
MoveToVersionId=token,
RemoveFromVersionId=current_version
)The rotation function uses the AWSPENDING version stage as a holding area for the new credential until it is tested and confirmed. If the testSecret step fails, the AWSCURRENT credential is still valid and the rotation aborts cleanly. Your application never sees a broken credential.
Auditing Secret Access with CloudTrail
Every call to GetSecretValue is logged in CloudTrail. This gives you a complete audit trail of which role, user, or service accessed which secret and when. A CloudWatch Insights query to review recent access:
fields @timestamp, userIdentity.arn, requestParameters.secretId, sourceIPAddress
| filter eventSource = "secretsmanager.amazonaws.com"
| filter eventName = "GetSecretValue"
| sort @timestamp desc
| limit 100Set up an alarm for unexpected access patterns. If a secret that is only accessed by your ECS task role starts getting accessed from an EC2 instance IP or a Lambda function that should not touch it, that is worth an immediate alert.
Using Secrets in ECS Task Definitions
ECS has native Secrets Manager integration. You reference secrets directly in your task definition and ECS injects them as environment variables at container startup. The container never sees the raw API call.
resource "aws_ecs_task_definition" "app" {
family = "app"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = 512
memory = 1024
execution_role_arn = aws_iam_role.ecs_execution.arn
task_role_arn = aws_iam_role.ecs_task.arn
container_definitions = jsonencode([{
name = "app"
image = "${aws_ecr_repository.app.repository_url}:latest"
essential = true
secrets = [
{
name = "DB_PASSWORD"
valueFrom = "${aws_secretsmanager_secret.database.arn}:password::"
},
{
name = "DB_HOST"
valueFrom = "${aws_secretsmanager_secret.database.arn}:host::"
},
{
name = "STRIPE_SECRET_KEY"
valueFrom = "${aws_secretsmanager_secret.third_party_api.arn}:secret_key::"
}
]
portMappings = [{
containerPort = 8080
protocol = "tcp"
}]
}])
}The valueFrom syntax with the colon-separated suffix lets you extract individual JSON keys from a secret. DB_PASSWORD gets only the password field, not the entire JSON blob. This is cleaner than fetching the whole secret and parsing it inside the container.
Closing Thoughts
Secrets Manager is not a complicated service. The concept is simple: credentials live in one place, access is controlled by IAM, and rotation happens automatically. The value comes from discipline: every secret your applications need goes through Secrets Manager, without exception.
The teams that get this wrong are the ones who use Secrets Manager for most things but still have a handful of hardcoded credentials that never got migrated. Those are the credentials that get leaked. Audit your environment, find every hardcoded credential, and migrate them. Set up rotation on everything that supports it. Check your CloudTrail logs for unexpected access patterns regularly.
Enjoy the cloud.
Osama
#AWS #SecretsManager #CloudSecurity #DevSecOps #AWSLambda #Terraform #InfrastructureAsCode #CredentialManagement #CloudNative #AmazonWebServices #SolutionsArchitect #ECS #CloudArchitecture #CloudComputing #BackendEngineering #SecurityEngineering #TechBlog #KMS #SecretRotation #CloudGovernance
Technology Geek 
Leave a comment