abhi-io · September 28, 2025 13:01 · Sep 28, 2025
diff --git a/DevOps Exercise TF b/DevOps Exercise TF
@@ -0,0 +1,106 @@
+# DevOps Exercise: Secure Serverless Data Ingestion Pipeline
+
+Objective
+
+Provision a secure, event-driven data pipeline entirely using Terraform. The pipeline will use an AWS Lambda function, triggered by an S3 event, to ingest data into a relational database (RDS). Critically, all database credentials must be managed securely through AWS Secrets Manager, and the Lambda must operate within a VPC to connect to the private RDS instance.
+
+This exercise validates your ability to provision a complex, secure, and networked infrastructure stack using IaC.
+
+Scenario
+
+A financial service needs a fully automated mechanism to record transaction logs deposited as files into an S3 bucket. A serverless function must process this event, retrieve the necessary database credentials securely, and insert a new record into a private relational database (PostgreSQL or MySQL).
+
+##The Environment Stack
+
+    Cloud Provider: AWS
+
+    Infrastructure as Code (IaC): Terraform
+
+    Database: AWS RDS (PostgreSQL or MySQL, db.t2.micro or equivalent free-tier eligible instance)
+
+    Compute: AWS Lambda (Python runtime)
+
+    Storage: AWS S3
+
+    Security: AWS Secrets Manager, IAM, and VPC/Security Groups
+
+Requirements
+
+1. Infrastructure as Code (Terraform)
+
+You must use Terraform to define and provision the entire infrastructure stack:
+
+    VPC & Networking: Create a dedicated VPC, at least two public subnets, and two private subnets. Include a NAT Gateway in the public subnet to allow private resources (like Lambda) to access public services (like S3 and Secrets Manager APIs).
+
+    Security Groups: Define necessary Security Groups (SGs).
+
+        RDS SG: Must only allow inbound traffic on the database port (e.g., 3306 or 5432) from the Lambda function's Security Group.
+
+        Lambda SG: Must allow outbound traffic to the RDS Security Group.
+
+    RDS Instance: Provision a single, non-publicly accessible (private) RDS instance in the private subnets.
+
+    S3 Bucket: Create an S3 bucket configured for event notifications.
+
+    Secrets Manager: Create an aws_secretsmanager_secret resource to store the RDS credentials (username and a dynamically generated password).
+
+2. Database Setup
+
+    Schema: The RDS instance must have a database and a simple table structure pre-defined (if possible via Terraform's provisioners or a post-deployment script) for the log insertion:
+    SQL
+
+    CREATE TABLE transaction_records (
+        id SERIAL PRIMARY KEY,
+        file_key VARCHAR(255) NOT NULL,
+        insertion_time TIMESTAMP NOT NULL DEFAULT NOW()
+    );
+
+3. AWS Lambda Function (Python)
+
+    Deployment: Package the Python code and deploy it using Terraform's aws_lambda_function resource.
+
+    VPC Configuration: The Lambda must be attached to the private subnets and the Lambda Security Group (vpc_config).
+
+    Secure Credential Retrieval: The Python handler (lambda_function.py) must use the boto3 SDK to retrieve the database credentials from Secrets Manager at runtime. The Lambda code must not contain hardcoded secrets.
+
+    Logic:
+
+        Upon invocation, extract the file_key (object key) from the S3 event data.
+
+        Retrieve RDS credentials from Secrets Manager.
+
+        Connect to the RDS instance using the retrieved credentials and the instance endpoint.
+
+        Insert a new record into the transaction_records table, logging the file_key.
+
+        Execute a SELECT COUNT(*) query and return the total record count.
+
+4. IAM Policies and Roles
+
+    Define a robust IAM Role for the Lambda function.
+
+    The policy must grant only the minimum necessary permissions:
+
+        Permissions to read the specific secret from Secrets Manager (secretsmanager:GetSecretValue).
+
+        Permissions to create, manage, and delete network interfaces (required when running Lambda inside a VPC: ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, etc.).
+
+        Permissions to write logs to CloudWatch.
+
+5. S3 Trigger
+
+    Configure an S3 bucket notification event in Terraform that invokes the Lambda function whenever a new object is created in the bucket (s3:ObjectCreated:*).
+
+Submission Requirements
+
+    Terraform Code: A complete set of .tf files defining the VPC, Subnets, NAT Gateway, Security Groups, Secrets Manager, IAM Role, RDS instance, S3 bucket, and Lambda function, pushed to a repository.
+
+    Lambda Code: The lambda_function.py Python script that connects to RDS using credentials from Secrets Manager.
+
+    Setup Instructions (README.md):
+
+        Step-by-step instructions on how to initialize, plan, and apply the Terraform configuration.
+
+        Verification steps (e.g., how to upload a file to S3 and check the Lambda logs/CloudWatch for the total record count).
+
+    Security Review: A section in the README explaining how the security requirements (VPC isolation, Secrets Manager usage, least-privilege IAM) were met.
No results found