abhi-io · September 28, 2025 13:01
diff --git a/DevOps Exercise TF b/DevOps Exercise TF
 # DevOps Exercise: Secure Serverless Data Ingestion Pipeline

 Objective

 Provision a secure, event-driven data pipeline entirely using Terraform. The pipeline will use an AWS Lambda function, triggered by an S3 event, to ingest data into a relational database (RDS). Critically, all database credentials must be managed securely through AWS Secrets Manager, and the Lambda must operate within a VPC to connect to the private RDS instance.

 This exercise validates your ability to provision a complex, secure, and networked infrastructure stack using IaC.

 Scenario

 A financial service needs a fully automated mechanism to record transaction logs deposited as files into an S3 bucket. A serverless function must process this event, retrieve the necessary database credentials securely, and insert a new record into a private relational database (PostgreSQL or MySQL).

 ##The Environment Stack

    Cloud Provider: AWS

    Infrastructure as Code (IaC): Terraform

    Database: AWS RDS (PostgreSQL or MySQL, db.t2.micro or equivalent free-tier eligible instance)

    Compute: AWS Lambda (Python runtime)

    Storage: AWS S3

    Security: AWS Secrets Manager, IAM, and VPC/Security Groups

 Requirements

 1. Infrastructure as Code (Terraform)

 You must use Terraform to define and provision the entire infrastructure stack:

    VPC & Networking: Create a dedicated VPC, at least two public subnets, and two private subnets. Include a NAT Gateway in the public subnet to allow private resources (like Lambda) to access public services (like S3 and Secrets Manager APIs).

    Security Groups: Define necessary Security Groups (SGs).

        RDS SG: Must only allow inbound traffic on the database port (e.g., 3306 or 5432) from the Lambda function's Security Group.

        Lambda SG: Must allow outbound traffic to the RDS Security Group.

    RDS Instance: Provision a single, non-publicly accessible (private) RDS instance in the private subnets.

    S3 Bucket: Create an S3 bucket configured for event notifications.

    Secrets Manager: Create an aws_secretsmanager_secret resource to store the RDS credentials (username and a dynamically generated password).

 2. Database Setup

    Schema: The RDS instance must have a database and a simple table structure pre-defined (if possible via Terraform's provisioners or a post-deployment script) for the log insertion:
    SQL

    CREATE TABLE transaction_records (
        id SERIAL PRIMARY KEY,
        file_key VARCHAR(255) NOT NULL,
        insertion_time TIMESTAMP NOT NULL DEFAULT NOW()
    );

 3. AWS Lambda Function (Python)

    Deployment: Package the Python code and deploy it using Terraform's aws_lambda_function resource.

    VPC Configuration: The Lambda must be attached to the private subnets and the Lambda Security Group (vpc_config).

    Secure Credential Retrieval: The Python handler (lambda_function.py) must use the boto3 SDK to retrieve the database credentials from Secrets Manager at runtime. The Lambda code must not contain hardcoded secrets.

    Logic:

        Upon invocation, extract the file_key (object key) from the S3 event data.

        Retrieve RDS credentials from Secrets Manager.

        Connect to the RDS instance using the retrieved credentials and the instance endpoint.

        Insert a new record into the transaction_records table, logging the file_key.

        Execute a SELECT COUNT(*) query and return the total record count.

 4. IAM Policies and Roles

    Define a robust IAM Role for the Lambda function.

    The policy must grant only the minimum necessary permissions:

        Permissions to read the specific secret from Secrets Manager (secretsmanager:GetSecretValue).

        Permissions to create, manage, and delete network interfaces (required when running Lambda inside a VPC: ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, etc.).

        Permissions to write logs to CloudWatch.

 5. S3 Trigger

    Configure an S3 bucket notification event in Terraform that invokes the Lambda function whenever a new object is created in the bucket (s3:ObjectCreated:*).

 Submission Requirements

    Terraform Code: A complete set of .tf files defining the VPC, Subnets, NAT Gateway, Security Groups, Secrets Manager, IAM Role, RDS instance, S3 bucket, and Lambda function, pushed to a repository.

    Lambda Code: The lambda_function.py Python script that connects to RDS using credentials from Secrets Manager.

    Setup Instructions (README.md):

        Step-by-step instructions on how to initialize, plan, and apply the Terraform configuration.

        Verification steps (e.g., how to upload a file to S3 and check the Lambda logs/CloudWatch for the total record count).

    Security Review: A section in the README explaining how the security requirements (VPC isolation, Secrets Manager usage, least-privilege IAM) were met.
	# DevOps Exercise: Secure Serverless Data Ingestion Pipeline

	Objective

	Provision a secure, event-driven data pipeline entirely using Terraform. The pipeline will use an AWS Lambda function, triggered by an S3 event, to ingest data into a relational database (RDS). Critically, all database credentials must be managed securely through AWS Secrets Manager, and the Lambda must operate within a VPC to connect to the private RDS instance.

	This exercise validates your ability to provision a complex, secure, and networked infrastructure stack using IaC.

	Scenario

	A financial service needs a fully automated mechanism to record transaction logs deposited as files into an S3 bucket. A serverless function must process this event, retrieve the necessary database credentials securely, and insert a new record into a private relational database (PostgreSQL or MySQL).

	##The Environment Stack

	Cloud Provider: AWS

	Infrastructure as Code (IaC): Terraform

	Database: AWS RDS (PostgreSQL or MySQL, db.t2.micro or equivalent free-tier eligible instance)

	Compute: AWS Lambda (Python runtime)

	Storage: AWS S3

	Security: AWS Secrets Manager, IAM, and VPC/Security Groups

	Requirements

	1. Infrastructure as Code (Terraform)

	You must use Terraform to define and provision the entire infrastructure stack:

	VPC & Networking: Create a dedicated VPC, at least two public subnets, and two private subnets. Include a NAT Gateway in the public subnet to allow private resources (like Lambda) to access public services (like S3 and Secrets Manager APIs).

	Security Groups: Define necessary Security Groups (SGs).

	RDS SG: Must only allow inbound traffic on the database port (e.g., 3306 or 5432) from the Lambda function's Security Group.

	Lambda SG: Must allow outbound traffic to the RDS Security Group.

	RDS Instance: Provision a single, non-publicly accessible (private) RDS instance in the private subnets.

	S3 Bucket: Create an S3 bucket configured for event notifications.

	Secrets Manager: Create an aws_secretsmanager_secret resource to store the RDS credentials (username and a dynamically generated password).

	2. Database Setup

	Schema: The RDS instance must have a database and a simple table structure pre-defined (if possible via Terraform's provisioners or a post-deployment script) for the log insertion:
	SQL

	CREATE TABLE transaction_records (
	id SERIAL PRIMARY KEY,
	file_key VARCHAR(255) NOT NULL,
	insertion_time TIMESTAMP NOT NULL DEFAULT NOW()
	);

	3. AWS Lambda Function (Python)

	Deployment: Package the Python code and deploy it using Terraform's aws_lambda_function resource.

	VPC Configuration: The Lambda must be attached to the private subnets and the Lambda Security Group (vpc_config).

	Secure Credential Retrieval: The Python handler (lambda_function.py) must use the boto3 SDK to retrieve the database credentials from Secrets Manager at runtime. The Lambda code must not contain hardcoded secrets.

	Logic:

	Upon invocation, extract the file_key (object key) from the S3 event data.

	Retrieve RDS credentials from Secrets Manager.

	Connect to the RDS instance using the retrieved credentials and the instance endpoint.

	Insert a new record into the transaction_records table, logging the file_key.

	Execute a SELECT COUNT(*) query and return the total record count.

	4. IAM Policies and Roles

	Define a robust IAM Role for the Lambda function.

	The policy must grant only the minimum necessary permissions:

	Permissions to read the specific secret from Secrets Manager (secretsmanager:GetSecretValue).

	Permissions to create, manage, and delete network interfaces (required when running Lambda inside a VPC: ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, etc.).

	Permissions to write logs to CloudWatch.

	5. S3 Trigger

	Configure an S3 bucket notification event in Terraform that invokes the Lambda function whenever a new object is created in the bucket (s3:ObjectCreated:*).

	Submission Requirements

	Terraform Code: A complete set of .tf files defining the VPC, Subnets, NAT Gateway, Security Groups, Secrets Manager, IAM Role, RDS instance, S3 bucket, and Lambda function, pushed to a repository.

	Lambda Code: The lambda_function.py Python script that connects to RDS using credentials from Secrets Manager.

	Setup Instructions (README.md):

	Step-by-step instructions on how to initialize, plan, and apply the Terraform configuration.

	Verification steps (e.g., how to upload a file to S3 and check the Lambda logs/CloudWatch for the total record count).

	Security Review: A section in the README explaining how the security requirements (VPC isolation, Secrets Manager usage, least-privilege IAM) were met.