Goutham R mahtuog
mahtuog
/ spark_aws_lambda.py
Created
August 27, 2020 18:55
— forked from tomron/spark_aws_lambda.py
Example of python code to submit spark process as an emr step to AWS emr cluster in AWS lambda function
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| import time | |
| import boto3 | |
| def lambda_handler(event, context): | |
| conn = boto3.client("emr") | |
| # chooses the first cluster which is Running or Waiting | |
| # possibly can also choose by name or already have the cluster id | |
| clusters = conn.list_clusters() |