Skip to content

Instantly share code, notes, and snippets.

@IrishBird

IrishBird/aws_glue_boto3_example.md

Forked from ejlp12/aws_glue_boto3_example.md
Created October 7, 2019 16:02
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save IrishBird/b3c01aad8de3bf2b470f34feff80959d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save IrishBird/b3c01aad8de3bf2b470f34feff80959d to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @ejlp12 ejlp12 revised this gist Nov 15, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion aws_glue_boto3_example.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,7 +11,7 @@ response = client.create_crawler(
    Targets={
    'S3Targets': [
    {
    'Path': 's3://eryan-etl-demo-bucket/data/csv',
    'Path': 's3://ejlp12-etl-demo-bucket/data/csv',
    'Exclusions': [
    ]
    },
  2. @ejlp12 ejlp12 revised this gist Nov 15, 2018. No changes.
  3. @ejlp12 ejlp12 created this gist Nov 15, 2018.
    47 changes: 47 additions & 0 deletions aws_glue_boto3_example.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,47 @@
    ```
    import boto3
    client = boto3.client('glue')
    response = client.create_crawler(
    Name='SalesCSVCrawler',
    Role='AWSGlueServiceRoleDefault',
    DatabaseName='sales-cvs',
    Description='Crawler for generated Sales schema',
    Targets={
    'S3Targets': [
    {
    'Path': 's3://eryan-etl-demo-bucket/data/csv',
    'Exclusions': [
    ]
    },
    ]
    },
    SchemaChangePolicy={
    'UpdateBehavior': 'UPDATE_IN_DATABASE',
    'DeleteBehavior': 'DELETE_FROM_DATABASE'
    }
    #,Configuration='{ "Version": 1.0, "CrawlerOutput": { "Partitions": { "AddOrUpdateBehavior": "InheritFromTable" } } }'
    )
    response = client.start_crawler(
    Name='SalesCSVCrawler'
    )
    response = client.update_table(
    DatabaseName='sales-cvs',
    TableInput={
    'Name': 'csv',
    'Description': 'Table Sales',
    'StorageDescriptor': {
    'SerdeInfo': {
    'Name': 'OpenCSVSerde',
    'SerializationLibrary': 'org.apache.hadoop.hive.serde2.OpenCSVSerde',
    'Parameters': {
    'separatorChar': ','
    }
    }
    }
    }
    )
    ```