Skip to content

Instantly share code, notes, and snippets.

@tommydangerous

tommydangerous/mage_delta-lake_export_data_into_avatar.py

Last active May 11, 2023 18:18
Show Gist options
  • Save tommydangerous/8a154269818cdce996fcff9946de4384 to your computer and use it in GitHub Desktop.
Save tommydangerous/8a154269818cdce996fcff9946de4384 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. tommydangerous revised this gist May 11, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions mage_delta-lake_export_data_into_avatar.py
    View file
    Original file line number Diff line number Diff line change
    @@ -15,4 +15,5 @@ def export_data(combined_data, *args, **kwargs):
    'AWS_SECRET_ACCESS_KEY': '...',
    'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
    },
    partition_by=['planet'],
    )
  2. tommydangerous revised this gist May 9, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions mage_delta-lake_export_data_into_avatar.py
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,7 @@
    @data_exporter
    def export_data(combined_data, *args, **kwargs):
    write_deltalake(
    # Change this URI to your own unique URI
    's3://mage-demo-public/magic-energy-and-battle-history/1337',
    data=combined_data,
    mode='overwrite',
  3. tommydangerous created this gist May 9, 2023.
    17 changes: 17 additions & 0 deletions mage_delta-lake_export_data_into_avatar.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,17 @@
    from deltalake.writer import write_deltalake


    @data_exporter
    def export_data(combined_data, *args, **kwargs):
    write_deltalake(
    's3://mage-demo-public/magic-energy-and-battle-history/1337',
    data=combined_data,
    mode='overwrite',
    overwrite_schema=True,
    storage_options={
    'AWS_REGION': '...',
    'AWS_ACCESS_KEY_ID': '...',
    'AWS_SECRET_ACCESS_KEY': '...',
    'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
    },
    )