Skip to content

Instantly share code, notes, and snippets.

@tommydangerous

tommydangerous/mage_delta-lake_combine_magic_energy_and_battle_history.py

Last active May 9, 2023 23:46
Show Gist options
  • Save tommydangerous/470ed247a6c503ef04ba978723fa7930 to your computer and use it in GitHub Desktop.
Save tommydangerous/470ed247a6c503ef04ba978723fa7930 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. tommydangerous revised this gist May 9, 2023. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions mage_delta-lake_combine_magic_energy_and_battle_history.py
    View file
    Original file line number Diff line number Diff line change
    @@ -5,6 +5,8 @@
    @transformer
    def transform(magic_energy, *args, **kwargs):
    dt = DeltaTable(
    # Change this to your unique URI from a previous step
    # if you’re using your own AWS credentials.
    's3://mage-demo-public/battle-history/1337',
    storage_options={
    'AWS_ACCESS_KEY_ID': '...',
  2. tommydangerous revised this gist May 9, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions mage_delta-lake_combine_magic_energy_and_battle_history.py
    View file
    Original file line number Diff line number Diff line change
    @@ -7,8 +7,8 @@ def transform(magic_energy, *args, **kwargs):
    dt = DeltaTable(
    's3://mage-demo-public/battle-history/1337',
    storage_options={
    'AWS_ACCESS_KEY_ID': 'AKIAZ4SRK3YKRLA66XKN',
    'AWS_SECRET_ACCESS_KEY': '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',
    'AWS_ACCESS_KEY_ID': '...',
    'AWS_SECRET_ACCESS_KEY': '...',
    },
    )
    battle_history = dt.to_pandas()
  3. tommydangerous created this gist May 9, 2023.
    16 changes: 16 additions & 0 deletions mage_delta-lake_combine_magic_energy_and_battle_history.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,16 @@
    from deltalake import DeltaTable
    import pandas as pd


    @transformer
    def transform(magic_energy, *args, **kwargs):
    dt = DeltaTable(
    's3://mage-demo-public/battle-history/1337',
    storage_options={
    'AWS_ACCESS_KEY_ID': 'AKIAZ4SRK3YKRLA66XKN',
    'AWS_SECRET_ACCESS_KEY': '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',
    },
    )
    battle_history = dt.to_pandas()

    return pd.concat([magic_energy, battle_history])