Skip to content

Instantly share code, notes, and snippets.

@tommydangerous

tommydangerous/mage_delta-lake_load_magic_energy.py

Last active May 9, 2023 23:30
Show Gist options
  • Save tommydangerous/98f14bcbef5d73aac7c284d3d0662cba to your computer and use it in GitHub Desktop.
Save tommydangerous/98f14bcbef5d73aac7c284d3d0662cba to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. tommydangerous revised this gist May 9, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions mage_delta-lake_load_magic_energy.py
    View file
    Original file line number Diff line number Diff line change
    @@ -6,8 +6,8 @@ def load_data(*args, **kwargs):
    dt = DeltaTable(
    's3://mage-demo-public/magic-energy/1337',
    storage_options={
    'AWS_ACCESS_KEY_ID': 'AKIAZ4SRK3YKRLA66XKN',
    'AWS_SECRET_ACCESS_KEY': '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',
    'AWS_ACCESS_KEY_ID': '...',
    'AWS_SECRET_ACCESS_KEY': '...',
    },
    )
    return dt.to_pandas()
  2. tommydangerous created this gist May 9, 2023.
    13 changes: 13 additions & 0 deletions mage_delta-lake_load_magic_energy.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,13 @@
    from deltalake import DeltaTable


    @data_loader
    def load_data(*args, **kwargs):
    dt = DeltaTable(
    's3://mage-demo-public/magic-energy/1337',
    storage_options={
    'AWS_ACCESS_KEY_ID': 'AKIAZ4SRK3YKRLA66XKN',
    'AWS_SECRET_ACCESS_KEY': '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',
    },
    )
    return dt.to_pandas()