Noah Kennedy noahfkennedy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {% macro get_table_exists(input_table, threshold) %} | |
| {# Try adapter to see if table exists #} | |
| {%- set table_exists = adapter.get_relation(database=var('project_id'), | |
| schema=test_schema, | |
| identifier=test_alias) is not none -%} | |
| {# If table doesn't exist, return FALSE #} | |
| {% if not table_exists %} | |
| {{ return(table_exists) }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {% macro generate_schema_name(custom_schema_name, node) -%} | |
| {%- set default_schema = target.schema -%} | |
| {%- if custom_schema_name is none -%} | |
| {{ default_schema }} | |
| {%- else -%} | |
| {#-- dbt defaults to concatenating the custom schema to the target schema --#} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {%- macro clean_date(dt) -%} | |
| {% if dt | length > 10 %} | |
| left({{ dt }}, 10) | |
| {% else %} | |
| {{ dt }} | |
| {% endif %} | |
| {%- endmacro -%} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {{ config( materialized='table', | |
| tags=['example_test'] | |
| ) | |
| }} | |
| select {{ example_macro(1, 2) }} as example_column |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {% macro example_macro(column1, column2) %} | |
| {{column1}} + {{column2}} | |
| {% endmacro %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyspark.sql import SparkSession | |
| import os | |
| import json | |
| # Setting spark session | |
| spark = SparkSession.builder \ | |
| .master("yarn") \ | |
| .config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED") \ | |
| .config("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED") \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| placement: | |
| managedCluster: | |
| clusterName: my-managed-cluster | |
| config: | |
| gceClusterConfig: | |
| zoneUri: us-central1-a | |
| jobs: | |
| - pysparkJob: | |
| fileUris: | |
| mainPythonFileUri: your_pyspark_script.py |
noahfkennedy
/ DataProc_Params.yaml
Last active
March 2, 2023 15:35
DataProc YAML Config - Parameters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| parameters: | |
| - name: MY_VAR1 | |
| fields: | |
| - jobs['STEP_NAME_123'].pysparkJob.properties['spark.executorEnv.MY_VAR1'] | |
| - name: MY_VAR2 | |
| fields: | |
| - jobs['STEP_NAME_123'].pysparkJob.properties['spark.executorEnv.MY_VAR2'] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| jobs: | |
| - pysparkJob: | |
| fileUris: | |
| mainPythonFileUri: your_pyspark_script.py | |
| properties: | |
| spark.executorEnv.MY_VAR1: 'default_value' | |
| spark.executorEnv.MY_VAR2: 'default_value' | |
| stepId: STEP_NAME_123 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| placement: | |
| managedCluster: | |
| clusterName: my-managed-cluster | |
| config: | |
| gceClusterConfig: | |
| zoneUri: us-central1-a |
NewerOlder