Skip to content

Instantly share code, notes, and snippets.

@findepi

findepi/iceberg-dev-cheat-sheet.md

Last active December 5, 2024 08:05
Show Gist options
  • Save findepi/da2ea85d095eaa0f3cfdcbb311ff59ed to your computer and use it in GitHub Desktop.
Save findepi/da2ea85d095eaa0f3cfdcbb311ff59ed to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. findepi revised this gist Dec 5, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    #### quick build

    ```bash
    ./gradlew -x test -x integrationTest build
    ./gradlew build -x test -x integrationTest -x spotlessJavaCheck -x checkstyleTest
    ```

    #### full build, like `mvn install`
  2. findepi revised this gist Aug 20, 2024. 1 changed file with 2 additions and 10 deletions.
    12 changes: 2 additions & 10 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,20 +10,12 @@ By default only modules for one Spark/Flink/Scala versions are built.
    To get the full build, component versions need to be enumerated.

    ```bash
    ./gradlew \
    -DallVersions=true \
    `# TODO drop scala versions after https://github.com/apache/iceberg/pull/10478` \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    ./gradlew -DallModules=true -x test -x integrationTest build publishToMavenLocal
    ```

    #### reformat code
    ```bash
    ./gradlew \
    -DallVersions=true \
    `# TODO drop scala versions after https://github.com/apache/iceberg/pull/10478` \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    spotlessApply
    ./gradlew -DallModules=true spotlessApply
    ```

    #### run a single Spark test class
  3. findepi revised this gist Jun 10, 2024. 1 changed file with 4 additions and 6 deletions.
    10 changes: 4 additions & 6 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,20 +11,18 @@ To get the full build, component versions need to be enumerated.

    ```bash
    ./gradlew \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    -DhiveVersions=$(sed -n 's/^systemProp.knownHiveVersions=//p' gradle.properties) \
    -DallVersions=true \
    `# TODO drop scala versions after https://github.com/apache/iceberg/pull/10478` \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    ```

    #### reformat code
    ```bash
    ./gradlew \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    -DhiveVersions=$(sed -n 's/^systemProp.knownHiveVersions=//p' gradle.properties) \
    -DallVersions=true \
    `# TODO drop scala versions after https://github.com/apache/iceberg/pull/10478` \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    spotlessApply
    ```

  4. findepi revised this gist Jun 10, 2024. 1 changed file with 6 additions and 4 deletions.
    10 changes: 6 additions & 4 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,18 +11,20 @@ To get the full build, component versions need to be enumerated.

    ```bash
    ./gradlew \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    -DhiveVersions=$(sed -n 's/^systemProp.knownHiveVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    ```

    #### reformat code
    ```bash
    ./gradlew \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    -DhiveVersions=$(sed -n 's/^systemProp.knownHiveVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    spotlessApply
    ```

  5. findepi revised this gist Jun 10, 2024. 1 changed file with 14 additions and 7 deletions.
    21 changes: 14 additions & 7 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,18 +5,25 @@
    ```

    #### full build, like `mvn install`
    ```bash
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal

    # ... or pull the list automatically
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    By default only modules for one Spark/Flink/Scala versions are built.
    To get the full build, component versions need to be enumerated.

    ```bash
    ./gradlew \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    ```

    #### reformat code
    ```bash
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply
    ./gradlew \
    -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -DscalaVersions=$(sed -n 's/^systemProp.knownScalaVersions=//p' gradle.properties) \
    -DflinkVersions=$(sed -n 's/^systemProp.knownFlinkVersions=//p' gradle.properties) \
    spotlessApply
    ```

    #### run a single Spark test class
  6. findepi revised this gist Jun 10, 2024. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,9 @@
    #### quick build

    ```bash
    ./gradlew -x test -x integrationTest build
    ```

    #### full build, like `mvn install`
    ```bash
    # See knownSparkVersions in gradle.properties for current list Spark versions
  7. findepi revised this gist Aug 29, 2022. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,7 +23,8 @@

    #### run individual Spark test methods
    ```bash
    # Note that * at the end of the method name is required
    ./gradlew -DsparkVersions=3.3 :iceberg-spark:iceberg-spark-extensions:test \
    --tests TestRemoveOrphanFilesProcedure.testRemoveOrphanFilesWithStatisticFiles\* `# * is required` \
    --tests TestRemoveOrphanFilesAction3.testRemoveOrphanFilesWithStatisticFiles\* `# * is required`
    --tests TestRemoveOrphanFilesProcedure.testRemoveOrphanFilesWithStatisticFiles\* \
    --tests TestRemoveOrphanFilesAction3.testRemoveOrphanFilesWithStatisticFiles\*
    ```
  8. findepi revised this gist Aug 29, 2022. 1 changed file with 9 additions and 2 deletions.
    11 changes: 9 additions & 2 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,10 +13,17 @@
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply
    ```

    ### run a single Spark test
    #### run a single Spark test class
    ```bash
    ./gradlew :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure

    # or with non-default Spark version
    ./gradlew -DsparkVersions=3.2 :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure
    ```
    ```

    #### run individual Spark test methods
    ```bash
    ./gradlew -DsparkVersions=3.3 :iceberg-spark:iceberg-spark-extensions:test \
    --tests TestRemoveOrphanFilesProcedure.testRemoveOrphanFilesWithStatisticFiles\* `# * is required` \
    --tests TestRemoveOrphanFilesAction3.testRemoveOrphanFilesWithStatisticFiles\* `# * is required`
    ```
  9. findepi revised this gist Aug 29, 2022. 1 changed file with 1 addition and 2 deletions.
    3 changes: 1 addition & 2 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,8 +10,7 @@

    #### reformat code
    ```bash
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    spotlessApply
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply
    ```

    ### run a single Spark test
  10. findepi revised this gist Aug 29, 2022. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -12,4 +12,12 @@
    ```bash
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    spotlessApply
    ```

    ### run a single Spark test
    ```bash
    ./gradlew :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure

    # or with non-default Spark version
    ./gradlew -DsparkVersions=3.2 :iceberg-spark:iceberg-spark-extensions:test --tests TestRemoveOrphanFilesProcedure
    ```
  11. findepi revised this gist Aug 29, 2022. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,5 +10,6 @@

    #### reformat code
    ```bash
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    spotlessApply
    ```
  12. findepi revised this gist Aug 29, 2022. 1 changed file with 1 addition and 5 deletions.
    6 changes: 1 addition & 5 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,9 +10,5 @@

    #### reformat code
    ```bash
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 :iceberg-api:spotlessApply :iceberg-core:spotlessApply \
    :iceberg-spark:iceberg-spark-2.4:spotlessApply :iceberg-spark:iceberg-spark-3.0:spotlessApply \
    :iceberg-spark:iceberg-spark-3.1:spotlessApply :iceberg-spark:iceberg-spark-3.2:spotlessApply \
    :iceberg-spark:iceberg-spark-3.3:spotlessApply
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) spotlessApply
    ```
  13. findepi revised this gist Aug 29, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    #### full build, like `mvn install`
    ```
    ```bash
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal

    @@ -9,7 +9,7 @@
    ```

    #### reformat code
    ```
    ```bash
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 :iceberg-api:spotlessApply :iceberg-core:spotlessApply \
    :iceberg-spark:iceberg-spark-2.4:spotlessApply :iceberg-spark:iceberg-spark-3.0:spotlessApply \
  14. findepi revised this gist Aug 29, 2022. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,6 +2,10 @@
    ```
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal
    # ... or pull the list automatically
    ./gradlew -DsparkVersions=$(sed -n 's/^systemProp.knownSparkVersions=//p' gradle.properties) \
    -x test -x integrationTest build publishToMavenLocal
    ```

    #### reformat code
  15. findepi revised this gist Aug 29, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,10 +1,10 @@
    Full build, like `mvn install`
    #### full build, like `mvn install`
    ```
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal
    ```

    Reformat code
    #### reformat code
    ```
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 :iceberg-api:spotlessApply :iceberg-core:spotlessApply \
  16. findepi created this gist Aug 29, 2022.
    14 changes: 14 additions & 0 deletions iceberg-dev-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,14 @@
    Full build, like `mvn install`
    ```
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 -x test -x integrationTest build publishToMavenLocal
    ```

    Reformat code
    ```
    # See knownSparkVersions in gradle.properties for current list Spark versions
    ./gradlew -DsparkVersions=2.4,3.0,3.1,3.2,3.3 :iceberg-api:spotlessApply :iceberg-core:spotlessApply \
    :iceberg-spark:iceberg-spark-2.4:spotlessApply :iceberg-spark:iceberg-spark-3.0:spotlessApply \
    :iceberg-spark:iceberg-spark-3.1:spotlessApply :iceberg-spark:iceberg-spark-3.2:spotlessApply \
    :iceberg-spark:iceberg-spark-3.3:spotlessApply
    ```