1ambda/spark-conf.yaml

Created January 2, 2022 02:13

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/1ambda/b51e2d1bc627d6bbbd5700494174a435.js"></script>
Save 1ambda/b51e2d1bc627d6bbbd5700494174a435 to your computer and use it in GitHub Desktop.

Download ZIP

Revisions

1ambda created this gist Jan 2, 2022.

spark-conf.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,8 @@
+    spark.executor.instances = 10
+    spark.executor.cores = 10
+    spark.executor.memory = 30g (GiB)
+    spark.memory.memoryOverhead = 0.1
+    spark.memory.fraction = 0.8
+    spark.memory.storageFraction = 0.5
+    spark.memory.offHeap.enabled = false