Skip to content

Instantly share code, notes, and snippets.

@dotneet

dotneet/file0.scala

Last active October 24, 2016 06:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dotneet/c8f39836bc9fd87dc742 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dotneet/c8f39836bc9fd87dc742 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. dotneet revised this gist Oct 24, 2016. No changes.
  2. dotneet revised this gist Jun 27, 2015. No changes.
  3. dotneet revised this gist Jun 27, 2015. No changes.
  4. dotneet revised this gist Jun 27, 2015. No changes.
  5. dotneet revised this gist Jun 27, 2015. No changes.
  6. dotneet created this gist Jun 27, 2015.
    3 changes: 3 additions & 0 deletions file0.scala
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    sqlContext.setConf("spark.sql.parquet.compression.codec", "snappy")
    val jsonDataFrame = sqlContext.read.json(jsonFilePath)
    val jsonDataFrame.select("id","type","arg1","arg2","arg3","time").write.format("parquet").save(parquetPath)
    3 changes: 3 additions & 0 deletions file1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    spark.serializer org.apache.spark.serializer.KryoSerializer
    spark.executor.memory 6g
    spark.driver.memory 2g