Spark-a-thon - Development Activities

Structured Streaming

1. Developing a custom StreamSourceProvider
2. Migrating TextSocketStream to SparkSession (currently uses SQLContext)
3. Developing Sink and Source for Apache Kafka
4. JDBC support (with PostgreSQL as the database)

Spark SQL

8. Creating custom Encoder
   • SPARK-17668 Support representing structs with case classes and tuples in spark sql udf inputs
   • Create an encoder between your custom domain object of type T and JSON or CSV
   • See Encoders for available encoders.
   • Read Encoders - Internal Row Converters
   • (advanced/integration) Create an encoder for Apache Arrow (esp. after the arrow-0.1.0 RC0 release candidate has recently been announced) and ARROW-288 Implement Arrow adapter for Spark Datasets.
9. Custom format, i.e. spark.read.format(...) or spark.write.format(...)
10. Multiline JSON reader / writer
11. SQLQueryTestSuite - this is a very fresh thing in Spark 2.0 to write tests for Spark SQL

• Changelog

4. http://stackoverflow.com/questions/39073602/i-am-running-gbt-in-spark-ml-for-ctr-prediction-i-am-getting-exception-because
5. ExecutionListenerManager
6. (done) Developing a custom RuleExecutor and enabling it in Spark
   • Answering Extending Spark Catalyst optimizer with own rules on StackOverflow
   • Sparkathon - Developing Spark Extensions in Scala on Sep 28th

Spark MLlib

5. Creating custom Transformer

• Example: Tokenizer
• Jonatan + Kuba + lejdis (Justyna + Magda)
• Problem to zapis Pipeline z tym Transformera, odczyt i użycie.

8. Spark MLlib 2.0 Activator

Core

1. Monitoring executors (metrics, e.g. memory usage) using SparkListener.onExecutorMetricsUpdate.

Misc

1. Develop a new Scala-only TCP-based Apache Kafka client
   • A Guide To The Kafka Protocol
   • KAFKA-3360 Add a protocol page/section to the official Kafka documentation
   • See Scala Kafka Client for inspiration yet it's just "a thin Scala wrapper over the official Apache Kafka Java Driver"
2. Working on Issues reported in TensorFrames.
3. Review open issues in Spark's JIRA and pick one to work on.