I hereby claim:
- I am giefferre on github.
- I am giefferre (https://keybase.io/giefferre) on keybase.
- I have a public key ASC5MZ4LhY4fbsPqLY6kt-PA8ifPTGW5zEAt5P-jkDmLCwo
To claim this, I am signing this object:
Gianfranco Reppucci giefferre
giefferre
/ store_and_reuse_dataframe_schema.py
Last active
July 10, 2021 18:01
Save the schema of a Spark DataFrame to be able to reuse it when reading json files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # read a part of the whole datalake just to extract the schema | |
| part = spark.read.json("s3a://path/to/json/part") | |
| # create a temporary rdd in order to store the schema as binary file | |
| temp_rdd = sc.parallelize(part.schema) | |
| temp_rdd.coalesce(1).saveAsPickleFile("s3a://path/to/destination_schema.pickle") | |
| # from now on, the schema will be saved. | |
| # it could be used to improve the speed of reading json files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package main | |
| import ( | |
| "log" | |
| "log/syslog" | |
| ) | |
| func main() { | |
| logwriter, e := syslog.Dial("tcp","123.123.123.123:12345", syslog.LOG_DEBUG, "your.software.identifier") | |
| if e != nil { |