Skip to content

Instantly share code, notes, and snippets.

@ethiraj

ethiraj/csv_to_parquet.sh

Forked from mehd-io/csv_to_parquet.sh
Created March 21, 2023 21:14
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ethiraj/b416d25429d68016158160f638e2dcb1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ethiraj/b416d25429d68016158160f638e2dcb1 to your computer and use it in GitHub Desktop.
Download ZIP
Convert CSV to Parquet using DuckDB CLI
Raw
csv_to_parquet.sh
#!/bin/bash
# You can put this in your .bashrc or .zshrc
function csv_to_parquet() {
file_path="$1"
duckdb -c "COPY (SELECT * FROM read_csv_auto('$file_path')) TO '${file_path%.*}.parquet' (FORMAT PARQUET);"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment