Skip to content

Instantly share code, notes, and snippets.

@DecisionNerd

DecisionNerd/csvLogstash.sh

Last active November 13, 2015 02:37
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save DecisionNerd/e42d51af0679acb8a2ad to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save DecisionNerd/e42d51af0679acb8a2ad to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. DecisionNerd revised this gist Nov 13, 2015. 1 changed file with 5 additions and 2 deletions.
    7 changes: 5 additions & 2 deletions csvLogstash.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,9 @@
    # first line of the file. This script processes csv files in the directory
    # where it is executed. Script runs continuously until stopping with ctrl+Z

    #specify directory to send files for ingest (without trailing /)
    DEST=/home/user/Test/forIngest

    echo "processing csv files in $(pwd) for ingest by logstash..."
    while true
    do
    @@ -16,9 +19,9 @@ do
    echo "removing header row of $file"
    # remove the first line in the csv file
    sed -i 1d $file
    echo "moving edited $file to /home/user/Test/forIngest"
    echo "moving edited $file to $DEST"
    # move file to where logstash will grab
    mv $file /home/user/Test/forIngest
    mv $file $DEST
    echo "waiting on csv files..."
    # only run command every second
    sleep 1
  2. DecisionNerd created this gist Nov 13, 2015.
    28 changes: 28 additions & 0 deletions csvLogstash.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,28 @@
    #!/bin/bash

    # ingesting csv files into logstash requires removal of the header row in the
    # first line of the file. This script processes csv files in the directory
    # where it is executed. Script runs continuously until stopping with ctrl+Z

    echo "processing csv files in $(pwd) for ingest by logstash..."
    while true
    do
    # if there are csv files in the current directory
    if [ -a $(pwd)/*.csv ]
    then
    # select each csv file in the current directory
    for file in $(pwd)/*.csv
    do
    echo "removing header row of $file"
    # remove the first line in the csv file
    sed -i 1d $file
    echo "moving edited $file to /home/user/Test/forIngest"
    # move file to where logstash will grab
    mv $file /home/user/Test/forIngest
    echo "waiting on csv files..."
    # only run command every second
    sleep 1
    done
    #else
    fi
    done