Skip to content

Instantly share code, notes, and snippets.

@julianthome

julianthome/doinparallel.sh

Last active October 5, 2016 08:40
Show Gist options
  • Save julianthome/161e6734c36611fcf03c91c9f76ebd5a to your computer and use it in GitHub Desktop.
Save julianthome/161e6734c36611fcf03c91c9f76ebd5a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. Julian Thomé revised this gist Oct 5, 2016. 1 changed file with 5 additions and 4 deletions.
    9 changes: 5 additions & 4 deletions doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -6,9 +6,6 @@ export FINALRES="result"
    export WPSIZE=5
    export JOBLOG="joblog"

    # setup fifos and files
    [ ! -e "${FINALRES}" ] && touch "${FINALRES}"

    function getfiles() {
    find ./* -name "*.txt"
    }
    @@ -24,7 +21,11 @@ function worker() {
    # make function worker known to gnu parallel
    export -f worker

    getfiles | parallel --joblog "${JOBLOG}" -n"${WPSIZE}" -j +0 worker >> "${FINALRES}"
    if [ ! -e "${JOBLOG}" ]; then
    getfiles | parallel --joblog "${JOBLOG}" -n"${WPSIZE}" -j +0 worker >> "${FINALRES}"
    else
    getfiles | parallel --resume --joblog "${JOBLOG}" -n"${WPSIZE}" -j +0 worker >> "${FINALRES}"
    fi

    echo "all jobs are finished"

  2. Julian Thomé revised this gist Oct 2, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -21,7 +21,7 @@ function worker() {
    exit 0
    }

    # make function worker known to gnu semaphore
    # make function worker known to gnu parallel
    export -f worker

    getfiles | parallel --joblog "${JOBLOG}" -n"${WPSIZE}" -j +0 worker >> "${FINALRES}"
  3. Julian Thomé revised this gist Oct 2, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,7 @@ function getfiles() {
    find ./* -name "*.txt"
    }

    function worker () {
    function worker() {
    [ ${#@} -eq 0 ] && exit 0
    for fil in "${@}"; do
    echo "process $fil"
  4. Julian Thomé revised this gist Oct 2, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -4,7 +4,7 @@
    # gnu parallel
    export FINALRES="result"
    export WPSIZE=5
    export JOBLOG="z3.log"
    export JOBLOG="joblog"

    # setup fifos and files
    [ ! -e "${FINALRES}" ] && touch "${FINALRES}"
  5. Julian Thomé revised this gist Oct 2, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    #!/bin/bash

    # make variables available in function started by
    # gnu semaphore
    # gnu parallel
    export FINALRES="result"
    export WPSIZE=5
    export JOBLOG="z3.log"
  6. Julian Thomé revised this gist Oct 2, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -16,7 +16,7 @@ function getfiles() {
    function worker () {
    [ ${#@} -eq 0 ] && exit 0
    for fil in "${@}"; do
    echo "process $fil"
    echo "process $fil"
    done
    exit 0
    }
  7. Julian Thomé created this gist Oct 2, 2016.
    31 changes: 31 additions & 0 deletions doinparallel.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,31 @@
    #!/bin/bash

    # make variables available in function started by
    # gnu semaphore
    export FINALRES="result"
    export WPSIZE=5
    export JOBLOG="z3.log"

    # setup fifos and files
    [ ! -e "${FINALRES}" ] && touch "${FINALRES}"

    function getfiles() {
    find ./* -name "*.txt"
    }

    function worker () {
    [ ${#@} -eq 0 ] && exit 0
    for fil in "${@}"; do
    echo "process $fil"
    done
    exit 0
    }

    # make function worker known to gnu semaphore
    export -f worker

    getfiles | parallel --joblog "${JOBLOG}" -n"${WPSIZE}" -j +0 worker >> "${FINALRES}"

    echo "all jobs are finished"

    exit 0