Skip to content

Instantly share code, notes, and snippets.

@gmgitx
Created September 19, 2018 02:56
Show Gist options
  • Save gmgitx/db14eea62974973e336de4ff1f90c43c to your computer and use it in GitHub Desktop.
Save gmgitx/db14eea62974973e336de4ff1f90c43c to your computer and use it in GitHub Desktop.

Revisions

  1. gmgitx created this gist Sep 19, 2018.
    236 changes: 236 additions & 0 deletions slurm-49805362.out
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,236 @@
    ```
    [2018-09-19 02:34:46,62] [info] Running with database db.url = jdbc:hsqldb:mem:5c60b7f9-5aef-406c-9ad2-9ff13776de58;shutdown=false;hsqldb.tx=mvcc
    [2018-09-19 02:35:04,66] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
    [2018-09-19 02:35:04,69] [info] [RenameWorkflowOptionsInMetadata] 100%
    [2018-09-19 02:35:05,18] [info] Running with database db.url = jdbc:hsqldb:mem:2dbdec8e-bc18-4085-b951-aaf4371a1be5;shutdown=false;hsqldb.tx=mvcc
    [2018-09-19 02:35:05,81] [warn] This actor factory is deprecated. Please use cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory for PAPI v1 or cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory for PAPI v2
    [2018-09-19 02:35:05,86] [warn] Couldn't find a suitable DSN, defaulting to a Noop one.
    [2018-09-19 02:35:05,87] [info] Using noop to send events.
    [2018-09-19 02:35:06,19] [info] Slf4jLogger started
    [2018-09-19 02:35:06,47] [info] Workflow heartbeat configuration:
    {
    "cromwellId" : "cromid-c4f43a3",
    "heartbeatInterval" : "2 minutes",
    "ttl" : "10 minutes",
    "writeBatchSize" : 10000,
    "writeThreshold" : 10000
    }
    [2018-09-19 02:35:06,72] [info] Metadata summary refreshing every 2 seconds.
    [2018-09-19 02:35:06,98] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
    [2018-09-19 02:35:06,99] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
    [2018-09-19 02:35:07,02] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
    [2018-09-19 02:35:10,60] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
    [2018-09-19 02:35:10,68] [info] SingleWorkflowRunnerActor: Version 34
    [2018-09-19 02:35:10,69] [info] SingleWorkflowRunnerActor: Submitting workflow
    [2018-09-19 02:35:10,78] [info] PAPIQueryManager Running with 3 workers
    [2018-09-19 02:35:10,78] [info] JES batch polling interval is 33333 milliseconds
    [2018-09-19 02:35:10,79] [info] JES batch polling interval is 33333 milliseconds
    [2018-09-19 02:35:10,87] [info] JES batch polling interval is 33333 milliseconds
    [2018-09-19 02:35:10,87] [info] Unspecified type (Unspecified version) workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 submitted
    [2018-09-19 02:35:11,00] [info] SingleWorkflowRunnerActor: Workflow submitted 2a0ed041-872c-4e5f-b2b5-54735758a707
    [2018-09-19 02:35:11,01] [info] 1 new workflows fetched
    [2018-09-19 02:35:11,01] [info] WorkflowManagerActor Starting workflow 2a0ed041-872c-4e5f-b2b5-54735758a707
    [2018-09-19 02:35:11,02] [info] WorkflowManagerActor Successfully started WorkflowActor-2a0ed041-872c-4e5f-b2b5-54735758a707
    [2018-09-19 02:35:11,02] [info] Retrieved 1 workflows from the WorkflowStoreActor
    [2018-09-19 02:35:11,04] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
    [2018-09-19 02:35:11,05] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
    [2018-09-19 02:35:11,19] [info] MaterializeWorkflowDescriptorActor [2a0ed041]: Parsing workflow as WDL draft-2
    [2018-09-19 02:35:31,05] [info] MaterializeWorkflowDescriptorActor [2a0ed041]: Call-to-Backend assignments: atac.bowtie2 -> slurm, atac.ataqc -> slurm, atac.macs2_pooled -> slurm, atac.idr_pr -> slurm, atac.overlap -> slurm, atac.bam2ta -> slurm, atac.macs2_pr2 -> slurm, atac.xcor -> slurm, atac.filter -> slurm, atac.macs2 -> slurm, atac.pool_ta -> slurm, atac.reproducibility_idr -> slurm, atac.spr -> slurm, atac.read_genome_tsv -> slurm, atac.macs2_ppr2 -> slurm, atac.macs2_pr1 -> slurm, atac.qc_report -> slurm, atac.idr_ppr -> slurm, atac.pool_ta_pr2 -> slurm, atac.overlap_ppr -> slurm, atac.trim_adapter -> slurm, atac.idr -> slurm, atac.pool_ta_pr1 -> slurm, atac.overlap_pr -> slurm, atac.reproducibility_overlap -> slurm, atac.macs2_ppr1 -> slurm
    [2018-09-19 02:35:31,40] [warn] slurm [2a0ed041]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,43] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,44] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,45] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,48] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,49] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,49] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
    [2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.read_genome_tsv
    [2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: 'enable_idr'. Running conditional section
    [2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!align_only && !true_rep_only && enable_idr'. Running conditional section
    [2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!align_only && !true_rep_only'. Running conditional section
    [2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!disable_xcor'. Running conditional section
    [2018-09-19 02:35:33,83] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: 'enable_idr'. Running conditional section
    [2018-09-19 02:35:33,84] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!true_rep_only'. Running conditional section
    [2018-09-19 02:35:34,71] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks
    [2018-09-19 02:35:36,66] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: cat /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/inputs/378634365/hg19.tsv
    [2018-09-19 02:35:36,72] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: executing: sbatch \
    --export=ALL \
    -J cromwell_2a0ed041_read_genome_tsv \
    -D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv \
    -o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/stdout \
    -e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/stderr \
    -t 60 \
    -n 1 \
    --ntasks-per-node=1 \
    --cpus-per-task=1 \
    --mem=4000 \
    -p broadwl \
    \
    --wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/script"
    [2018-09-19 02:35:37,08] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: job id: 49805408
    [2018-09-19 02:35:37,09] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCodeFile
    [2018-09-19 02:35:38,60] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.trim_adapter (2 shards)
    [2018-09-19 02:35:38,63] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Unrecognized runtime attribute keys: disks
    [2018-09-19 02:35:38,64] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Unrecognized runtime attribute keys: disks
    [2018-09-19 02:35:38,94] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: python $(which encode_trim_adapter.py) \
    /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/write_tsv_da8046fb49e6a4af98b0ba5e9b4e96ab.tmp \
    --adapters /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --min-trim-len 5 \
    --err-rate 0.1 \
    --nth 1
    [2018-09-19 02:35:38,95] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: python $(which encode_trim_adapter.py) \
    /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/write_tsv_0052698ca1a6bc6408797374c5d078e3.tmp \
    --adapters /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --min-trim-len 5 \
    --err-rate 0.1 \
    --nth 1
    [2018-09-19 02:35:38,98] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: executing: sbatch \
    --export=ALL \
    -J cromwell_2a0ed041_trim_adapter \
    -D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0 \
    -o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/stdout \
    -e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/stderr \
    -t 1440 \
    -n 1 \
    --ntasks-per-node=1 \
    --cpus-per-task=1 \
    --mem=12000 \
    -p broadwl \
    \
    --wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/script"
    [2018-09-19 02:35:38,98] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: executing: sbatch \
    --export=ALL \
    -J cromwell_2a0ed041_trim_adapter \
    -D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1 \
    -o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/stdout \
    -e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/stderr \
    -t 1440 \
    -n 1 \
    --ntasks-per-node=1 \
    --cpus-per-task=1 \
    --mem=12000 \
    -p broadwl \
    \
    --wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/script"
    [2018-09-19 02:35:42,02] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: job id: 49805411
    [2018-09-19 02:35:42,03] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: job id: 49805410
    [2018-09-19 02:35:42,12] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile
    [2018-09-19 02:35:42,13] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Status change from - to WaitingForReturnCodeFile
    [2018-09-19 02:36:15,21] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done
    [2018-09-19 02:37:40,90] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done
    [2018-09-19 02:37:42,15] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.bowtie2
    [2018-09-19 02:37:42,62] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: Unrecognized runtime attribute keys: preemptible, disks
    [2018-09-19 02:37:42,67] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: python $(which encode_bowtie2.py) \
    /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/646531885/male.hg19.fa.tar \
    /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/362506696/merge_fastqs_R1_ENCFF341MYG.subsampled.400.trim.merged.fastq.gz /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/362506696/merge_fastqs_R2_ENCFF248EJF.subsampled.400.trim.merged.fastq.gz \
    --paired-end \
    --multimapping 4 \
    \
    --nth 2
    [2018-09-19 02:37:42,72] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: executing: sbatch \
    --export=ALL \
    -J cromwell_2a0ed041_bowtie2 \
    -D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0 \
    -o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stdout \
    -e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stderr \
    -t 2880 \
    -n 1 \
    --ntasks-per-node=1 \
    --cpus-per-task=2 \
    --mem=16000 \
    -p broadwl \
    \
    --wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/script"
    [2018-09-19 02:38:01,41] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Status change from WaitingForReturnCodeFile to Done
    [2018-09-19 02:38:02,71] [error] WorkflowManagerActor Workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 failed (during ExecutingWorkflowState): java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stderr.submit
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:131)
    at scala.util.Either.fold(Either.scala:188)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:126)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
    at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:208)
    at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:600)
    at scala.util.Try$.apply(Try.scala:209)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:600)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:600)
    at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:208)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:915)
    at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:907)
    at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:208)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
    at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:208)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
    at akka.actor.ActorCell.invoke(ActorCell.scala:557)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
    at akka.dispatch.Mailbox.run(Mailbox.scala:225)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

    [2018-09-19 02:38:02,83] [info] WorkflowManagerActor WorkflowActor-2a0ed041-872c-4e5f-b2b5-54735758a707 is in a terminal state: WorkflowFailedState
    [2018-09-19 02:38:27,85] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
    [2018-09-19 02:38:32,09] [info] Workflow polling stopped
    [2018-09-19 02:38:32,20] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
    [2018-09-19 02:38:32,20] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
    [2018-09-19 02:38:32,25] [info] Aborting all running workflows.
    [2018-09-19 02:38:32,46] [info] WorkflowStoreActor stopped
    [2018-09-19 02:38:32,51] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
    [2018-09-19 02:38:32,51] [info] WorkflowLogCopyRouter stopped
    [2018-09-19 02:38:32,51] [info] JobExecutionTokenDispenser stopped
    [2018-09-19 02:38:32,51] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
    [2018-09-19 02:38:32,51] [info] Connection pools shut down
    [2018-09-19 02:38:32,52] [info] WorkflowManagerActor stopped
    [2018-09-19 02:38:32,52] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] Shutting down IoProxy - Timeout = 1800 seconds
    [2018-09-19 02:38:32,52] [info] SubWorkflowStoreActor stopped
    [2018-09-19 02:38:32,52] [info] WorkflowManagerActor All workflows finished
    [2018-09-19 02:38:32,55] [info] DockerHashActor stopped
    [2018-09-19 02:38:32,55] [info] IoProxy stopped
    [2018-09-19 02:38:32,55] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
    [2018-09-19 02:38:32,55] [info] KvWriteActor Shutting down: 0 queued messages to process
    [2018-09-19 02:38:32,55] [info] JobStoreActor stopped
    [2018-09-19 02:38:32,55] [info] CallCacheWriteActor stopped
    [2018-09-19 02:38:32,55] [info] WriteMetadataActor Shutting down: 0 queued messages to process
    [2018-09-19 02:38:32,56] [info] ServiceRegistryActor stopped
    [2018-09-19 02:38:32,63] [info] Database closed
    [2018-09-19 02:38:32,63] [info] Stream materializer shut down
    Workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 transitioned to state Failed
    [2018-09-19 02:38:32,73] [info] Automatic shutdown of the async connection
    [2018-09-19 02:38:32,73] [info] Gracefully shutdown sentry threads.
    [2018-09-19 02:38:32,73] [info] Shutdown finished.
    ```