2020-08-15 01:10:54,597 DEBUG MainThread:141411 [wandb_config.py:_load_defaults():154] no defaults not found in config-defaults.yaml 2020-08-15 01:10:54,747 DEBUG MainThread:141411 [meta.py:_setup_code_git():49] probe for git information 2020-08-15 01:10:54,986 DEBUG MainThread:141411 [meta.py:setup():104] code probe starting 2020-08-15 01:10:54,987 DEBUG MainThread:141411 [meta.py:_setup_code_program():58] save program starting 2020-08-15 01:10:54,987 DEBUG MainThread:141411 [meta.py:_setup_code_program():60] save program starting: /home/jramapuram/sshfs/kanerva_plus_plus/./main.py 2020-08-15 01:10:54,999 DEBUG MainThread:141411 [meta.py:_setup_code_program():68] save program saved: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/code/main.py 2020-08-15 01:10:55,003 DEBUG MainThread:141411 [meta.py:_setup_code_program():70] save program 2020-08-15 01:10:55,036 DEBUG MainThread:141411 [meta.py:setup():124] code probe done 2020-08-15 01:10:55,084 DEBUG MainThread:141411 [run_manager.py:__init__():547] Initialized sync for kanerva_plus_plus/2kcpmyzj 2020-08-15 01:10:55,102 INFO MainThread:141411 [run_manager.py:wrap_existing_process():1149] wrapping existing process 141328 2020-08-15 01:10:55,103 WARNING MainThread:141411 [io_wrap.py:register():108] SIGWINCH handler was not None: 2020-08-15 01:10:55,247 INFO MainThread:141411 [run_manager.py:init_run():928] system metrics and metadata threads started 2020-08-15 01:10:55,248 INFO MainThread:141411 [run_manager.py:init_run():967] upserting run before process can begin, waiting at most 10 seconds 2020-08-15 01:10:55,389 INFO Thread-15 :141411 [run_manager.py:_upsert_run():1052] saving patches 2020-08-15 01:10:55,762 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/config.yaml 2020-08-15 01:11:04,313 ERROR Thread-15 :141411 [internal.py:save_patches():210] Error generating diff: Command '['git', 'diff', '--submodule=diff', '3d9e8cf427f8b8d1868fa378e64a5ca4c8afdfe9']' timed out after 5 seconds 2020-08-15 01:11:04,314 INFO Thread-15 :141411 [run_manager.py:_upsert_run():1056] saving pip packages 2020-08-15 01:11:04,334 INFO Thread-15 :141411 [run_manager.py:_upsert_run():1058] initializing streaming files api 2020-08-15 01:11:04,336 INFO Thread-15 :141411 [run_manager.py:_upsert_run():1065] unblocking file change observer, beginning sync with W&B servers 2020-08-15 01:11:04,337 INFO MainThread:141411 [run_manager.py:wrap_existing_process():1166] informing user process we are ready to proceed 2020-08-15 01:11:04,344 INFO MainThread:141411 [run_manager.py:_sync_etc():1298] entering loop for messages from user process 2020-08-15 01:11:04,398 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:04,399 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-metadata.json 2020-08-15 01:11:04,399 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/code/main.py 2020-08-15 01:11:04,400 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/code 2020-08-15 01:11:04,400 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/diff.patch 2020-08-15 01:11:04,401 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/diff.patch 2020-08-15 01:11:04,401 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/upstream_diff_3d9e8cf427f8b8d1868fa378e64a5ca4c8afdfe9.patch 2020-08-15 01:11:04,406 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/upstream_diff_3d9e8cf427f8b8d1868fa378e64a5ca4c8afdfe9.patch 2020-08-15 01:11:04,831 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/upstream_diff_3d9e8cf427f8b8d1868fa378e64a5ca4c8afdfe9.patch 2020-08-15 01:11:04,832 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:04,833 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/config.yaml 2020-08-15 01:11:04,834 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/requirements.txt 2020-08-15 01:11:11,847 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-metadata.json 2020-08-15 01:11:12,880 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:12,881 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/graph/graph_0_summary_28e98832.graph.json 2020-08-15 01:11:12,882 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:12,911 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/graph 2020-08-15 01:11:12,912 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media 2020-08-15 01:11:14,892 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:15,901 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:17,914 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:19,942 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:19,943 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_read-traces-minibatch0_1_c5108def.png 2020-08-15 01:11:19,944 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_input_1_683fe2ec.png 2020-08-15 01:11:19,945 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_memory_1_49033288.png 2020-08-15 01:11:19,946 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_reconstruction_1_e4b6f0d0.png 2020-08-15 01:11:19,947 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media 2020-08-15 01:11:19,948 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:20,960 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-history.jsonl 2020-08-15 01:11:20,963 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_reconstruction_1_ed252709.png 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mean-perturbed-generated_1_491d352c.png 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_input_1_5a4bb46f.png 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_memory_1_4c75315f.png 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mosiac-generated_1_c061bdfe.png 2020-08-15 01:11:20,974 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_read-traces-minibatch0_1_12312dca.png 2020-08-15 01:11:20,975 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mean-perturbed-generated_1_043b279d.png 2020-08-15 01:11:20,975 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mosiac-generated_1_4befb5d2.png 2020-08-15 01:11:20,975 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:23,976 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:23,977 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-events.jsonl 2020-08-15 01:11:25,996 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-history.jsonl 2020-08-15 01:11:25,997 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:25,997 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:26,005 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_read-traces-minibatch0_2_bc2cdc4d.png 2020-08-15 01:11:26,005 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mosiac-generated_2_3c8dc0f9.png 2020-08-15 01:11:26,005 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_input_2_f4d6518c.png 2020-08-15 01:11:26,006 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_memory_2_e1a020e8.png 2020-08-15 01:11:26,006 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_reconstruction_2_d6b38c13.png 2020-08-15 01:11:26,006 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_input_2_5a4bb46f.png 2020-08-15 01:11:26,006 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mean-perturbed-generated_2_c414b854.png 2020-08-15 01:11:26,006 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mean-perturbed-generated_2_7e71184e.png 2020-08-15 01:11:26,007 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_read-traces-minibatch0_2_6afea02d.png 2020-08-15 01:11:26,007 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_memory_2_061862ff.png 2020-08-15 01:11:26,007 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mosiac-generated_2_39566a9b.png 2020-08-15 01:11:26,007 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_reconstruction_2_85dd3043.png 2020-08-15 01:11:26,007 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:28,007 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-metadata.json 2020-08-15 01:11:29,013 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:31,039 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-history.jsonl 2020-08-15 01:11:31,040 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:31,040 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:31,049 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_input_3_5a4bb46f.png 2020-08-15 01:11:31,049 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_read-traces-minibatch0_3_10a1678f.png 2020-08-15 01:11:31,049 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_reconstruction_3_4dc87bf9.png 2020-08-15 01:11:31,049 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_read-traces-minibatch0_3_b62e3529.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_memory_3_2ece532a.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_reconstruction_3_7b70b2f2.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mosiac-generated_3_4999ff5a.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mosiac-generated_3_586a2788.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mean-perturbed-generated_3_f9f9bb79.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_memory_3_0de8d7ab.png 2020-08-15 01:11:31,050 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_input_3_83960a2f.png 2020-08-15 01:11:31,051 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mean-perturbed-generated_3_cb126348.png 2020-08-15 01:11:31,051 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:34,056 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:36,101 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/output.log 2020-08-15 01:11:36,102 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_reconstruction_4_dd881ea5.png 2020-08-15 01:11:36,103 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_input_4_098748ce.png 2020-08-15 01:11:36,104 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_reconstruction_4_b997b6a7.png 2020-08-15 01:11:36,105 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_memory_4_7a43be3a.png 2020-08-15 01:11:36,107 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mean-perturbed-generated_4_58355424.png 2020-08-15 01:11:36,108 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_read-traces-minibatch0_4_b58a9e29.png 2020-08-15 01:11:36,109 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_memory_4_2380023d.png 2020-08-15 01:11:36,110 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_mosiac-generated_4_851c2d7c.png 2020-08-15 01:11:36,111 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/train_read-traces-minibatch0_4_1a136185.png 2020-08-15 01:11:36,112 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:36,644 INFO MainThread:141411 [run_manager.py:_sync_etc():1355] process received interrupt signal, shutting down 2020-08-15 01:11:36,645 INFO MainThread:141411 [run_manager.py:_sync_etc():1408] closing log streams and sending exitcode to W&B 2020-08-15 01:11:36,645 INFO MainThread:141411 [run_manager.py:shutdown():1072] shutting down system stats and metadata service 2020-08-15 01:11:37,135 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-history.jsonl 2020-08-15 01:11:37,136 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-events.jsonl 2020-08-15 01:11:37,137 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-summary.json 2020-08-15 01:11:37,146 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mean-perturbed-generated_4_d58e4dd5.png 2020-08-15 01:11:37,147 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_input_4_5a4bb46f.png 2020-08-15 01:11:37,148 INFO Thread-4 :141411 [run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mosiac-generated_4_437fed77.png 2020-08-15 01:11:37,150 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:37,336 INFO MainThread:141411 [run_manager.py:shutdown():1086] stopping streaming files and file change observer 2020-08-15 01:11:38,149 INFO Thread-4 :141411 [run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-metadata.json 2020-08-15 01:11:40,034 INFO MainThread:141411 [run_manager.py:_sync_etc():1427] rendering summary 2020-08-15 01:11:40,045 INFO MainThread:141411 [run_manager.py:_sync_etc():1459] syncing files to cloud storage 2020-08-15 01:11:40,300 INFO MainThread:141411 [run_manager.py:_sync_etc():1468] syncing complete: http://devbox:8080/jramapuram/kanerva_plus_plus/runs/2kcpmyzj 8 INFO Thread-4 :141411 [2kcpmyzj:run_manager.py:_on_file_created():681] file/dir created: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images/test_mosiac-generated_4_437fed77.png 2020-08-15 01:11:37,150 INFO Thread-4 :141411 [2kcpmyzj:run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/media/images 2020-08-15 01:11:37,336 INFO MainThread:141411 [2kcpmyzj:run_manager.py:shutdown():1086] stopping streaming files and file change observer 2020-08-15 01:11:38,149 INFO Thread-4 :141411 [2kcpmyzj:run_manager.py:_on_file_modified():692] file/dir modified: /home/jramapuram/sshfs/kanerva_plus_plus/wandb/run-20200815_011054-2kcpmyzj/wandb-metadata.json 2020-08-15 01:11:40,034 INFO MainThread:141411 [2kcpmyzj:run_manager.py:_sync_etc():1427] rendering summary 2020-08-15 01:11:40,045 INFO MainThread:141411 [2kcpmyzj:run_manager.py:_sync_etc():1459] syncing files to cloud storage 2020-08-15 01:11:40,300 INFO MainThread:141411 [2kcpmyzj:run_manager.py:_sync_etc():1468] syncing complete: http://devbox:8080/jramapuram/kanerva_plus_plus/runs/2kcpmyzj