Skip to content

Instantly share code, notes, and snippets.

@sirselim

sirselim/basecalling_notes.md

Last active August 16, 2024 07:37
Show Gist options
  • Save sirselim/2ebe2807112fae93809aa18f096dbb94 to your computer and use it in GitHub Desktop.
Save sirselim/2ebe2807112fae93809aa18f096dbb94 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. sirselim revised this gist Feb 4, 2020. No changes.
  2. sirselim revised this gist Feb 4, 2020. No changes.
  3. sirselim revised this gist Feb 4, 2020. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -276,6 +276,8 @@ Running the Xavier in different power states obviously influences the amount of
    | 30W 4 cores | 10.8 mins |
    | 30W MAX (8 cores) | 7.5 mins |

    The above benchmarks were performed on data generated from a flongle run (~0.5 Mb of sequence or 5.5 Gb of actual data).

    -----

    ## potential V100 examples
  4. sirselim revised this gist Feb 4, 2020. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -253,10 +253,15 @@ The below parameters seem to provide the 'optimal' speed increase with a resulta

    We are currently using a 27000mAh AC Portable Charger from Ravpower.

    Below: Ravpower Xtreme Series 27000mAh AC Portable Charger
    ![ravpower battery package](https://user-images.githubusercontent.com/5932864/73702469-c59b5280-46e4-11ea-9ef2-9b44f2f9cfb5.jpg)
    This battery bank/charger has a built in 220V AC outlet and 1 usb-c and 2 usb 3.1 outputs.

    Below: powerbank charging from the wall.
    ![ravpower battery on charge](https://user-images.githubusercontent.com/5932864/73702482-cb913380-46e4-11ea-84e5-3da374016481.jpg)
    Ravpower claim this powerbank will charge a smartphone 11 times, a tablet 4 times or a laptop 3 times.

    Below: running our first portable Xavier GPU basecalling of nanopore data!
    ![xavier running on battery power](https://user-images.githubusercontent.com/5932864/73702499-d21fab00-46e4-11ea-8789-ac26c17855f9.jpg)

    ### changing power modes
  5. sirselim revised this gist Feb 4, 2020. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -253,11 +253,11 @@ The below parameters seem to provide the 'optimal' speed increase with a resulta

    We are currently using a 27000mAh AC Portable Charger from Ravpower.

    ![](/photos/battery_case.jpg)
    ![ravpower battery package](https://user-images.githubusercontent.com/5932864/73702469-c59b5280-46e4-11ea-9ef2-9b44f2f9cfb5.jpg)

    ![](/photos/battery_case.jpg)
    ![ravpower battery on charge](https://user-images.githubusercontent.com/5932864/73702482-cb913380-46e4-11ea-84e5-3da374016481.jpg)

    ![](xavier_in_use.jpg)
    ![xavier running on battery power](https://user-images.githubusercontent.com/5932864/73702499-d21fab00-46e4-11ea-8789-ac26c17855f9.jpg)

    ### changing power modes

  6. sirselim revised this gist Feb 4, 2020. 1 changed file with 24 additions and 0 deletions.
    24 changes: 24 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -249,6 +249,30 @@ The below parameters seem to provide the 'optimal' speed increase with a resulta

    -----

    ## exploring portable batteries and power modes

    We are currently using a 27000mAh AC Portable Charger from Ravpower.

    ![](/photos/battery_case.jpg)

    ![](/photos/battery_case.jpg)

    ![](xavier_in_use.jpg)

    ### changing power modes

    Running the Xavier in different power states obviously influences the amount of run time on the battery.

    | Power mode | Time |
    | -------- | -------- |
    | 10W | 33.4 mins |
    | 15W | 14.3 mins |
    | 30W 2 cores | 10.8 mins |
    | 30W 4 cores | 10.8 mins |
    | 30W MAX (8 cores) | 7.5 mins |

    -----

    ## potential V100 examples

    ##### V100 config example for high accuracy model
  7. sirselim revised this gist Dec 13, 2019. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -329,7 +329,7 @@ For each Guppy version I ran the basecaller three times in an attempt to ensure
    | 3.4.3 (no vbz compressed fast5) | 90.674 | 4.37862e+07 |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files.*
    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files (pre 3.4.X from memory).*

    ### Summary to date

    @@ -338,7 +338,7 @@ I initially thought that there was something off with the compression imlementat
    What was quite interesting is that running on vbz compressed fast5 data appears to be in the range of 8-10 seconds faster than uncompressed. So there is a slight added speed benefit on top of the nice reduction in file size - which is a little nicer for the SSD/HDD.

    So at this stage I can't confirm any detrimental speed issues when using Guppy version 3.4.X, but this needs to be caveated with all the usual disclaimers:
    * all systems are different (I'm not on Unbuntu for instance).
    * all systems are different (I'm not on Ubuntu for instance).
    * drivers are different (I need to update).
    * GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
    * For what it's worth, I can add a comment here saying that I haven't had any speed issues with basecalling on our Nvidia Jetson Xaviers using Guppy 3.4.1.
  8. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -110,7 +110,7 @@ I was able to shave a minute off the fast model on the Xavier (above) getting it

    **Update**: (13th Dec 2019)

    Just modifying the number of chunks per running has allowed me to get the time down to under 6.5 mins (see table below).
    Just modifying the number of chunks per runner has allowed me to get the time down to under 6.5 mins (see table below).

    | chunks_per_runner | time |
    |:-----------------:|:---------:|
  9. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -284,7 +284,7 @@ The below parameters seem to provide the 'optimal' speed increase with a resulta

    -----

    ## Titan RTX
    ## Guppy basecalling benchmarking on a Titan RTX

    There has been some discussion about the recent release of Guppy (3.4.1 and 3.4.2) in terms of speed. I was interested in running some benchmarks across different versions. I had a hunch it may have been something to do with the newly introduced compression of the fast5 files...

  10. sirselim revised this gist Dec 13, 2019. 1 changed file with 4 additions and 3 deletions.
    7 changes: 4 additions & 3 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -338,10 +338,11 @@ I initially thought that there was something off with the compression imlementat
    What was quite interesting is that running on vbz compressed fast5 data appears to be in the range of 8-10 seconds faster than uncompressed. So there is a slight added speed benefit on top of the nice reduction in file size - which is a little nicer for the SSD/HDD.

    So at this stage I can't confirm any detrimental speed issues when using Guppy version 3.4.X, but this needs to be caveated with all the usual disclaimers:
    * all systems are different (I'm not on Unbuntu for instance)
    * drivers are different (I need to update)
    * GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
    * all systems are different (I'm not on Unbuntu for instance).
    * drivers are different (I need to update).
    * GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
    * For what it's worth, I can add a comment here saying that I haven't had any speed issues with basecalling on our Nvidia Jetson Xaviers using Guppy 3.4.1.
    * our 'little' Linux server isn't exactly a slouch either - so laptop/desktop builds could be very different.
    * I ran the fast basecaller (I'm currently flat out and can't wait for the high accuracy caller) - I may take a subset of data and revist with hac at some stage.

    You can view the 'raw' results/output for each run below:
  11. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -297,7 +297,7 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * 2x 12-Core Intel Xeon Gold 5118 (48 threads)
    * 256Gb RAM
    * Titan RTX
    * Nvidia drivers - 418.56
    * Nvidia drivers: 418.56
    * CUDA Version: 10.1

    **Guppy GPU basecalling parameters:**
  12. sirselim revised this gist Dec 13, 2019. 1 changed file with 9 additions and 3 deletions.
    12 changes: 9 additions & 3 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -335,7 +335,14 @@ For each Guppy version I ran the basecaller three times in an attempt to ensure

    I initially thought that there was something off with the compression imlementation in 3,4,3 as my first run on uncompressed data was ~3x slower than the run on the compressed data. When I grabbed 3.4.1 to perform the same check I noticed that it was fairly consistent between compressed and not. So I went back and was more rigorous and performed 3 iterations of each run for each version, ditto for versions 3.4.X compressed and not. This proved that the initial run was an anomaly and should be disregarded.

    What was quite interesting is that running on vbz compressed
    What was quite interesting is that running on vbz compressed fast5 data appears to be in the range of 8-10 seconds faster than uncompressed. So there is a slight added speed benefit on top of the nice reduction in file size - which is a little nicer for the SSD/HDD.

    So at this stage I can't confirm any detrimental speed issues when using Guppy version 3.4.X, but this needs to be caveated with all the usual disclaimers:
    * all systems are different (I'm not on Unbuntu for instance)
    * drivers are different (I need to update)
    * GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
    * For what it's worth, I can add a comment here saying that I haven't had any speed issues with basecalling on our Nvidia Jetson Xaviers using Guppy 3.4.1.
    * I ran the fast basecaller (I'm currently flat out and can't wait for the high accuracy caller) - I may take a subset of data and revist with hac at some stage.

    You can view the 'raw' results/output for each run below:

    @@ -512,8 +519,7 @@ You can view the 'raw' results/output for each run below:
    input path: flongle_fast5_pass
    save path: testrun_fast_3.4.1
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    chunks per runner: 256CUDA Version: 10.1
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
  13. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -298,6 +298,7 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * 256Gb RAM
    * Titan RTX
    * Nvidia drivers - 418.56
    * CUDA Version: 10.1

    **Guppy GPU basecalling parameters:**
    * --disable_pings
  14. sirselim revised this gist Dec 13, 2019. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -282,6 +282,8 @@ The below parameters seem to provide the 'optimal' speed increase with a resulta
    -i fast5_input \
    -s fastq_output

    -----

    ## Titan RTX

    There has been some discussion about the recent release of Guppy (3.4.1 and 3.4.2) in terms of speed. I was interested in running some benchmarks across different versions. I had a hunch it may have been something to do with the newly introduced compression of the fast5 files...
  15. sirselim revised this gist Dec 13, 2019. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -328,7 +328,11 @@ For each Guppy version I ran the basecaller three times in an attempt to ensure

    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files.*

    So it indeed looks like
    ### Summary to date

    I initially thought that there was something off with the compression imlementation in 3,4,3 as my first run on uncompressed data was ~3x slower than the run on the compressed data. When I grabbed 3.4.1 to perform the same check I noticed that it was fairly consistent between compressed and not. So I went back and was more rigorous and performed 3 iterations of each run for each version, ditto for versions 3.4.X compressed and not. This proved that the initial run was an anomaly and should be disregarded.

    What was quite interesting is that running on vbz compressed

    You can view the 'raw' results/output for each run below:

  16. sirselim revised this gist Dec 13, 2019. 1 changed file with 25 additions and 4 deletions.
    29 changes: 25 additions & 4 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -323,7 +323,7 @@ For each Guppy version I ran the basecaller three times in an attempt to ensure
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | 3.4.1 (no vbz compressed fast5) | 79.913 | 4.96824e+07 |
    | 3.4.1 (vbz compressed fast5) | 90.895 | 4.36797e+07 |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (no vbz compressed fast5) | 90.674 | 4.37862e+07 |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files.*
    @@ -629,7 +629,29 @@ You can view the 'raw' results/output for each run below:

    ### third run


    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    input path: flongle_fast5_pass
    save path: testrun_fast_3.4.3_uncompressed3
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
    kernel path:
    runners per device: 64

    Found 105 fast5 files to process.
    Init time: 719 ms

    0% 10 20 30 40 50 60 70 80 90 100%
    |----|----|----|----|----|----|----|----|----|----|
    ***************************************************
    Caller time: 94516 ms, Samples called: 3970269916, samples/s: 4.20063e+07
    Finishing up any open output files.
    Basecalling completed successfully.

    #### Guppy 3.4.3 (compressed)

    @@ -667,5 +689,4 @@ You can view the 'raw' results/output for each run below:
    ***************************************************
    Caller time: 82877 ms, Samples called: 3970269916, samples/s: 4.79056e+07
    Finishing up any open output files.
    Basecalling completed successfully.

    Basecalling completed successfully.
  17. sirselim revised this gist Dec 13, 2019. 1 changed file with 8 additions and 1 deletion.
    9 changes: 8 additions & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -307,8 +307,12 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * --device "cuda:0"
    * --recursive

    For each Guppy version I ran the basecaller three times in an attempt to ensure that results were consistent<sup>*</sup>.

    **Note:** I chose the fast basecalling model as I wanted to do a quick set of benchmarks. If I feel up to it I may do the same thing for the high accuracy caller...

    <sup>*</sup> Spoiler, I didn't originally do this and it proved misleading...

    ### Results

    | guppy version | time (seconds) | samples/s |
    @@ -571,7 +575,7 @@ You can view the 'raw' results/output for each run below:
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed

    ##### first run
    ##### first run (it looks like this is anomaly)

    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    @@ -623,6 +627,9 @@ You can view the 'raw' results/output for each run below:
    Finishing up any open output files.
    Basecalling completed successfully.

    ### third run



    #### Guppy 3.4.3 (compressed)

  18. sirselim revised this gist Dec 13, 2019. 1 changed file with 29 additions and 0 deletions.
    29 changes: 29 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -571,6 +571,8 @@ You can view the 'raw' results/output for each run below:
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed

    ##### first run

    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    @@ -595,6 +597,33 @@ You can view the 'raw' results/output for each run below:
    Finishing up any open output files.
    Basecalling completed successfully.

    ##### second run

    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    input path: flongle_fast5_pass
    save path: testrun_fast_3.4.3_uncompressed
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
    kernel path:
    runners per device: 64

    Found 105 fast5 files to process.
    Init time: 705 ms

    0% 10 20 30 40 50 60 70 80 90 100%
    |----|----|----|----|----|----|----|----|----|----|
    ***************************************************
    Caller time: 90674 ms, Samples called: 3970269916, samples/s: 4.37862e+07
    Finishing up any open output files.
    Basecalling completed successfully.


    #### Guppy 3.4.3 (compressed)

    ~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
  19. sirselim revised this gist Dec 13, 2019. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -307,6 +307,8 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * --device "cuda:0"
    * --recursive

    **Note:** I chose the fast basecalling model as I wanted to do a quick set of benchmarks. If I feel up to it I may do the same thing for the high accuracy caller...

    ### Results

    | guppy version | time (seconds) | samples/s |
  20. sirselim revised this gist Dec 13, 2019. No changes.
  21. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -316,7 +316,7 @@ The only things I am changing are the version of Guppy being used, and in the ca
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | 3.4.1 (no vbz compressed fast5) | 79.913 | 4.96824e+07 |
    | 3.4.1 (vbz compressed fast5) | | |
    | 3.4.1 (vbz compressed fast5) | 90.895 | 4.36797e+07 |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

  22. sirselim revised this gist Dec 13, 2019. 1 changed file with 74 additions and 10 deletions.
    84 changes: 74 additions & 10 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -315,6 +315,8 @@ The only things I am changing are the version of Guppy being used, and in the ca
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | 3.4.1 (no vbz compressed fast5) | 79.913 | 4.96824e+07 |
    | 3.4.1 (vbz compressed fast5) | | |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

    @@ -477,9 +479,9 @@ You can view the 'raw' results/output for each run below:
    Finishing up any open output files.
    Basecalling completed successfully.

    #### Guppy 3.4.3 (not compressed)
    #### Guppy 3.4.1 (not compressed)

    ~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    ~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    @@ -489,29 +491,29 @@ You can view the 'raw' results/output for each run below:
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed
    -s testrun_fast_3.4.1

    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    ONT Guppy basecalling software version 3.4.1+ad4f8b9
    config file: /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    input path: flongle_fast5_pass
    save path: testrun_fast_3.4.3_uncompressed
    save path: testrun_fast_3.4.1
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
    kernel path:
    kernel path:
    runners per device: 64

    Found 105 fast5 files to process.
    Init time: 738 ms
    Init time: 728 ms

    0% 10 20 30 40 50 60 70 80 90 100%
    |----|----|----|----|----|----|----|----|----|----|
    ***************************************************
    Caller time: 270953 ms, Samples called: 3970269916, samples/s: 1.4653e+07
    Caller time: 90895 ms, Samples called: 3970269916, samples/s: 4.36797e+07
    Finishing up any open output files.
    Basecalling completed successfully.

    @@ -529,6 +531,68 @@ You can view the 'raw' results/output for each run below:
    -i flongle_compressed \
    -s testrun_fast_3.4.1

    ONT Guppy basecalling software version 3.4.1+ad4f8b9
    config file: /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    input path: flongle_compressed
    save path: testrun_fast_3.4.1
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
    kernel path:
    runners per device: 64

    Found 105 fast5 files to process.
    Init time: 725 ms

    0% 10 20 30 40 50 60 70 80 90 100%
    |----|----|----|----|----|----|----|----|----|----|
    ***************************************************
    Caller time: 79913 ms, Samples called: 3970269916, samples/s: 4.96824e+07
    Finishing up any open output files.
    Basecalling completed successfully.

    #### Guppy 3.4.3 (not compressed)

    ~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed

    ONT Guppy basecalling software version 3.4.3+f4fc735
    config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
    model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
    input path: flongle_fast5_pass
    save path: testrun_fast_3.4.3_uncompressed
    chunk size: 1000
    chunks per runner: 256
    records per file: 4000
    fastq compression: ON
    num basecallers: 8
    gpu device: cuda:0
    kernel path:
    runners per device: 64

    Found 105 fast5 files to process.
    Init time: 738 ms

    0% 10 20 30 40 50 60 70 80 90 100%
    |----|----|----|----|----|----|----|----|----|----|
    ***************************************************
    Caller time: 270953 ms, Samples called: 3970269916, samples/s: 1.4653e+07
    Finishing up any open output files.
    Basecalling completed successfully.

    #### Guppy 3.4.3 (compressed)

    ~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
  23. sirselim revised this gist Dec 13, 2019. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -320,6 +320,8 @@ The only things I am changing are the version of Guppy being used, and in the ca

    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files.*

    So it indeed looks like

    You can view the 'raw' results/output for each run below:

    #### Guppy 3.1.5
    @@ -513,6 +515,20 @@ You can view the 'raw' results/output for each run below:
    Finishing up any open output files.
    Basecalling completed successfully.

    #### Guppy 3.4.1 (compressed)

    ~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_compressed \
    -s testrun_fast_3.4.1

    #### Guppy 3.4.3 (compressed)

    ~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
  24. sirselim revised this gist Dec 13, 2019. No changes.
  25. sirselim revised this gist Dec 13, 2019. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -311,10 +311,10 @@ The only things I am changing are the version of Guppy being used, and in the ca

    | guppy version | time (seconds) | samples/s |
    |-------------------------------|:--------------:|-------------|
    | 3.1.5*<sup>#</sup>* | 93.278 | 4.25638e+07 |
    | 3.2.4*<sup>#</sup>* | 94.141 | 4.21737e+07 |
    | 3.3.0*<sup>#</sup>* | 94.953 | 4.1813e+07 |
    | 3.3.3*<sup>#</sup>* | 95.802 | 4.14425e+07 |
    | 3.1.5<sup>#</sup> | 93.278 | 4.25638e+07 |
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

  26. sirselim revised this gist Dec 13, 2019. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -311,10 +311,10 @@ The only things I am changing are the version of Guppy being used, and in the ca

    | guppy version | time (seconds) | samples/s |
    |-------------------------------|:--------------:|-------------|
    | 3.1.5<sup>#</sup> | 93.278 | 4.25638e+07 |
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | 3.1.5*<sup>#</sup>* | 93.278 | 4.25638e+07 |
    | 3.2.4*<sup>#</sup>* | 94.141 | 4.21737e+07 |
    | 3.3.0*<sup>#</sup>* | 94.953 | 4.1813e+07 |
    | 3.3.3*<sup>#</sup>* | 95.802 | 4.14425e+07 |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

  27. sirselim revised this gist Dec 13, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -310,7 +310,7 @@ The only things I am changing are the version of Guppy being used, and in the ca
    ### Results

    | guppy version | time (seconds) | samples/s |
    |:-------------------------------:|:--------------:|-------------|
    |-------------------------------|:--------------:|-------------|
    | 3.1.5<sup>#</sup> | 93.278 | 4.25638e+07 |
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
  28. sirselim revised this gist Dec 13, 2019. 1 changed file with 5 additions and 5 deletions.
    10 changes: 5 additions & 5 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -311,12 +311,12 @@ The only things I am changing are the version of Guppy being used, and in the ca

    | guppy version | time (seconds) | samples/s |
    |:-------------------------------:|:--------------:|-------------|
    | 3.1.5<sup>#</sup> | 93.278 | 4.25638e+07 |
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | 3.1.5<sup>#</sup> | 93.278 | 4.25638e+07 |
    | 3.2.4<sup>#</sup> | 94.141 | 4.21737e+07 |
    | 3.3.0<sup>#</sup> | 94.953 | 4.1813e+07 |
    | 3.3.3<sup>#</sup> | 95.802 | 4.14425e+07 |
    | **3.4.3 (no vbz compressed fast5)** | **270.953** | **1.4653e+07** |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |
    | 3.4.3 (vbz compressed fast5) | 82.877 | 4.79056e+07 |

    *<sup>#</sup> these versions of Guppy did not support vbz compression of fast5 files.*

  29. sirselim revised this gist Dec 13, 2019. 1 changed file with 8 additions and 8 deletions.
    16 changes: 8 additions & 8 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -298,14 +298,14 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * Nvidia drivers - 418.56

    **Guppy GPU basecalling parameters:**
    * disable_pings
    * compress_fastq
    * dna_r9.4.1_450bps_fast.cfg
    * num_callers 8
    * gpu_runners_per_device 64
    * chunks_per_runner 256
    * device "cuda:0"
    * recursive
    * --disable_pings
    * --compress_fastq
    * --dna_r9.4.1_450bps_fast.cfg
    * --num_callers 8
    * --gpu_runners_per_device 64
    * --chunks_per_runner 256
    * --device "cuda:0"
    * --recursive

    ### Results

  30. sirselim revised this gist Dec 13, 2019. 1 changed file with 0 additions and 2 deletions.
    2 changes: 0 additions & 2 deletions basecalling_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -307,8 +307,6 @@ The only things I am changing are the version of Guppy being used, and in the ca
    * device "cuda:0"
    * recursive



    ### Results

    | guppy version | time (seconds) | samples/s |