Skip to content

Instantly share code, notes, and snippets.

@tyrauber

tyrauber/daylight maps buildscript

Last active November 3, 2023 15:42
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save tyrauber/ad03568f399521e9c3b3189aa99e3b51 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save tyrauber/ad03568f399521e9c3b3189aa99e3b51 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. tyrauber revised this gist Nov 3, 2023. 1 changed file with 44 additions and 1 deletion.
    45 changes: 44 additions & 1 deletion daylight maps buildscript
    View file
    Original file line number Diff line number Diff line change
    @@ -20,5 +20,48 @@
    ) &\
    wait

    /* apply-changes admin and roads first */
    osmium apply-changes planet-v$1.osm.pbf admin-v$1.osc.gz fb-ml-roads-v$1.osc.gz -o planet-osc-v$1.osm.pbf --verbose --progress
    osmium merge planet-osc-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-daylight-v$1.osm.pbf --progress

    /* merge buildings second */
    osmium merge planet-osc-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-daylight-v$1.osm.pbf --progress


    /*
    MEMORY USAGE
    osmium apply-changes keeps the contents of all the change files in main memory. This will take roughly 10 times as much memory as the files take on disk in .osm.bz2 format.
    2688M admin-v1.33.osc.gz
    4912M fb-ml-roads-v1.33.osc.gz
    = 7600M
    * 10 = 76GB

    I ran this script on a r5a.2xlarge, 8 x 64 GiB, for $0.452/hr.

    According to the logs only 33983Mb memory used. Might be able to squeek by with 48GB, but 32GB is doutbfull.

    [ 0:00] Started osmium apply-changes
    [ 0:00] osmium version 1.14.0
    [ 0:00] libosmium version 2.18.0
    [ 0:00] Command line options and default settings:
    [ 0:00] input data file name: planet-v1.33.osm.pbf
    [ 0:00] input change file names:
    [ 0:00] admin-v1.33.osc.gz
    [ 0:00] fb-ml-roads-v1.33.osc.gz
    [ 0:00] data file format:
    [ 0:00] change file format:
    [ 0:00] output options:
    [ 0:00] file name: planet-osc-v1.33.osm.pbf
    [ 0:00] file format:
    [ 0:00] generator: osmium/1.14.0
    [ 0:00] overwrite: no
    [ 0:00] fsync: no
    [ 0:00] reading and writing history file: no
    [ 0:00] locations on ways: no
    [ 0:00] Reading change file contents...
    [32:46] Opening input file...
    [32:46] Opening output file...
    [32:46] Sorting change data...
    [33:44] Applying changes and writing them to output...
    [> ] 0% ^M[> ] 1% ^M[=> >
    [85:04] Peak memory used: 33983 MBytes
    */
  2. tyrauber revised this gist Nov 3, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions daylight maps buildscript
    View file
    Original file line number Diff line number Diff line change
    @@ -20,5 +20,5 @@
    ) &\
    wait

    osmium merge planet-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-plus-buildings-v$1.osm.pbf --overwrite
    osmium apply-changes planet-plus-buildings-v$1.osm.pbf fb-ml-roads-v$1.osc.gz admin-v$1.osc.gz -o planet-daylight-v$1.pbf --overwrite
    osmium apply-changes planet-v$1.osm.pbf admin-v$1.osc.gz fb-ml-roads-v$1.osc.gz -o planet-osc-v$1.osm.pbf --verbose --progress
    osmium merge planet-osc-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-daylight-v$1.osm.pbf --progress
  3. tyrauber revised this gist Nov 2, 2023. 1 changed file with 6 additions and 17 deletions.
    23 changes: 6 additions & 17 deletions daylight maps buildscript
    View file
    Original file line number Diff line number Diff line change
    @@ -1,35 +1,24 @@
    # nohup ./daylight.sh 1.23 >> daylight.log 2>&1 &
    # nohup ./daylight.sh 1.33 >> daylight.log 2>&1 &
    (
    echo "Getting daylight $1 planet file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/planet-v$1.osm.pbf
    echo "Determining starting id for renumber"
    osmium fileinfo -e -t node -g data.maxid.nodes planet-v$1.osm.pbf > max-id-planet-v$1.txt
    ) &\
    (
    (
    echo "Getting daylight $1 buildings file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/ms-ml-buildings-v$1.osc.bz2
    ) &\
    (
    echo "Getting daylight $1 ms roads file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/grab-ml-roads-v$1.osc.bz2
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/ml-buildings-v$1.osm.pbf
    ) &\
    (
    echo "Getting daylight $1 fb roads file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/fb-ml-roads-v$1.osc.bz2
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/fb-ml-roads-v$1.osc.gz
    ) &\
    (
    echo "Getting daylight $1 admin file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/admin-v$1.osc.bz2
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/admin-v$1.osc.gz
    ) &\
    wait
    echo "Sorting and merging $1 admin and buildings files"
    osmium sort --progress -v --output merged-sorted-oscs-v$1.osm.pbf admin-v$1.osc.bz2 fb-ml-roads-v$1.osc.bz2 grab-ml-roads-v$1.osc.bz2 ms-ml-buildings-v$1.osc.bz2 --overwrite
    ) &\
    wait

    echo "Renumbering merged $1 file"
    osmium renumber --progress -v --output merged-renumbered-oscs-v$1.osm.pbf merged-sorted-oscs-v$1.osm.pbf --overwrite

    echo "Concatenating planet file with merged renumbered file"
    osmium apply-changes --progress -v --output planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf planet-v$1.osm.pbf merged-renumbered-oscs-v$1.osm.pbf --overwrite
    osmium merge planet-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-plus-buildings-v$1.osm.pbf --overwrite
    osmium apply-changes planet-plus-buildings-v$1.osm.pbf fb-ml-roads-v$1.osc.gz admin-v$1.osc.gz -o planet-daylight-v$1.pbf --overwrite
  4. tyrauber revised this gist Mar 12, 2023. 1 changed file with 2 additions and 7 deletions.
    9 changes: 2 additions & 7 deletions daylight maps buildscript
    View file
    Original file line number Diff line number Diff line change
    @@ -28,13 +28,8 @@
    ) &\
    wait

    export MAX_ID=`cat max-id-planet-v$1.txt`
    export START_ID=$((MAX_ID + 1))
    echo "Renumber starting ID is $START_ID"

    echo "Renumbering merged $1 file"
    osmium renumber --progress -v --start-id=$START_ID --output merged-renumbered-oscs-v$1.osm.pbf merged-sorted-oscs-v$1.osm.pbf --overwrite
    osmium renumber --progress -v --output merged-renumbered-oscs-v$1.osm.pbf merged-sorted-oscs-v$1.osm.pbf --overwrite

    echo "Concatenating planet file with merged renumbered file"
    osmium apply-changes --progress -v --output planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf planet-v$1.osm.pbf merged-renumbered-oscs-v$1.osm.pbf --overwrite
    md5sum planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf > planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf.md5
    osmium apply-changes --progress -v --output planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf planet-v$1.osm.pbf merged-renumbered-oscs-v$1.osm.pbf --overwrite
  5. tyrauber created this gist Mar 9, 2023.
    40 changes: 40 additions & 0 deletions daylight maps buildscript
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,40 @@
    # nohup ./daylight.sh 1.23 >> daylight.log 2>&1 &
    (
    echo "Getting daylight $1 planet file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/planet-v$1.osm.pbf
    echo "Determining starting id for renumber"
    osmium fileinfo -e -t node -g data.maxid.nodes planet-v$1.osm.pbf > max-id-planet-v$1.txt
    ) &\
    (
    (
    echo "Getting daylight $1 buildings file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/ms-ml-buildings-v$1.osc.bz2
    ) &\
    (
    echo "Getting daylight $1 ms roads file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/grab-ml-roads-v$1.osc.bz2
    ) &\
    (
    echo "Getting daylight $1 fb roads file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/fb-ml-roads-v$1.osc.bz2
    ) &\
    (
    echo "Getting daylight $1 admin file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/admin-v$1.osc.bz2
    ) &\
    wait
    echo "Sorting and merging $1 admin and buildings files"
    osmium sort --progress -v --output merged-sorted-oscs-v$1.osm.pbf admin-v$1.osc.bz2 fb-ml-roads-v$1.osc.bz2 grab-ml-roads-v$1.osc.bz2 ms-ml-buildings-v$1.osc.bz2 --overwrite
    ) &\
    wait

    export MAX_ID=`cat max-id-planet-v$1.txt`
    export START_ID=$((MAX_ID + 1))
    echo "Renumber starting ID is $START_ID"

    echo "Renumbering merged $1 file"
    osmium renumber --progress -v --start-id=$START_ID --output merged-renumbered-oscs-v$1.osm.pbf merged-sorted-oscs-v$1.osm.pbf --overwrite

    echo "Concatenating planet file with merged renumbered file"
    osmium apply-changes --progress -v --output planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf planet-v$1.osm.pbf merged-renumbered-oscs-v$1.osm.pbf --overwrite
    md5sum planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf > planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf.md5