Skip to content

Instantly share code, notes, and snippets.

@DavisVaughan

DavisVaughan/AWS-furrr.R

Last active October 9, 2022 06:41
Show Gist options
  • Save DavisVaughan/865d95cf0101c24df27b37f4047dd2e5 to your computer and use it in GitHub Desktop.
Save DavisVaughan/865d95cf0101c24df27b37f4047dd2e5 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. Davis Vaughan revised this gist May 31, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions AWS-furrr.R
    View file
    Original file line number Diff line number Diff line change
    @@ -23,6 +23,8 @@ cl <- makeClusterPSOCK(

    ## Use private SSH key registered with AWS
    rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
    ),

  2. Davis Vaughan revised this gist May 31, 2018. 1 changed file with 1 addition and 5 deletions.
    6 changes: 1 addition & 5 deletions AWS-furrr.R
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,3 @@
    ``` r
    # This example demonstrates running furrr code distributed on 2 AWS instances ("nodes").

    # The instances have already been created.
    @@ -71,7 +70,4 @@ toc()
    #> 13.158 sec elapsed

    # Shut down
    parallel::stopCluster(cl)
    ```

    Created on 2018-05-31 by the [reprex package](http://reprex.tidyverse.org) (v0.2.0).
    parallel::stopCluster(cl)
  3. Davis Vaughan created this gist May 31, 2018.
    77 changes: 77 additions & 0 deletions AWS-furrr.R
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,77 @@
    ``` r
    # This example demonstrates running furrr code distributed on 2 AWS instances ("nodes").

    # The instances have already been created.

    library(future)
    library(furrr)

    # Two t2.micro AWS instances
    # Created from http://www.louisaslett.com/RStudio_AMI/
    public_ip <- c("34.205.155.182", "34.201.26.217")

    # This is where my pem file lives (password to connect essentially).
    ssh_private_key_file <- "~/Desktop/programming/AWS/key-pair/dvaughan.pem"

    # Connect!
    cl <- makeClusterPSOCK(

    ## Public IP number of EC2 instance
    public_ip,

    ## User name (always 'ubuntu')
    user = "ubuntu",

    ## Use private SSH key registered with AWS
    rshopts = c(
    "-i", ssh_private_key_file
    ),

    ## Set up .libPaths() for the 'ubuntu' user and
    ## install future/purrr/furrr packages
    rscript_args = c(
    "-e", shQuote("local({p <- Sys.getenv('R_LIBS_USER'); dir.create(p, recursive = TRUE, showWarnings = FALSE); .libPaths(p)})"),
    "-e", shQuote("install.packages(c('future', 'purrr', 'furrr'))")
    ),

    dryrun = FALSE
    )

    # Set the plan to use the cluster workers!
    plan(cluster, workers = cl)

    # Run some code distributed evenly on the two workers!
    x <- 1
    future_map(1:5, ~{.x + x})
    #> [[1]]
    #> [1] 2
    #>
    #> [[2]]
    #> [1] 3
    #>
    #> [[3]]
    #> [1] 4
    #>
    #> [[4]]
    #> [1] 5
    #>
    #> [[5]]
    #> [1] 6

    # Are we reaallllly running in parallel?
    library(tictoc)
    tic()
    future_map(1:2, ~{ Sys.sleep(10) })
    #> [[1]]
    #> NULL
    #>
    #> [[2]]
    #> NULL
    toc()
    #> 13.158 sec elapsed

    # Shut down
    parallel::stopCluster(cl)
    ```

    Created on 2018-05-31 by the [reprex package](http://reprex.tidyverse.org) (v0.2.0).