Skip to content

Instantly share code, notes, and snippets.

@sudo-suhas
Created August 25, 2018 04:43
Show Gist options
  • Select an option

  • Save sudo-suhas/9cd7fae904b94f93730843b118729230 to your computer and use it in GitHub Desktop.

Select an option

Save sudo-suhas/9cd7fae904b94f93730843b118729230 to your computer and use it in GitHub Desktop.

Revisions

  1. sudo-suhas created this gist Aug 25, 2018.
    151 changes: 151 additions & 0 deletions pprof.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,151 @@
    # pprof

    With a single `import _ "net/http/pprof"` you can add profiling endpoints to a
    HTTP server.

    ```go
    package main

    import (
    "fmt"
    "log"
    "net/http"
    _ "net/http/pprof" // register debug routes on default mux
    )

    func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello World!")
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
    }
    ```

    This will now report diagnostics via `/debug/pprof/`.

    - `/debug/pprof/profile`: 30-second CPU profile `/debug/pprof/heap`: heap
    profile
    - `/debug/pprof/goroutine?debug=1`: all goroutines with stack traces
    - `/debug/pprof/trace`: take a trace

    ## Risks

    This mechanism is _dangerously_ simple. It only requires one import which could
    be anywhere!

    Security issues:

    - Function names and file paths are revealed.
    - Profiling data may reveal business sensitive information (for example traffic
    to a web server)
    - Profiling degrades performance, providing a vector for a DoS attack

    Depending on your application, leaving a debugging server may not necessarily be
    a critical security hole. At a minimum it’s inadvisable, but it could be much
    worse.

    ## Access Control

    A simple and effective option is to put the pprof http server on a separate port
    on localhost, separate from the application http server. If the application does
    not use the http default multiplexer, starting the profiling http server is as
    simple as:

    ```go
    go func() {
    log.Println(http.ListenAndServe("localhost:8081"))
    }()
    ```

    Otherwise, you can run the following additional setup prior to any
    `http.HandleFunc()` calls:

    ```go
    // Save pprof handlers first.
    pprofMux := http.DefaultServeMux
    http.DefaultServeMux = http.NewServeMux()

    // Pprof server.
    go func() {
    log.Println(http.ListenAndServe("localhost:8081", pprofMux))
    }()

    // Application server.
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello World!")
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
    ```

    Either of the above snippets can be conditionally run, so the profiling server
    may be turned on or off by command line flags or application configuration.
    However, since they are now only locally accessible, there is no downside to
    leaving them on.

    ## Remote Control

    Now that you have the profiling hooks safely exposed via a localhost-only
    interface, you can invoke the following:

    ```shell
    $ go tool pprof http://localhost:8081/debug/pprof/profile
    ```

    However, since go compiles to a static binary which can be installed without any
    go-related dependencies, there is a possibility that you don't have go tools
    where the program is running. SSH port forwarding(AKA SSH tunneling) can be used
    to make `go tool pprof` on your local machine profile the remote code over the
    SSH tunnel.

    ```shell
    $ ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected]

    $ go tool pprof -inuse_objects /path/to/binary/myapp http://localhost:8081/debug/pprof/heap
    Fetching profile over HTTP from http://localhost:8081/debug/pprof/heap
    Saved profile in /home/suhaskaranth/pprof/pprof.myapp.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz
    File: myapp
    Type: inuse_objects
    Time: Aug 11, 2018 at 11:45am (IST)
    Entering interactive mode (type "help" for commands, "o" for options)
    (pprof) top
    Showing nodes accounting for 65536, 99.73% of 65714 total
    Dropped 50 nodes (cum <= 328)
    Showing top 10 nodes out of 20
    flat flat% sum% cum cum%
    32768 49.86% 49.86% 65536 99.73% myapp/vendor/github.com/ugorji/go/codec.init.0
    32768 49.86% 99.73% 32768 49.86% sync.(*Map).Range
    0 0% 99.73% 32768 49.86% encoding/gob.(*Decoder).decOpFor
    0 0% 99.73% 32768 49.86% encoding/gob.(*Decoder).decodeInterface
    0 0% 99.73% 32768 49.86% encoding/gob.decInt32Slice
    0 0% 99.73% 32768 49.86% encoding/gob.decInt64Slice
    0 0% 99.73% 32768 49.86% encoding/gob.decStringSlice
    0 0% 99.73% 32768 49.86% myapp/adapter/internal.SendNotifications
    0 0% 99.73% 32768 49.86% myapp/cache.prepStaticComponent
    0 0% 99.73% 32768 49.86% myapp/model/redis.(*Client).Del
    (pprof) quit
    ```

    To stop the SSH session running in the background, check the running processes
    for the specified port:

    ```shell
    $ ps -aux | grep 8081
    suhaska+ 23340 0.0 0.0 52516 752 ? Ss 16:02 0:00 ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected]
    suhaska+ 23386 0.0 0.0 14432 1052 pts/0 S+ 16:02 0:00 grep --color=auto 8081

    $ kill -QUIT 23340
    ```

    ## References

    - https://golang.org/doc/diagnostics.html
    - https://blog.golang.org/profiling-go-programs
    - https://golang.org/pkg/net/http/pprof/
    - https://www.farsightsecurity.com/2016/10/28/cmikk-go-remote-profiling/
    - https://mmcloughlin.com/posts/your-pprof-is-showing
    - https://blog.trackets.com/2014/05/17/ssh-tunnel-local-and-remote-port-forwarding-explained-with-examples.html
    - https://stackoverflow.com/questions/9447226/how-to-close-this-ssh-tunnel
    - https://superuser.com/questions/827934/ssh-port-forwarding-without-session
    - https://stackoverflow.com/questions/24863164/how-to-analyse-golang-memory
    - https://dave.cheney.net/2014/07/11/visualising-the-go-garbage-collector
    - https://github.com/pkg/profile