Created
August 25, 2018 04:43
-
-
Save sudo-suhas/9cd7fae904b94f93730843b118729230 to your computer and use it in GitHub Desktop.
Revisions
-
sudo-suhas created this gist
Aug 25, 2018 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,151 @@ # pprof With a single `import _ "net/http/pprof"` you can add profiling endpoints to a HTTP server. ```go package main import ( "fmt" "log" "net/http" _ "net/http/pprof" // register debug routes on default mux ) func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello World!") }) log.Fatal(http.ListenAndServe(":8080", nil)) } ``` This will now report diagnostics via `/debug/pprof/`. - `/debug/pprof/profile`: 30-second CPU profile `/debug/pprof/heap`: heap profile - `/debug/pprof/goroutine?debug=1`: all goroutines with stack traces - `/debug/pprof/trace`: take a trace ## Risks This mechanism is _dangerously_ simple. It only requires one import which could be anywhere! Security issues: - Function names and file paths are revealed. - Profiling data may reveal business sensitive information (for example traffic to a web server) - Profiling degrades performance, providing a vector for a DoS attack Depending on your application, leaving a debugging server may not necessarily be a critical security hole. At a minimum it’s inadvisable, but it could be much worse. ## Access Control A simple and effective option is to put the pprof http server on a separate port on localhost, separate from the application http server. If the application does not use the http default multiplexer, starting the profiling http server is as simple as: ```go go func() { log.Println(http.ListenAndServe("localhost:8081")) }() ``` Otherwise, you can run the following additional setup prior to any `http.HandleFunc()` calls: ```go // Save pprof handlers first. pprofMux := http.DefaultServeMux http.DefaultServeMux = http.NewServeMux() // Pprof server. go func() { log.Println(http.ListenAndServe("localhost:8081", pprofMux)) }() // Application server. http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello World!") }) log.Fatal(http.ListenAndServe(":8080", nil)) ``` Either of the above snippets can be conditionally run, so the profiling server may be turned on or off by command line flags or application configuration. However, since they are now only locally accessible, there is no downside to leaving them on. ## Remote Control Now that you have the profiling hooks safely exposed via a localhost-only interface, you can invoke the following: ```shell $ go tool pprof http://localhost:8081/debug/pprof/profile ``` However, since go compiles to a static binary which can be installed without any go-related dependencies, there is a possibility that you don't have go tools where the program is running. SSH port forwarding(AKA SSH tunneling) can be used to make `go tool pprof` on your local machine profile the remote code over the SSH tunnel. ```shell $ ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected] $ go tool pprof -inuse_objects /path/to/binary/myapp http://localhost:8081/debug/pprof/heap Fetching profile over HTTP from http://localhost:8081/debug/pprof/heap Saved profile in /home/suhaskaranth/pprof/pprof.myapp.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz File: myapp Type: inuse_objects Time: Aug 11, 2018 at 11:45am (IST) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 65536, 99.73% of 65714 total Dropped 50 nodes (cum <= 328) Showing top 10 nodes out of 20 flat flat% sum% cum cum% 32768 49.86% 49.86% 65536 99.73% myapp/vendor/github.com/ugorji/go/codec.init.0 32768 49.86% 99.73% 32768 49.86% sync.(*Map).Range 0 0% 99.73% 32768 49.86% encoding/gob.(*Decoder).decOpFor 0 0% 99.73% 32768 49.86% encoding/gob.(*Decoder).decodeInterface 0 0% 99.73% 32768 49.86% encoding/gob.decInt32Slice 0 0% 99.73% 32768 49.86% encoding/gob.decInt64Slice 0 0% 99.73% 32768 49.86% encoding/gob.decStringSlice 0 0% 99.73% 32768 49.86% myapp/adapter/internal.SendNotifications 0 0% 99.73% 32768 49.86% myapp/cache.prepStaticComponent 0 0% 99.73% 32768 49.86% myapp/model/redis.(*Client).Del (pprof) quit ``` To stop the SSH session running in the background, check the running processes for the specified port: ```shell $ ps -aux | grep 8081 suhaska+ 23340 0.0 0.0 52516 752 ? Ss 16:02 0:00 ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected] suhaska+ 23386 0.0 0.0 14432 1052 pts/0 S+ 16:02 0:00 grep --color=auto 8081 $ kill -QUIT 23340 ``` ## References - https://golang.org/doc/diagnostics.html - https://blog.golang.org/profiling-go-programs - https://golang.org/pkg/net/http/pprof/ - https://www.farsightsecurity.com/2016/10/28/cmikk-go-remote-profiling/ - https://mmcloughlin.com/posts/your-pprof-is-showing - https://blog.trackets.com/2014/05/17/ssh-tunnel-local-and-remote-port-forwarding-explained-with-examples.html - https://stackoverflow.com/questions/9447226/how-to-close-this-ssh-tunnel - https://superuser.com/questions/827934/ssh-port-forwarding-without-session - https://stackoverflow.com/questions/24863164/how-to-analyse-golang-memory - https://dave.cheney.net/2014/07/11/visualising-the-go-garbage-collector - https://github.com/pkg/profile