spacejam · August 11, 2025 18:21 · Sep 24, 2017 · Sep 24, 2017 · Sep 24, 2017 · Sep 24, 2017
diff --git a/rr-with-rust.md b/rr-with-rust.md
@@ -31,6 +31,13 @@ and `sudo cpupower frequency-set -g performance` to get better results.
 
 [this gdb cheatsheet is helpful](http://darkdust.net/files/GDB%20Cheat%20Sheet.pdf)
 
+## tui mode
+
+if you are used to using a debugger from an IDE, you may be much more comfortable while using GDB
+with the TUI. you can enter it by hitting `Ctrl+x` then `Ctrl+a`. for more info about using
+and configuring the TUI, check out its [official docs](https://sourceware.org/gdb/onlinedocs/gdb/TUI.html).
+it can be configured to show or hide lots of interesting info.
+
 ## navigation
 feature | control | feature | control
 --- | --- | --- | ---

diff --git a/rr-with-rust.md b/rr-with-rust.md
@@ -27,6 +27,10 @@ and `sudo cpupower frequency-set -g performance` to get better results.
 
 `rr replay` to open the debugger (rr is a bunch of functionality on top of gdb)
 
+## gdb cheatsheet
+
+[this gdb cheatsheet is helpful](http://darkdust.net/files/GDB%20Cheat%20Sheet.pdf)
+
 ## navigation
 feature | control | feature | control
 --- | --- | --- | ---

diff --git a/rr-with-rust.md b/rr-with-rust.md
@@ -47,6 +47,8 @@ self = 0x7f8d07dfd9d0
 key = &[u8](len: 1) = {165} 
 ```
 
+`list` (`l` for short) can be used to view the contents of the current file, or another by providing the module name
+(tab-complete is your friend!)
 ```
 (rr) list sled::tree::Tree::path_for_key
 ...

diff --git a/rr-with-rust.md b/rr-with-rust.md
@@ -1,4 +1,4 @@
-# using rust pretty printers with rr
+# using rust with rr
 `rr` is a great debugging tool. it records a trace of a program's execution, as well as the results of
 any syscalls it executes, so that you can "rewind" while you debug, and get deterministic forward and reverse
 instrumented playback. it works with rust, but by default if you try it out, it could be pretty ugly when you

diff --git a/rr-with-rust.md b/rr-with-rust.md
@@ -0,0 +1,138 @@
+# using rust pretty printers with rr
+`rr` is a great debugging tool. it records a trace of a program's execution, as well as the results of
+any syscalls it executes, so that you can "rewind" while you debug, and get deterministic forward and reverse
+instrumented playback. it works with rust, but by default if you try it out, it could be pretty ugly when you
+inspect variables. if this bothers you, [configure gdb to use a rust pretty-printer](https://gist.github.com/spacejam/0ec039d4b3bb0ac976245975de8f3a25)
+
+`rr` is probably in your system's package manager.
+
+## usage
+
+1. setup
+  * On some linux systems, you may need to `echo 1 | sudo tee  /proc/sys/kernel/perf_event_paranoid` 
+and `sudo cpupower frequency-set -g performance` to get better results.
+2. compile with debug symbols
+  * this is on by default for tests. 
+  * You can enable it for release builds in `Cargo.toml` if you need to:
+    ```
+    [profile.release]
+    debug = 2
+    ```
+3. record a trace for deterministic replay / rewind
+  * `rr record /path/to/executable`, or
+  * `rr record cargo test problem_test_7`, or
+  * `rr record -n cargo run` (may be necessary when you can't control perf event paranoia)
+
+## replay
+
+`rr replay` to open the debugger (rr is a bunch of functionality on top of gdb)
+
+## navigation
+feature | control | feature | control
+--- | --- | --- | ---
+step (goes into called functions) | s | reverse step | rs
+next (doesn't go into called functions) | n | reverse next | rn
+finish (current function) | f | reverse finish | reverse-finish
+continue (to next breakpoint) | c | reverse continue | rc
+
+## inspecting state
+`info locals` prints local variables (`i lo` for short, gdb is big on shortcuts)
+`p <var name>` prints contents 
+
+```
+(rr) info locals
+ret = Some = {Vec<u8>(len: 0, cap: 5120)}
+start = 169602351
+self = 0x7f8d07dfd9d0
+key = &[u8](len: 1) = {165} 
+```
+
+```
+(rr) list sled::tree::Tree::path_for_key
+...
+479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);
+...
+```
+
+## adding breakpoints
+the line number provided matches the last file `list`ed
+```
+(rr) break 479
+Breakpoint 1 at 0x555929578ec4: /home/t/src/sled/src/tree/mod.rs:479. (2 locations)
+```
+
+this will be hit next time the code corresponding to this line is reached:
+```
+(rr) c
+Continuing.
+
+Thread 4 hit Breakpoint 1, sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:479
+479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);
+```
+
+conditional breakpoints
+
+```
+(rr) list sled::tree::Tree::path_for_key                                                                                                                                                                                                                       
+...
+464         fn path_for_key(&self, key: &[u8]) -> Vec<(Node, CasKey<Frag>)> {
+...
+473             let mut not_found_loops = 0;
+474             loop {
+475                 let get_cursor = self.pages.get(cursor);
+476                 if get_cursor.is_none() {
+477                     // restart search from the tree's root
+478                     not_found_loops += 1;
+(rr) 
+479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);
+...
+(rr) b 476 if not_found_loops > 5
+Breakpoint 70 at 0x555929578e11: file src/tree/mod.rs, line 476.
+(rr) i br
+Num     Type           Disp Enb Address            What
+70      breakpoint     keep y   0x0000555929578e11 in sled::tree::Tree::path_for_key at src/tree/mod.rs:476
+        stop only if not_found_loops > 5 (host evals)
+(rr) c
+Continuing.
+[New Thread 3140.3145]
+[Switching to Thread 3140.3141]
+
+Thread 3 hit Breakpoint 70, sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:476
+476                 if get_cursor.is_none() {
+(rr) i lo
+get_cursor = core::option::Option::None
+not_found_loops = 6
+unsplit_parent = core::option::Option::None
+path = Vec<(sled::tree::node::Node, sled::page::CasKey<sled::tree::frag::Frag>)>(len: 0, cap: 0)
+cursor = 31
+key_bound = Inc = {Vec<u8>(len: 1, cap: 1) = {165}}
+self = 0x7f8d07dfd9d0
+key = &[u8](len: 1) = {165}
+```
+
+## adding watchpoints
+[watchpoints](https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.html) are able to break when a memory location is accessed,
+or given expression returns true. this is super powerfulfor speeding up the debugging process.
+
+break whenever a variable of interest is accessed:
+
+```
+(rr) awatch self.root
+Hardware access (read/write) watchpoint 4: self.root
+(rr) c
+Continuing.
+
+Thread 2 hit Hardware access (read/write) watchpoint 4: self.root
+
+Value = AtomicUsize = {v = UnsafeCell<usize> = {value = 0}}
+
+Thread 2 hit Hardware access (read/write) watchpoint 4: self.root
+
+Value = AtomicUsize = {v = UnsafeCell<usize> = {value = 0}} 
+0x0000555929578d42 in sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:466
+466             let mut cursor = self.root.load(SeqCst);
+```
+
+Lots more info available in the official docs:
+* [breakpoints](https://sourceware.org/gdb/onlinedocs/gdb/Breakpoints.html#Breakpoints)
+* [rust caveats](https://sourceware.org/gdb/onlinedocs/gdb/Rust.html#Rust)