# Debugging Best Practices

## Preparation
* Establish a clear understanding of the issue
  * Ask issue reporter/lead about issue if needed
* Reproduce the issue
  * Is it a workflow that is causing the issue
  * Is it particular data that is causing the issue
  * Is it a combination of the above
  * Is there a timing issue or a race condition issue
  * Is there a system constraint we are hitting
* Establish high level mental model of what the system is doing
  * What is the system currently doing
    * eg A system sends a message via kafka to our application, it processes the message and stores it in the database 
  * What should the system be doing
    * eg The system should be storing the name in the kafka message we recieve 
## Before you look at Code
* Read error messages carefully
  * Check for exceptions inside exceptions
    * Investigate the root exception 
  * For stack traces with multiple error messages find the most useful error messages
    * eg `/etc/truststore.jks not found` is more useful than `Auth failed` 
  * Try to understand what the system is telling you with the error message
  * Unfortunately sometimes error messages can not very useful or even worse misleading
  * Search most useful error messages and try other error messages and even not error messages
    * Carefully evaluate solutions to errors
      * They may solve your error/problem, but create other problems  
## Debugging
* Start from a point that works and work your way to a point that doesn't
  * eg The UI sends a message to the backend that is correct, but db doesn't have the correct values 
    * debug from the backend api endoint to storing in the db
  * Use a debugger!
* Test different sets of changes
  * Bisect small code changes
  * Bisect commits
  * Bisect versions
  * Narrow down which set of changes are causing the issue
* For harder to reproduce bugs read through the code carefully
  * Walk through the flow of the code and try to think of anything that can go wrong
  * Use a debugger to test assumptions on flow and variables
  * Consider Race conditions
  * Consider scope of code
  * Issues with assumptions on database/frameworks/libraries
# Process of Elimination
- If you eliminate the environment and the code, maybe it's the data?