Debugging in Go with Delve

There are different ways of debugging a Go program. You could use print statements to view the values stored in your variables during the execution or you could use a debugger. Some IDEs, like GoLand, offer this functionality by default but it is also interesting to see how to use the integrated solutions from the command line. Delve offers both command line clients and an API that can be used to integrate it to other tools (which is actually what GoLand uses under the hood).

Delve logo

Getting started

The easiest way to install Delve is to use the standard go get command:

$ go get github.com/go-delve/delve/cmd/dlv

Once installed you can use it to debug your Go application. I am going to use a simple Go program that you can find in my Github repository (under delve-debug). It has a main function with two nested loops that increment the value of a variable and two auxiliary functions that don’t do much other than call each other.

🖥️ The full code is on my Github repo: https://github.com/efrag/blog-posts

Navigate to the directory that has your main.go file and type

$ cd github.com/efrag/blog-posts/delve-debug/
$ dlv debug
(dlv) 

This will bring up the (dlv) terminal. Keep in mind that this will compile and build your program which is now associated with a running process. Typing help will bring up all available options and help <command> will display information about the specific command.

Breakpoints

One of the most used commands is the continue one. If we go ahead and run it we will see the following:

(dlv) continue
Process 237146 has exited with status 0

which just means that are program run successfully and there is nothing else to see here 👀. Happy as we are that are program finished successfully we probably wanted to see more information about what happens. The way to do that is by setting breakpoints, like we would in our IDE but a bit more manual 🙂.

main.go file with the for loop and the two auxiliary functions used to call each other

Add breakpoints

So let’s set some breakpoints. We will set one breakpoint on line 25 (the outer most loop that we have in our main function) and one on line 27 (the line where we change the value of k).

(dlv) restart
Process restarted with PID 238493

The restart here is going to restart the execution of our program (required as we have completed the current run and there is nothing else delve can do at this point).

(dlv) break main.go:25
Breakpoint 1 set at 0x467aaa for main.main() ./main.go:25
(dlv) break main.go:27
Breakpoint 2 set at 0x467ad5 for main.main() ./main.go:27

The following two commands are the break commands that create breakpoints on lines 25 and 27 of the main.go file that we have. That’s it 🎉!

Listing existing breakpoints

The breakpoints command allows us to see what is already set and we can see our 2 breakpoints set on lines 25 and 27.

(dlv) breakpoints
Breakpoint 1 at 0x467aaa for main.main() ./main.go:25 (0)
Breakpoint 2 at 0x467ad5 for main.main() ./main.go:27 (0)

Removing breakpoints

The clear command allows us to delete an endpoint that we have already set. It takes one argument which is the numeric value of the endpoint that we saw previously. Running the following and then listing the breakpoints again will only show us the one endpoint that we have remaining on line 27.

(dlv) clear 1
Breakpoint 1 cleared at 0x467aaa for main.main() ./main.go:25
(dlv) breakpoints
Breakpoint 2 at 0x467ad5 for main.main() ./main.go:27 (0)

Debugging

Let’s start our debugging and see what else we can read from delve. We saw previously that using the continue command will execute the code to the next available breakpoint (and since we had none it run the entire program). Now that we do have a breakpoint though we would expect that the execution of our program will stop at the point where we set our breakpoint. Let’s try it.

(dlv) continue
> main.main() ./main.go:27 (hits goroutine(1):1 total:1) (PC: 0x467ad5)
    22: func main() {
    23:         k := 100
    24:
    25:         for i := 1; i <= 10; i++ {
    26:                 for j := i; j <= 10; j++ {
=>  27:                         k += i
    28:                 }
    29:         }
    30:
    31:         _ = nest1(1)
    32: }
(dlv)

The output of continue now is quite different. It has run the program up until the line 27, it has printed the lines around our breakpoint and has returned to the (dlv) prompt in our terminal. If we wish the execution to resume again we should give the continue command again.

So what else would we like to do now? It would super interesting to see the values of our variables at this point of the execution. We can do that by using another command called print.

(dlv) print i
1
(dlv) print j
1
(dlv) print k
100

If we wanted to move to the next point we would do continue again and get a similar output with above. And if we wanted to see the values of i, j and k we would need to type in again the print commands that is a bit tedious to repeat. Luckily delve provides a nice command to help us out with this.

(dlv) on 2 print i
(dlv) on 2 print j
(dlv) on 2 print k

These commands instruct delve to run the print commands every time we hit breakpoint 2. So running this again we now get the following output with the values of our variables already printed out 🎉!

(dlv) continue
> main.main() ./main.go:27 (hits goroutine(1):2 total:2) (PC: 0x467ad5)
        i: 1
        j: 2
        k: 101
    22: func main() {
    23:         k := 100
    24:
    25:         for i := 1; i <= 10; i++ {
    26:                 for j := i; j <= 10; j++ {
=>  27:                         k += i
    28:                 }
    29:         }
    30:
    31:         _ = nest1(1)
    32: }
(dlv)

Stack traces

Another very useful thing that we can see in our terminal is the stack trace for the code that we are currently executing.

(dlv) stack
0  0x0000000000467ad5 in main.main
   at ./main.go:27
1  0x0000000000435b2f in runtime.main
   at /usr/lib/go-1.15/src/runtime/proc.go:204
2  0x0000000000464261 in runtime.goexit
   at /usr/lib/go-1.15/src/runtime/asm_amd64.s:1374
(dlv)

Let’s see something going a bit deeper than just hitting our main function. I will set up a breakpoint inside one of the auxiliary functions mentioned above and run the program up until that point.

(dlv) break main.go:19
Breakpoint 3 set at 0x4679dc for main.nest2() ./main.go:19
(dlv) clear 2 # removing to quickly go to the next breakpoint just created
Breakpoint 2 cleared at 0x467ad5 for main.main() ./main.go:27
(dlv) continue
> main.nest2() ./main.go:19 (hits goroutine(1):1 total:1) (PC: 0x4679dc)
    14:         ret = append(ret, 2)
    15:
    16:         i1 += 1
    17:         i2 += 1
    18:
=>  19:         return ret
    20: }
    21:
    22: func main() {
    23:         k := 100
    24:

The output of continue is different now, since we moved to a different part of the program. Asking to see the stack trace again will show us more information about the depth of the calls we have made

(dlv) stack
0  0x00000000004679dc in main.nest2
   at ./main.go:19
1  0x00000000004678b2 in main.nest1
   at ./main.go:6
2  0x0000000000467b13 in main.main
   at ./main.go:31
3  0x0000000000435b2f in runtime.main
   at /usr/lib/go-1.15/src/runtime/proc.go:204
4  0x0000000000464261 in runtime.goexit
   at /usr/lib/go-1.15/src/runtime/asm_amd64.s:1374

Now the trace tells us, that apart from the core go calls and the call to main we have 2 more calls in our stack trace, the first one is from the main function to the nest1 function and from there to the nest2 function.

And that’s it – you can now use delve from the command line and explore all available options. You can get information about the threads and goroutines of your program, on the fly change the values of your variables and even inspect the contents of your CPU registers!

[Bonus] CPU registers

This is a bit more of a deep dive on how the processes store information about the current execution of the program, so feel free to skip but I thought it was so interesting to see that I had to put a reference in 😎. By typing regs in the terminal you can get a dump of the CPU registers at this point in time of the execution of your program. The output will look something like this.

(dlv) regs
    Rip = 0x0000000000467ad5
    Rsp = 0x000000c000030750
    Rax = 0x0000000000000002
    Rbx = 0x0000000000000000
    Rcx = 0x000000c000000180
    Rdx = 0x0000000000483370
    Rsi = 0x0000000000000060
    Rdi = 0x0000000000000000
    Rbp = 0x000000c000030778
     R8 = 0xffffffffffffffff
     R9 = 0x7fffffffffffffff
    R10 = 0x0000000004000000
    R11 = 0x00000000004dd460
    R12 = 0x0000000000203000
    R13 = 0x0000000000000000
    R14 = 0x0000000000000058
    R15 = 0x0000000000000004
 Rflags = 0x0000000000000293    [CF AF SF IF IOPL=0]
     Es = 0x0000000000000000
     Cs = 0x0000000000000033
     Ss = 0x000000000000002b
     Ds = 0x0000000000000000
     Fs = 0x0000000000000000
     Gs = 0x0000000000000000
Fs_base = 0x00000000004c71f0
Gs_base = 0x0000000000000000

Where the Rip register is the Instruction Pointer that holds the address of the current instruction being executed. If you remember from before we had this line being printed once we hit continue on our breakpoint.

(dlv) continue
> main.main() ./main.go:27 (hits goroutine(1):2 total:2) (PC: 0x467ad5)

If you notice at the end there is a (PC: 0x467ad5) which is actually the value of the Rip register!

Additional resources

Leave a Reply