fdb, the Flex debugger: Internals of how stepping works

Since I left Adobe last year, I haven’t done much work in Flash or Flex, so I haven’t written much about them on this blog. (In fact, I haven’t written very much about anything – this poor blog has been somewhat neglected.)

But a recent email exchange with Ilan Avigdor and Yoav Moran contained some information that I think is worth sharing here. They are “creating a tool for visualizing code execution for AS3,” and had some questions about the internal workings of fdb, the free open-source command-line debugger that comes with the Flex SDK. I wrote back to them with some details, and I thought I would repost my notes here. (This post is pretty rough, because I’m just taking the email conversation, doing a little light editing, and then posting it.)

The part they wanted to know about is how stepping (step into, step over, etc.) is handled by fdb.

A command-line debugger for Flex? And it’s free? And it’s open-source?

But first: Yes, it’s true, there is a free, open-source command-line debugger for Flex. I get the feeling a lot of people are unaware of that. Just as the command-line compiler, mxmlc, is free and open-source, the same is true of the debugger, fdb. It, like all the other Flex tools, is written in Java.

fdb.jar actually serves two purposes:

1. It is a Java library with an API that is callable from any Java program you write. E.g. if you wanted to write your own WYSIWYG Flex debugger, you could just put fdb.jar inside your app, and then call its Java API – stepInto() and so on.
2. In addition, in the same jar is the command-line debugger. The command-line debugger portion of fdb.jar makes calls to the API portion of fdb.jar.

The Java library portion of fdb.jar is actually opening a socket and connecting to the Flash Player which is running in a separate process (e.g. in your web browser). Yes, there is a wire protocol, so you don’t have to use fdb.jar – you could write directly to the wire protocol – but that is harder, especially since the wire protocol isn’t publicly documented.

Reading the source

Top-level location: http://opensource.adobe.com – this is the entry point for all of Adobe’s open-source code.

Flex SDK location: http://opensource.adobe.com/svn/opensource/flex/sdk/trunk – this is a Subversion repo, so for example you can download all the sources with this command, to put them in a subdir called “flex” under the current directory:

svn co http://opensource.adobe.com/svn/opensource/flex/sdk/trunk/ flex

(That will pull down a lot more than just fdb.) fdb is in the modules/debugger subdirectory of that. Under modules/debugger, src/java/flash contains the API portion, and src/java/flex contains the code for the command-line fdb program. The class flex.tools.debugger.cli.DebugCLI is the entry class of the whole fdb program.

How stepping works

First, as in many debuggers, there are three kinds of stepping:

• Step into: Single-step, and if we step into a function call, then halt at the first line of that function call.
• Step over: Step over the current line until we reach the next line of the current function. Do not halt inside nested functions.
• Step out: Step until we exit the current function.

The function runningLoop() is one of the more important functions related to stepping: It is in charge of repeatedly telling the Flash player to resume until it decides it is time to stop.

Here is how stepping works:

When fdb sends any of the stepping-related commands to the flash player – step in, step over, or step out – the player says, “Okay then, how deep is the callstack at the present time?” and then uses either that number (for step over), or that number plus one (for step in), or that number minus one (for step out) as the target callstack depth for when the step command will be finished. And Flash’s opcode instruction set includes a “debugline” opcode that means “I am at the beginning of a line of source code” – so the Flash player will only do this check of the callstack depth each time it reaches a “debugline” opcode.

E.g. suppose the callstack has three functions on it, and you call “step over”. Then the player records the number 3, which means, “the next time I reach a debugline opcode and the callstack depth is 3 or less, halt.” At that point, the code might call into another function (so the callstack depth will be 4), and that function might call into a whole bunch of others (so the callstack depth will keep getting deeper); but eventually it will pop back to the next line of the original function, and the callstack depth will be 3, and it will halt.

Next scenario: Instead of step over, you call “step in”. The player then records a target callstack depth of 4 – again, this means, “the next time I reach a debugline opcode and the callstack depth is 4 or less, halt.” Which in essence means that the very next debugline opcode will cause it to halt, no matter whether the current line of code called another function (in which case the callstack depth will be 4, because you’ll be in the new function), or just did something like an assignment to a variable (in which case it will be 3, because you’ll be on the next line of the same function), or did a “return” (in which case it will be 2, because you will have returned to the caller). But it’ll halt.

Last scenario: You call “step out”. Then the player records a target callstack depth of 2. It will keep stepping until the current function eventually returns.

Okay, so, your app: You always call “step in”. Now suppose the current callstack has only one function on it (so it has a callstack depth of 1); and you call “step in,” so the player records this as “halt the next time we reach a debugline opcode and the callstack depth is 2 or less”; but then the current function does a “return”. We are no longer in ActionScript code – the Flash player’s native code is back in charge. It can do whatever it wants, but the key point is, eventually, the next time it calls any ActionScript code for any reason, and a “debugline” opcode is encountered, the ActionScript part of the Flash player will notice that it is supposed to stop. It doesn’t matter that the code that is now executing is totally unrelated to the code that was executing before (e.g. the beginning of a new frame or whatever) – it will stop.