A primer on debugging Native Client code in Chrome

This isn’t your father’s your average client-side app.

Chromium NaCl

Native what?

Yesterday, I was faced with an unfamiliar 10k line C program that did custom image manipulation. It takes in two arguments: an input image file and a destination for the resulting output file.

This is straightforward to run on the server-side, but I don’t want to maintain varying compute capacity just for an infrequently run, on-demand image conversion script! Before you mention it, no, I am not about to rewrite the entire program in JavaScript, no matter how fun that sounds.

Enter Chromium Native Client, an “open-source technology for running native compiled code in the browser”. Combined with Pepper.JS, I can run binaries in a browser window! Before you bring out the pitchforks, consider that the C program is almost a perfect fit for this! As long as I write a little code to manage that pesky file I/O, it should be smooth sailing from here, right? Not exactly. Turns out the Pepper.JS docs weren’t kidding about getting your hands dirty.

Moving from JS on the backend to C on the frontend

Downloading the SDK and playing with the File I/O example to get a feel for it showed that simple file I/O operations such as fseek were failing and I had no clue why. Error messages were useless of course, as is often the case:

NaCl I/O Demo

Firing up Developer Tools in Chrome shows that I am faced with a binary .nexe blob to debug, as expectedxkcd. A little searching reveals a few ways to debug Native Client apps, but for the purpose of this post, I am going to detail what I found to be the most straightforward and fruitful approach.

What I would give for a breakpoint

Let’s start with the prerequisites. Instructions on how to set up the Native Client SDK will get you set up with a working environment to build NaCl modules in. I am focusing on NaCl I/O, and working in a Mac OS X environment with Chrome 31, which has PNaCl enabled by default.

$ cd nacl_sdk/pepper_31/examples/demo/nacl_io

First let’s start up our server. I’ve found that the documentation isn’t very clear regarding the various toolchains and config options, but here is the simplest way to get up and running:

$ make CONFIG=Debug TOOLCHAIN=newlib
$ make serve CONFIG=Debug
make -C /Users/vinod/nacl_sdk/pepper_31/src/nacl_io STAMPDIR=/Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug/nacl_io.stamp
STAMP /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io/newlib/Debug/nacl_io.stamp
LINK newlib/Debug/nacl_io_x86_32.nexe
LINK newlib/Debug/nacl_io_x86_64.nexe
LINK newlib/Debug/nacl_io_arm.nexe
CREATE_NMF newlib/Debug/nacl_io.nmf
python /Users/vinod/nacl_sdk/pepper_31/tools/httpd.py -C /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io
Serving /Users/vinod/nacl_sdk/pepper_31/examples/demo/nacl_io on http://localhost:5103/...

This should start a http server at http://localhost:5103/ that you can navigate to and break, as I did above. Now we just need to get a grasp of what exactly broke, causing that vague error.

Enter GDB. GDB is by far the most powerful tool we have in our belt. If you are unfamiliar with how GDB works, I recommend glancing through an overview before proceeding further. As always, the first step is to modify our current Makefile to add debugging symbols to our executable:

--- Makefile    2013-11-18 04:45:56.000000000 -0800
+++ Makefile    2013-11-18 00:35:21.000000000 -0800
@@ -14,7 +14,7 @@
DEPS = nacl_io
LIBS = $(DEPS) ppapi pthread
-CFLAGS = -Wall
+CFLAGS = -Wall -g -O0
SOURCES = handlers.c  nacl_io_demo.c queue.c

We quit out of the server with Ctrl-C and run make serve CONFIG=Debug again.

The next step is to enable GDB debugging in Chome. Although the docs point you to running chrome off the command line with various flags, I’ve found that navigating to chrome:flags, enabling “Native Client GDB-based debugging” and re-launching Chrome is enough.

Navigating to  http://localhost:5103/ should now show the application waiting for us to connect via nacl-gdb.

NaCl I/O Demo

nacl-gdb is located in our SDK:

$ find ~/nacl_sdk -name "*-nacl-gdb"
/Users/vinod/nacl_sdk/pepper_31/toolchain/mac_x86_glibc/bin/i686-nacl-gdb
/Users/vinod/nacl_sdk/pepper_31/toolchain/mac_x86_glibc/bin/x86_64-nacl-gdb
/Users/vinod/nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/i686-nacl-gdb
/Users/vinod/nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/x86_64-nacl-gdb

Recall we are using newlib from earlier. Architecture varies, but uname -a should be able to give you a clue as to what to choose. In my case I am going to run nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/x86_64-nacl-gdb.

In addition, there are a couple of GDB commands we need to run first. We will put them all in a file, for easy repeated debugging:

nacl-manifest "newlib/Debug/nacl_io.nmf"
nacl-irt "/Applications/Google Chrome.app/Contents/Versions/31.0.1650.57/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe"
target remote localhost:4014
  • nacl-manifest points to our NaCl application’s manifest file. I use a relative path since I will be running gdb from the demo application directory.
  • nacl-irt points to the Native Client Integrated Runtime. You can find it easily:
    $ find "/Applications/Google Chrome.app" -name "*irt*"
    /Applications/Google Chrome.app/Contents/Versions/30.0.1599.101/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    /Applications/Google Chrome.app/Contents/Versions/31.0.1650.48/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    /Applications/Google Chrome.app/Contents/Versions/31.0.1650.57/Google Chrome Framework.framework/Internet Plug-Ins/nacl_irt_x86_32.nexe
    

    Just choose the one that corresponds to the version of Chrome you’re running. It’s usually the highest version listed.

  • target points to a TCP port, 4014, being listened on by Chrome that we connect to to debug.

Now we can invoke GDB. Make sure you first have the NaCl application server running in a separate terminal, and have navigated to localhost:5103. The order is important here. If all goes well, you should be presented with a prompt.

$ ~/nacl_sdk/pepper_31/toolchain/mac_x86_newlib/bin/x86_64-nacl-gdb -x nacl_gdb_debug
GNU gdb (GDB) 7.5.1 20130827 (Native Client r12067, Git Commit aeafd16b903de784108b15534ccdab8b1a83f8bb)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i386-apple-darwin10.8.0 --target=x86_64-nacl".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
0x0fa00200 in ?? ()
(gdb) b handlers.c:197
Breakpoint 1 at 0x206e6: file handlers.c, line 197.
(gdb) c
Continuing.

Above, I set a breakpoint at line 197 in handlers.c, where an fopen is performed in my application code. I then continue so the process can keep running. Navigating back to the web page, we see that the application is running now:

NaCl I/O Demo

I am going to try out the http mount point. Entering in “http/index.html” and clicking fopen results in the break point being triggered.

NaCl I/O Demo

Breakpoint 1, HandleFopen (num_params=2, params=0x3ec3ff8c, output=0x3ec3ff88) at handlers.c:197
197      file = fopen(filename, mode);
(gdb) s
198      if (!file) {
(gdb) l
193
194      filename = params[0];
195      mode = params[1];
196
197      file = fopen(filename, mode);
198      if (!file) {
199        *output = PrintfToNewString("Error: fopen returned a NULL FILE*.");
200        return 2;
201      }
202
(gdb) c
Continuing.

But wait! Trying to step in took us right past. Unfortunately, this appears to my untrained eye to be a kernel-level syscall, which is magically intercepted and handled by the SDK. But how do we get in there?

Stepping through all the things

The main application file, nacl_io_demo.c, contains headers pointing to a nacl_io library:

…
#include "ppapi/c/ppp_messaging.h"
#include "nacl_io/nacl_io.h"
…

That looks promising. The NaCl SDK source code is located at src/ under the root SDK directory. For me that would be ~/nacl_sdk/pepper_31/src/, with headers in include/. Opening nacl_io.cc reveals more clues, which eventually leads us to a file named suspiciously relevant to our hunt: mount_http.cc. This contains a MountHttp::Open method that appears like a great place to set a breakpoint. But how do we get there?

Just like we did for our application, we need to include debugging symbols for the Native Client SDK. We modify the Makefile to add the same flags we did above:

…
CFLAGS = -Wall -g -O0
…

and then compile:

$ make TOOLCHAIN=newlib CONFIG=Debug

With 4 am fast approaching, we delve back into our debugging process:

  • Kill GDB with Ctrl-C and then Ctrl-D, and quit.
  • Stop the application server, and start it back up again.
  • Refresh the web page to reload the application.
  • Start GDB back up again.

This time, we set a break point in mount_http.cc:

(gdb) b mount_http.cc:83
Breakpoint 1 at 0x49dc0: file mount_http.cc, line 83.
(gdb) c
Continuing.

Clicking fopen() on the web page brings us to it:

Breakpoint 1, nacl_io::MountHttp::Open (this=0x3eee1c60, path=..., mode=0, out_node=0x3ec3fdd0) at mount_http.cc:83
83        error = node->GetStat(NULL);
(gdb) bt
#0  nacl_io::MountHttp::Open (this=0x3eee1c60, path=..., mode=0, out_node=0x3ec3fdd0) at mount_http.cc:83
#1  0x00084ee0 in nacl_io::KernelObject::AcquireMountAndNode (this=0x3eee04d8, path=..., oflags=0, out_mount=0x3ec3fdd4, out_node=0x3ec3fdd0) at kernel_object.cc:102
#2  0x00028fa0 in nacl_io::KernelProxy::open (this=0x3eee04d8, path=0x3eee185e "http/index.html", oflags=0) at kernel_proxy.cc:161
#3  0x00024900 in ki_open (path=0x3eee185e "http/index.html", oflag=0) at kernel_intercept.cc:131
#4  0x00043260 in __nacl_irt_open_wrap (pathname=0x3eee185e "http/index.html", oflag=0, cmode=0, newfd=0x3ec3fe6c) at kernel_wrap_newlib.cc:114
#5  0x00176100 in open (pathname=0x3eee185e "http/index.html", flags=0) at open.c:34
#6  0x00149580 in _open_r ()
#7  0x00141e80 in _fopen_r ()
#8  0x00142060 in fopen ()
#9  0x00020760 in HandleFopen (num_params=2, params=0x3ec3ff8c, output=0x3ec3ff88) at handlers.c:197
#10 0x00022420 in HandleMessage (message=0x3eee1858 "fopen") at nacl_io_demo.c:246
#11 0x00022660 in HandleMessageThread (user_data=0x0) at nacl_io_demo.c:280
#12 0x00092840 in nc_thread_starter () at nc_thread.c:118
#13 0x0fa00700 in ?? ()
#14 0x00000000 in ?? ()

By typing in bt, I obtained a backtrace of how in the world we got here. This clears up some of the ambiguity about the black magic that makes NaCl apps work.

From here on, it’s just a matter of stepping through the code, setting breakpoints and, god forbid, print statements if you prefer the back-to-basics approach, till you know enough to solve the problem. Be aware that unless you turn off the debugging flag at chrome:flags, your NaCl application will always wait for nacl-gdb to connect. So be sure to turn it off after you’re done debugging.

In my case, the bug turned out to be in the SDK after all and I filed a bug report (here, if you’re curious). A workaround was easy enough to implement. More on what the hell I’m doing with my C script in a future blog post.

So that was my first day with NaCl… mostly debugging, although I hope to eventually be able to port over legacy desktop apps to the browser for fun and profit.

Does this kind of silliness interest you? Kloudless is hiring Front-End Engineers! If you know when to get your C++ JavaScript stirred rather than shaken, check out https://kloudless.com/jobs#frontend and shoot us your CoffeeScript/TypeScript/Dart preference at work@kloudless.com.

4 thoughts on “A primer on debugging Native Client code in Chrome

  1. Nice post! I’m sorry debugging was so difficult for you, but I did want to mention that there is a way to debug that is easier than you’ve described:

    try running “make debug” from inside the example directory. This will build your example, launch a web server serving on port 5103, launch Chrome with the “–enable-nacl-debug” flag, and launch a new terminal with nacl-gdb running and your .nexe loaded.

    At this point you can type “target remote :4014” in gdb to connect to the gdb debug stub in Chrome, then use gdb as normal (i.e. add breakpoints with “b”, continue with “c”, etc).

    This link in the documentation describes this as well: https://developers.google.com/native-client/dev/sdk/examples#debugging-the-sdk-examples

    It looks like this is not mentioned in the page on debugging, though it should be.

    • Thanks for the input binji! `make debug` was one of the approaches I took initially (did not link to it though), but I had trouble on my Mac with CHROME_PATH so I scrapped it and went with the manual approach above.

      I just took another glance at it and it turns out the wildcard expansion in common.mk doesn’t like spaces in the CHROME_PATH. In addition, the base name is always “google-chrome” so I’d have to create a symlink for it. I proceeded with those steps but still had trouble with subprocess forking.

  2. This was helpful to me thanks very much!

    It works for me as long as I am dealing with indexl.html loaded on the foreground thread. As soon as I try any of the examples using background threads I can no longer make these techniques work. (e.g. examples\api\file_io\ or examples\demo\nacl_io_demo.

    These apps work by using background.js to launch a separate window that operates on a different thread. I’m not sure where the process is breaking down.
    1. Make Serve Step succeeds.
    2. launch chrome. – Well, we don’t want to enter the index.html on the command line, because normally it appears in a separate window. I can see from the debug output in the ‘launchchrome” window that the background.js script has run, but no second window is launched.
    3. gdb command. gdb launches, and the nexe is loaded (I can view source and set breakpoints…) target remote localhost:4014 succeeds, and I can ‘c’ontinue.

    Other notes:
    * We are using pepper_37
    * had to drop back from the latest chrome, or nothing worked.
    Using chrome version 39.0.2171.95

    Still, app is not fully launched 😦
    Any hints would be appreciated.

    Thanks Steve

What're your thoughts!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s