Brave New-DB World

**TL;DR* This summer one of my projects as a Khan Academy intern was to work with Google App Engine’s new NDB datastore interface. This post is an account of our experience with NDB. If you’re interested in upgrading your own application to NDB, my next post is a condensed how-to based on what we learned.*

Google App Engine and its BigTable-backed datastore have been huge wins for the Khan Academy engineering team in terms of implementation speed and flexibility, which is why we recently became interested in NDB, the next-generation Python API for this datastore. Developed by Python’s benevolent-dictator-for-life Guido van Rossum, NDB incorporates a sophisticated automatic multi-layer caching mechanism and a Monocle-inspired asynchronous API that allows for auto-batching and parallel execution of datastore operations. That’s quite the mouthful of improvements, so it’s no surprise that upgrading db.Models to ndb.Models requires a little more work than a brute force find-and-replace. For several weeks during my summer internship at Khan Academy I set out to determine a feasible upgrade path that would allow us to transition over to NDB and begin taking advantage of all these new features.

I began my excursion into all that is NDB by upgrading our feedback models, which power the questions, answers, comments, and moderation tools underneath every video on the site. The initial goal was to do a “mechanical” translation to the new API without yet significantly reworking the code to take advantage of NDB’s new features. Guido’s NDB Cheat Sheet and the official NDB documentation served as my field guide to all of the API changes. Our codebase contains over 70 db.Model subclasses with about 20 ReferenceProperty relationships between them. That, combined with the speed at which we’re shipping these days (must be all those interns), meant that an all-or-nothing refactoring was pretty much off the table from the start. Unfortunately, NDB is different enough from “old DB” that an incremental upgrade brings its own set of challenges. ack and our test suite got me pretty far along to getting the feedback models working, but several challenges remained. Some interesting questions I faced when completing the conversion:

  • How can I avoid runtime errors popping up everywhere once the new code goes live and all of the code paths get hit by our users?
  • Will there be any unfavorable interaction between our custom layer cache and NDB’s automatic caching?
  • Does our data export and analytics pipeline need to be made NDB-capable?
  • Can a db.Model reference an ndb.Model, or vice-versa?

After several discussions and code reviews we ended up with a fairly solid changeset but also some remaining doubts about code coverage and performance characteristics. Feeling optimistic, we shipped it, watched our logs, and quickly pushed out fixes as we noticed errors stemming from things I had missed. Performance-wise, there were no problems and some things were even a little faster than they were before. We avoided caching issues by renaming the updated models' cache keys. And using aetycoon’s KeyProperty let us store ndb.Model references in db.Models.

After shipping those changes and working out the kinks, I was in a good position to turn on some of the more advanced capabilities of these newly NDB-ified models. Taking advantage of the async APIs yielded interesting efficiency improvements, with a couple of caveats:

  • Datastore, memcache, and urlfetch RPCs can each be auto-batched, but the auto-batcher’s decisionmaking process is complex, somewhat challenging to trace, and can sometimes behave different from how you’d expect it to.
  • The async APIs don’t play nice with appstats. Datastore RPCs in appstats might look like they’re taking a really long time when in fact the application was just doing other things in between getting a future from firing off the RPC and calling get_result() on that future. And coroutine control flow makes tracebacks pretty opaque; it’s hard to tell where an RPC actually originates from even if you bump up the appstats traceback limit.

All of this went well enough, but the biggest question remaining was how to scale up this refactoring work to succeed on more substantial segments of the codebase without causing painful, buggy deploys. The advantages of NDB are clear, but we had little interest in trudging down a long, winding road paved with TypeErrors to get there. For example, I wanted to convert our Video models next, but due to the design of our Topic Tree that would also require modifying the tree code and our Exercise models–which was going to result in a large diff that would be cringe-worthy to test and merge into production.

My mentor Chris had the interesting idea of using Python’s metaprogramming capabilities to provide an NDB interface for old-db models, thereby facilitating incremental upgrades on a per-codepath basis. He wrote a shim metaclass to do just that, and I tested it out on our code path that logs video views. It worked, and the resulting diff was a few hundred lines of code instead of a few thousand, but the cognitive load of using the shim was entirely different. Adding metaclasses to the mix proved to be a little too mind-bending for this intern, to the point where I felt more comfortable wrangling larger diffs that didn’t contain any magic.

Overall, our first NDB experience taught us a lot and gave us the knowledge we needed to create a well-informed plan for moving forward. We decided that henceforth all brand-new models should be done using NDB so that they can leverage its improvements from the start. As for existing models, we’re leaving the decision to individual teams. They’re likely to see at least some reward if they go through the conversion process, but having intimate knowledge of the code paths involved makes them the best ones to make that call. One thing is clear, though: NDB is yet another data point proving that App Engine’s future is bright, and we can’t wait to see what else the platform has in store.