Entries Tagged 'interviews' ↓

Adrian Holovaty is the co-creator of Django, author of the Django book and is currently the BDFL of Django along with Jacob Kaplan-Moss. He studied journalism at University of Missouri–Columbia and has worked with many news sites including Lawrence Journal World and Washington Post. He currently works at EveryBlock, a startup he founded.

Shabda: Would you tell a little about yourself? You majored in Journalism, how did you move to Programming?

Adrian: Sure! I have a dual background in journalism and computer science. I intended to get a degree in journalism and a minor in computer science, but things got a little off track. I heard from a professional colleague that a certain Web site would probably hire me if I graduated a semester early, so I hurriedly dropped my CS minor and got special permission to graduate early — but once the time came to ask for that job, the company was in the middle of a hiring freeze and didn’t have a position for me. So, in hindsight, I should’ve stuck with the minor, but things ended up working out OK.

Because journalism and computer science don’t normally go together, I’ve had some success in this silly little niche of employing Web development in news organizations — “journalism via computer programming.” Professionally, I’ve always worked as a Web developer at news organizations, up until my current gig, which is to run EveryBlock. But in its own way, it’s a news organization as well.

Aside from all of that, I’m quite into music, particularly gypsy jazz, which is the music of Django Reinhardt (hence the name of the Web framework). I just recently attended a week-long gypsy jazz guitar camp in Massachusetts and had a tremendous time staying up until early hours of the morning jamming with great musicians from various parts of the world. If money were no object, I would love to play music full time. In the meantime, I’ll settle for posting YouTube videos of my guitar playing, for an audience that seems to be comprised mostly of 16-year-old boys who constantly pester me for transcriptions of my arrangements.

I live with my wife in the beautiful city of Chicago.

Shabda: What are your current roles and responsibilities at Django?

Adrian: Well, I co-created Django back in 2003 with my friend Simon Willison while we were working together. Now, along with Jacob Kaplan-Moss, I’m Benevolent Dictator For Life of the project — a title we shamelessly ripped off from Guido/Python. This means that I have a hand in all sorts of things, from high-level API design to checking in patches and fixing bugs. I’ve touched probably every bit of the framework over the years, including implementing the initial ORM (back when it was a code generator!), pair-programming the original Django template system and URLconf/view framework with Simon and building the original admin application with design help from Wilson Miner.

One of the things I enjoy the most is writing documentation — both because I like technical writing and because I cannot stand bad open-source documentation.

Shabda: Can you describe the philosophy behind Django. What are its overarching goals?

Adrian: A couple of years ago, I wrote the “Design philosophies” document, which sums up the main points nicely.

Generally, the goal is to make Web development fast, fun and easy for the developer, while keeping performance as fast as possible and code as easy to understand as possible.

Shabda: How did the move at WorldOnline from PHP to Python happen? Why did you start from scratch (which finally lead to creation of Django), instead of using something like Zope?

Adrian: Simon and I had been big fans of Mark Pilgrim (and still are!), and we’d read his online book “Dive Into Python.” This was around the same time that the PHP “infrastructure” I’d developed had begun to feel really, really crufty, so…one day we just decided to start using Python! It’s quite nice working at a small organization with a very loose management structure; our boss, Rob Curley, was cool enough to let the developers themselves decide which technologies to use, as long as the work got done. “I don’t care how the sausage is made,” he always used to say.

We looked at the existing Python Web frameworks/libraries, but none of them felt 100% right to us. Simon and I are both quite into best practices, such as clean URLs, proper use of GET vs. POST, etc., so we were very picky in our analysis of those existing tools.

Shabda: Can you tell us about Everyblock? Why did you choose to create your own mapping engine instead of using something like Google Maps? How hard has it been creating a new mapping engine?

Adrian: EveryBlock is the project I lead for my day job. It’s funded by a two-year grant from the Knight Foundation, and the goal is to experiment with block-specific news — that is, the site lets you enter an address (currently only in Chicago, New York City and San Francisco) and view recent news within a very tight radius of that address. I’m pretty confident it’s the most granular attempt at local news ever attempted on the Web. Nobody else is crazy enough to do it, I guess!

We do a TON of work collecting local information, normalizing it and pulling it together for people in one place. A large part of what we publish is government data such as crime reports, restaurant inspections, building permits, business licenses…and even local movie filmings. Much of this stuff either is buried in “deep Web” government databases or has never before been available online. A second part of what we do is detecting addresses and locations in news articles and blog entries. Plus, we pull in various other geographic Web stuff like Flickr photos and Yelp reviews.

All in all, our goal is to show you recent, geographically relevant stuff that you might not have heard about. Say the local newspaper wrote something about your neighborhood on page B-23 — would you really have noticed that article? Say there was an aggravated assault on your block — would you really have remembered to check your police department’s crime-reports Web site on a daily basis? That’s the basic philosophy.

Regarding our custom mapping engine…Let’s face it: Google Maps is so passe. As one of the original Google Maps mashup guys (I developed chicagocrime.org by reverse-engineering Google’s JavaScript, before the API was released), I have all the respect in the world for what Google has done to invigorate the world of Web maps. But it’s time to take the next step. The Google Maps API doesn’t give you any control over the colors of the map tiles, or change the fonts in street labels, or disable building footprints, or hide one-way street markers or subway stations or bus stops or any of the other stuff that’s essentially hard-coded in the map. So, at EveryBlock, we rolled our own maps so we could have much, much more fine-grained control over all of these things. Not to mention it’s a great way to differentiate ourselves from the 2.5 million boring, same-old-yellow-blue-orange Google Maps mashups out there.

Paul Smith from the EveryBlock team has written an article at A List Apart with more of the technical specifics.

Shabda: As per the Knight grant rules, you would be releasing the code for Everyblock next year. Would you not be giving away your secret sauce? How do you plan to maintain competitive advantage once that code is freely avaliable under a permissive license?

Adrian: Our competitive advantage is that we’re an incredible team, and I’m sure we’ll come up with a way to feed our families.

Shabda: What problems would you like Django to tackle after 1.0? What big features would you be most interested in having in Django, after it hits 1.0?

Adrian: Generally I’d like to add higher and higher levels of abstraction to the framework. The Django admin application is a good example of that – it’s not just an abstraction of an HTTP request; it’s an abstraction of an entire Web application! We should do more of those.

Shabda: What is one thing about Django which you absolutely love, and one thing which you think Django should do differently?

Adrian: I love the way URLconfs work — like a table of contents for your Web app. I also love template inheritance. I don’t love the fact that we’re generally slow in keeping up with tickets and feature requests.

Shabda: A lot of people these days have to evaluate between Django and ROR. How can they make this decision? When should Adrian Holovaty use ROR? When should DHH use Django?

Adrian: My answer is simple: Try out both frameworks and see which one you like better.

Shabda: Does Django need to market itself differently? What can Django community do for this?

Adrian: I’m comfortable with the amount of attention (or lack thereof, depending on your perspective) that our project gets. I’m comfortable with the size of the community. I’m comfortable with the fact that the right people have found out about it through word of mouth, books, blogs, Google App Engine or wherever else. If things continue at their current rate, we’ll continue to do just fine, as a healthy open-source project.

The one thing I’d ask the community to do is to continue staying civil, polite and approachable.

Shabda: What does the phrase ‘journalism via computer programming’ mean? How can these two divergent fields be tied together?

Adrian: “Journalism via computer programming,” in my opinion, is when a journalist writes code to tell a story. Instead of talking on the phone with sources to gather facts, this could involve screen-scraping Web sites to gather raw data. Instead of writing a newspaper article, this could involve building a database-driven Web site.

Journalism has several subdisciplines — photography, information graphics, video. I advocate that computer programming should be another one of those subdisciplines. Just as a newspaper employs photographers and graphic artists, it should employ programmers who help gather information and tell stories with it.

Shabda: How can traditional journalists do more ‘journalism via computer programming’? How can programmers do more ‘journalism via computer programming’? Is it easier for Programmers to move to this field, or for Journalists?

Adrian: I believe it’s easier for programmers to become journalists than it is for journalists to become programmers, but both sides need to gain an appreciation for the other in order for this sort of thing to happen more often. Fortunately, some news organizations are starting to hire developers with this in mind, and some geeks are realizing journalism is a great, (mostly) pure field that lets you improve the world through information.

One concrete thing programmers can do is to look for jobs at news organizations, which desperately need technical talent. Developers, you will be loved, you will be treated like geniuses, and your non-techie coworkers will be very easily impressed!

Another route programmers can take is to get training in journalism – in fact, Northwestern University’s journalism school is giving out full-tuition scholarships for programmers who want to learn journalism.

Shabda: Thanks Adrian.

This was Adrian’s interview, the last in django-interviews series. But we have a lot of interesting things coming up, so stay tuned.

Jacob Kaplan-Moss is the co-creator of Django along with Adrian Holovaty, as well as the author of the Django Book. He has been involved with Django since before it was called Django. He is currently employed at Whiskey Media where his job is hacking at Django. He blogs on Jacobian.org. He graciously agreed to be interviewed at the 42topics blog.

Shabda: Would you tell a little about yourself? How did you get started with Django? What other software/applications have you worked with. (Both OSS and otherwise)?

Jacob: So a bit about me: I grew up in Silicon Valley, so like many other geeks I got started with computers really early; I was programming professionally before I graduated high school. I didn’t start out doing web development; in college I worked on video surveillance systems for airports, harbors, marinas, and highways. That’s where I found Python: I rewrote a Java-based camera controller in Python and haven’t looked back since. I started doing web development pretty seriously when I moved to New York and took a job for a design firm there. The job was pretty terrible, and the technologies were worse: the good sites were PHP, and there was a bunch of WebLogic crap that was absolute hell to maintain.
So in 2004 I saw the job opening that Simon and Adrian posted on their blogs — and jumped on it. Web development in Python seemed like a dream job, and it really was. So that’s how I ended up in Kansas working for the local paper. At that point I suppose I was using Django, though we didn’t have a name for it yet: we just called it “The CMS”.
I’d been getting more into Open Source all along, though I’d barely contributed (I think I got a single line into Python at one point years ago — that was my largest contribution).

Shabda: What areas of Django have you worked most in? What are you current areas of focus?

Jacob: Well, at this point I’ve touched pretty much every part of Django at some point weather in the form of improvements, refactoring, or just small patches and bug fixes. I’m far from an expert in any particular area, though, so I’d say my main role is more holistic: I’m most concerned with making sure that Django “feels” correct and that APIs and conventions match across the framework.
Right now my current focus is documentation: I’ve been working on improving the structure and organization of the documentation so that it’s easier to find what you’re looking for. We’ve got something like 40,000 lines of documentation so organization and metadata is critical.
Now that I’m lucky enough to get to work on Django at work, I’ll probably be able to take on some more big tasks like that in the future.

Shabda: One of the most loved things about Django is its comprehensive documentation. What is the motivation behind refactoring this? What is going to be the new organization? How is this progressing?

Jacob: Thanks — I’ve always thought that Django’s documentation set us apart from most Open Source projects, and I’ve been really proud of what we’ve done there. However, over the past year or so as the size of the documentation (and Django itself) grew I think we’ve slipped from “outstanding” to merely “above average.” To us (the core maintainers and I) that’s not acceptable.
The best place to read up on the project and goals is in a post I made to Django-dev a couple of months ago. In a nutshell, I’m breaking up the documentation into smaller, more manageable chunks — the current DB API is almost sixty printed pages! I’m also separating more high-level how-to’s and topical guides from the detailed API references that get in the way when you’re just learning.
There’s some tool improvements going on under the hood that’ll make the documentation easier to write, edit, maintain, and contribute to; hopefully that’ll help decrease the number of undocumented features.
The work’s pretty close to done, actually. I was trying to finish before I went on vacation a month ago, but didn’t quite get there. So that means I need to roll in a month’s worth of documentation improvements that happened to the current docs while I was gone, but that shouldn’t take much time. You can expect to see the new docs rolled out online very soon.

Shabda: What does Whiskey Media, the startup you are currently associated with do? Is your role there developing Django, or are you associated with other day to day activities as well?

Jacob: I think I’ll have to be a bit coy and not say very much about Whiskey Media; we’d rather let our actions speak for themselves than try to build some sort of artificial hype. I’ll just say that we’re trying to solve some of the big problems in web publishing; you can probably see why Django’s the best technical bet for a company trying to be the on the cutting edge of content publishing.
Most of my time at Whiskey is devoted to working on Django — coding, but also community management, evangelism, and organization. I also have some internal responsibilities, of course, but those are more nebulous: mostly I just help out wherever I’m needed.

Shabda: As you said “we’d rather let our actions speak for themselves than try to build some sort of artificial hype”. Do you see Django taking a potential hit in marketing due to similar belief of Django community? Does Django need to market itself differently than it has been doing till now? What can Django community do for this?

Jacob: Huh, that’s a tricky one. Well, let me break down the question a bit and look at a few different angles.
So first, I’ve always been a bit uncomfortable with the idea — pervasive in Open Source — that this is some sort of popularity contest. People are always comparing traffic figures, or numbers of job postings, or numbers of Google results, or whatever. I think the idea’s supposed to be that if Project Foo has more users than Project Bar that Foo somehow “wins”. But I don’t see it that way at all.
I mean, I moved from New York City to Lawrence, KS. New York City has a population of, what, 8 million or so? Lawrence has a population of around 80,000. Does that mean New York City is “better”? To most of those 8 million, sure… but to me? Hell no.
As in most areas, there’s no accounting for taste: one of my good friends here in Lawrence is a die-hard Perl fan, and though it makes him a bit twisted it doesn’t mean that I’m somehow “better” than him — we just have different needs and different ways of thinking.
All that said, though, there is undeniably a value in popularity.
Only a tiny number of people who use a given piece of Open Source software become involved in the community, and only a tiny fraction of those contribute back to the project, and only a tiny fraction of those contributors become long-term committers. So more users translates directly to more contributors, and more contributors brings more “value” to the project.
(By “value” I’m referring to code, good ideas, etc.)
So as you can probably tell this is something we struggle with. On one hand I think Django’s great, and it’s in the project’s best interest to persuade others to give it a shot. But on the other hand there’s a huge amount of bullshitting that goes on where software is concerned, and I try my best to not add to the pile.
It’s hard to tell when you’ve crossed the line from evangelism to hype, and hype can be toxic. Witness the recent backlash against Ruby on Rails: do you think they’d get such violent vitriol if they’d been more modest in their promotion?

Shabda: Talking of ROR, a lot of people these days have to evaluate between Django and ROR. What questions should they ask themselves to answer this question? (Well apart from “do you know Python better or Ruby better?”). To make this more interesting when would YOU choose ROR over Django, if you knew both Ruby and Python equally well?

Jacob: So first let me back up a step — I’ll answer directly in a moment — and make a quick point about web development in general. The past couple-three years have seen a radical improvement in the quality of tools available to web developers. It really wasn’t that long ago that CGI represented the state-of-the-art in web development.
Today, though, there’s really some fantastic tools available. Rails, Django, Symfony, Pylons, TurboGears, Seaside… any of these tools represent a major improvement over the CGI/PHP model of web development. If you’re still writing web sites the way you did five years ago, you’re missing out.
When it comes to which of these tools to choose, of course, we’re back to that question of taste. Each tool comes from a different world, and has a different “attitude” towards web development. The cool part, though, is that most new web development tools emphasize a quick start as a key feature: you could probably evaluate a dozen web development frameworks in just a couple of days. So my best advice is to to try a few and see what “clicks”.
It’s probably important to also pay attention to the language that the framework uses. One of the real pleasures of writing Django apps is that you get to take advantage of the awesome Python community. I’ve got a set of libraries in Python that I can’t imagine developing web apps without; many of those libraries don’t have analogues in any other environment.
So if I knew Python and Ruby equally well — I don’t — I’d still probably lean towards Python, and towards Django. However, there’s one place I can think of where Rails is far superior: Rails runs on the JVM. This is a big deal: there’s any number of large corporate environments where the JVM is the only game in town. And obviously if I had to choose between Java and Ruby I’d choose Ruby!
I’ll mention, though, that Jython (Python on the JVM) is improving by leaps and bounds, and that getting Django working perfectly on the JVM is one of the Google Summer of Code projects the Jython team is sponsoring.

Shabda: Coming back to the marketing question.. For most of non hackers choosing the framework to use is a big question, and the decision they will make depend on various non-tech factors, such as availability of capable skilled people. You mentioned that you would choose Django over Rails due to the awesome Python libraries. For the long term survival and growth of Django, do you not think that an early capture of mindshare in developer community is important? For example I have pitched Django to a fair number of people, and I always have to start with “Django, what?” as compared to “Yeah, we are evaluating Rails for our requirements.”

Jacob: Sure, there’s definitely a value of having Django be familiar to your friendly local Pointy Haired Boss, so in that context a certain amount of advertising is important. There’s nothing worse than being forced by management to use the wrong tool for the job. Keep in mind, though, that this cuts both ways: I’d feel pretty unhappy if there was a team using Django because someone read about it in CTO Monthly. I’d agree that we could be doing more in terms of “brand awareness” or something, but all in all I’m pretty happy with the size and quality of Django’s community.

Shabda: There is not much information about the early days of Django, so a little about that. How did the move at WorldOnline from PHP to Python happen? Why did you create a new framework, instead of reusing something like Zope?

Jacob: I wasn’t yet at World Online when we moved from PHP to Python, but from what I understand it was a pretty typical change. Adrian and Simon got fed up with the pain and suffering wrought by PHP, and wanted something cleaner and — most importantly — something that would be easy to maintain. Python really shines here.
The main reason we ended up building our own framework was that we didn’t know we were building a framework. We just wanted to “build cool shit” and, over time, we built tools to help us do that. It wasn’t until we started showing it off that we realized we had something that could be used by other people.
This, by the way, is one of the reasons I think Django turned out so great. If you sit down one day and say, “I’m going to develop a framework!” you’re almost certainly going to become an Architecture Astronaut, and if you ever actually finish the thing’ll be so over designed nobody will want to use it. If, on the other hand, you simply try to solve real-world problems in a clean, obvious way, you’ll eventually end up with a great tool.
Look at Rails, TurboGears, even PHP; they all started as simple libraries written by frustrated programmers just trying to get the job done.

Shabda: What are the overarching goals of Django? Again to make this more interesting, Here is a quote from you. “My work as a core developer of Django focuses on giving anyone — even (especially) non-programmers — the tools to create dynamic, content-driven websites.” Should not that be the job of something like Wordpress, while Django should aim to give “programmers, but not non-programmers — the tools to create dynamic, content-driven websites.”

Jacob: Wordpress is great if you want to publish a blog, but what if you want a website to track a book collection, or sell tickets to concerts, or organize a local farmer’s market? There’s a great deal that Wordpress (and other single-purpose tools) can do, but the amazing thing about the web is just how wide-open the possibilities are. There’s a whole world of possibilities out there, and the end-goal is to help anyone self-publish anything they can dream up.
One of the most fascinating aspects of the history of communication is how intertwined literacy is with social controls. For most of the history of humankind, literacy was strictly available to the elite. The Web created the greatest democratization of publishing ability in history, and has almost immediately turned into a battleground between traditional, centralized publishing and decentralized democratic publishing.
As a programmer, we don’t personally play much of a role in this sea change, but I do see the lowering of the barrier to self-publishing as something we ought to continuously think about.

Shabda: If Django’s goal is “and the end-goal is to help anyone self-publish “, does not that mean Django is trying to fill the niche of an Extensible CMS like Drupal, as compared to filling the gap left by PHP? Are non programmers really using Django, or are they using apps built with Django?

Jacob: No, you’re right that Django’s lower-level than something like Drupal. Django’s not trying to be a CMS but to be a tool you could use to build your own CMS. It’s easier to design your own content models than to shoehorn your publishing into a CMS limited by the ideas of its developers.
There are indeed quite a few non-programmers using Django, though in the long run I think the interesting trend is that more and more computer users are learning a little bit of programming — enough to develop a site with Django, say.

Shabda: (Since I ask this to everyone!) What is one thing about Django which you absolutely love, and one thing which you think Django should do differently?

Jacob: I only get to choose one thing I love?
I think my favorite bit of Django is the URLconf system. I love that Django forces me to think about URL design as part of my application instead of some byproduct; I’ve always hated web tools that try to someone pretend that you’re writing a desktop app. I’ll admit to obsession over my URL design from time-to-time, but I really enjoy clean, semantic URLs.
As for something we should do differently: the assumption that you’ll only use a single database is a bad one, and needs to be fixed. Unfortunately, it’s an assumption that we made really early on, which means that fixing it is going to be tricky. There’s some smart people working on it right now so I’ve got high hopes, but I wish we’d not made that assumption to begin with.

Shabda: Django started as a in house project and was later open sourced. At your current startup your major job is with Django itself. The Django book which you wrote was released under a permissive license. How difficult is it to convince people outside the hacker culture of the business value of open source? What are the best ways to do this? How difficult was doing this at World online?

Jacob: These days, thankfully, the business value of Open Source is pretty well-established. There really hasn’t been a lot of “convincing” necessary. For example, Apress didn’t just agree to release the Django Book under a permissive license: they actively encouraged it. You’d have to ask them if it was “worth it” in terms of sales, but I’m sure that being the first Google hit for “django book” didn’t hurt!
Releasing Django at World Online is actually an interesting story. We decided at the 2005 PyCon that we should Open Source some of our software, and started building a business case for doing so. We prepared a series of arguments — open source will increase our visibility and lead to easier sales and easier hiring, open source will improve the quality of our software, etc. — and took them to our management. All of those things turnned out to be true, by the way, but o our surprise, the argument that was the most effective was actually a “moral” one. We talked about how Open Source had helped our business (Apache, Linux, Python, etc.) and argued that it was time to “give back” to the community. The World Company has always been a company that’s tried to be a conscientious part of our local community here in Lawrence, and they really jumped at the chance to participate in the global Open Source community.
I like to tell this story, by the way, because it really makes me hopeful that this “hacker culture” is in fact compatible with business culture. Fact is that most businesses are in fact concerned with doing the “right thing” — they just often don’t know what that is when it comes to technology. Of course the business case for Open Source needs to be there, too — and it is — but I think there are a lot of companies that’ll jump on the chance to “give back.”

Shabda: Before we close would you like to share any tips with us? Jacob: If you don’t read the Django community aggregator you really should: there are some incredibly smart people blogging about Django and you’ll learn something new from all of them.

This was the interview of Jacob Kaplan-Moss. We have a few more Django interviews coming, before we close this series of Django interviews, so stay tuned.

Russell Keith-Magee is a longtime core contributor to Django. He has worked extensively with the Django testing and serialization components. He is currently working on django-evolution, the ability to do schema evolution from Django, an often requested feature, and is mentoring a GSOC proposal to add aggregate queries support to Django ORM. He is currently a Senior R&D Software Engineer at plugger.com.au. He graciously agreed to an email interview with the 42topics blog.

Shabda: Would you tell a little about yourself? How did you get started with Django? What areas of Django have you contributed to?

Russell: I’m a thirty-one year old, father of one, living in the most isolated capital city in the world - Perth, Western Australia. Strangely, for all my involvement in Django, my background has almost nothing to do with the web. I studied Physics as an undergraduate, and studied neural networks for my PhD. My first job was with a startup in the defence industry developing simulation frameworks. Over time, mostly as a result of my association with Django, I’ve become more involved in building web services.

Shabda: How did you get started with Django? What areas of Django have you contributed to?

Russell: I’ve always had a bunch of little pet projects that I’ve fiddled with in my spare time. Over time, these projects have migrated over a range of languages until I found (and fell in love with) Python. Around the time of Web 1.0, I started looking at various web frameworks (PHP, Zope, etc), but I never really managed to get mental traction - they were either so complex that it was hard to work out where to start, or they provided a bunch of tools but didn’t really make it obvious how the pieces all fitted together. Plus, the documentation for these frameworks was up to the common open source standard - an even mix of the awful, the incomprehensible and the non-existent.
Around mid 2005, the buzz about Ruby on Rails started to grow, so I took a look. I could see that it had promise - it was an interesting new way of looking at solving web problems - but again it suffered from the documentation problem. One day, I found a blog entry referring to Django, and finally all the pieces fell into place. The documentation was excellent, the simple stuff was painfully simple, and there was a clear separation of concerns - there is the URL module which routes requests, an ORM to access the database, etc.
At the time, the pet project I was using to test new frameworks was a time/task tracking system. After working through the tutorials, I tried implementing a task tracker, and quickly realized that Django didn’t have support for aggregate functions. I made a few suggestions, which were universally rejected (in retrospect, for very good reasons), so I went digging in the code to try and come up with some better ideas.
In the process, I ended up spending a lot of time in the query system, which put me in a position to submit patches for a few small bugs in queries. Once I had found my feet, I tackled some larger enhancements. One of my first big contributions was modifying the filter syntax so that queries could span reverse relations; that is, if Book has a foreign key on Author, I modified the syntax to allow Author queries to ask about their related books. Of course, between the Magic Removal changes and Malcolm’s Queryset refactor, most of this code has now been thrown to the wind, but at the time it was a big improvement. Adrian noticed the contributions I was making, and offered to give me commit access and make me a core developer.
Not long after I joined as a core developer, the Magic Removal branch started, which consumed pretty much all my efforts for about six months. This involved more changes to the query system, but also with the way models were defined and instantiated. Following that, I developed the test framework, which also involved working with the serialization framework. I did some work on newforms-admin in the early days, including developing the new Media definition framework and a refactor of form input fields for files. I have also spent some time on side projects that aren’t directly targeting the Django trunk, but might be integrated one day. The most notable of these is Django Evolution - an implementation of schema evolution for Django.
I’ve been a little bit quiet over the last few months; I’ve just changed jobs, which is consuming a lot of my time, and I’ve had a few bouts of illness which have slowed my progress. I have also been helping out James Bennett by acting as Technical Reviewer for his upcoming book “Practical Django Projects“; there won’t be much to show for this until the book is published, but when it comes out, you’re all in for a treat - it’s going to be a great resource. I’m hoping to get back into the swing of things in the next few months. One of the first new tasks will be a very old issue - I’m mentoring a student (Nicolas Lara) in a GSOC project to implement aggregates in the query system. So - after almost 3 years, my original itch may actually get scratched.

Shabda: Have you worked with other competing frameworks, say Ruby On Rails, Turbogears etc?

Russell:
I’ve looked briefly at a lot of other web frameworks. In the early days I looked at Zope and PHP while gaining a foothold in the web world. Since joining the Django project, I occasionally look at what the other frameworks - in particular, Ruby on Rails and TurboGears - are doing. It’s occasionally helpful to see how other people are (or aren’t) solving various problems, borrowing the good ideas where possible.

Shabda: What other software/applications have you worked with. (Both OSS and otherwise)?

Russell: A little bit of everything. I used Linux as a personal computing platform back in the glory days of Slackware and Yggdrasil. I was compiling KDE and GNOME before they were even v1.0, and I’ve toyed around with writing for both frameworks. I’ve used Windows in a commercial environment, and I’ve gotten to know the eccentricities of every version of Microsoft Visual Studio from v6 to 2008 in far too much detail. I’ve written Java, both server side and client side. I’ve spent a lot of time with a simulation protocol called HLA. One of the more notable products of that work was securing some funding and commercial support for an open source implementation of the HLA protocol. I’m a huge fan of Linux, especially on the server side. However, I attribute most of my recent productivity to being a Mac Switcher. I found that once I made the switch, I spent a lot less time messing around with compiling the latest, slightly less buggy version of some esoteric utility, and a lot more time building useful code. It’s possibly a little ‘post hoc ergo propter hoc‘, but my Mac switch correlates almost exactly with being accepted as a Django core developer. Regardless or the causation, give me a MacBook Pro, a copy of TextMate and a good net connection and I’ll be a very happy man for a long time.

Shabda: You have worked with a number of web frameworks. What made you choose Django over them? What are the strong points of Django compared to say ROR? Compared to Turbogears? What can Django learn from these other frameworks?

Russell: I haven’t worked extensively with any other web framework. I’ve had the occasional poke around - mostly to see how the other guys do things. To my eyes, at a technical level the various modern web frameworks have a lot more in common than they have differences. All of this generation of web frameworks are about providing a clear layer between the database and the user, proving a simple mechanism for mapping web requests to functions that serve those requests, and providing a clean way to render content for the responses. Sure, there differences in language of implementation. There are some philosophical differences like whether templates should allow executable code. There are also some feature differences. However, these are all minor in the grand scheme of things. The philosophical differences haven’t stopped anyone from building web applications in any of the frameworks. Given enough time and resources, any feature could be implemented in any framework. Language choice is also largely irrelevant. I’ve used a bit of Ruby, and while it has some interesting features as a language, the difference between Python and Ruby is a lot less than the difference between, say, Python and C++. What this leaves is the stuff outside the technical realm - resources, documentation, and the community. This is the reason I choice Django in the first place, and I still think that they are areas of strength - I certainly don’t regret my original decision. I remember when I first found the Django website - I very quickly got the impression that it was built by a group of people that really cared about what they were building, and how it was perceived. The website and other resources weren’t just rapid constructions or reflections of this week’s design fad - they were designed with care. The same went for the documentation - it wasn’t just a bunch of documents autogenerated from source code comments or a sprawling wiki - they were custom written by someone who knew how to write, and more importantly, knew how to edit. As a result, Django’s tutorials and documentation made getting started an almost painless process - both as a user, and as a developer. The attitude of the community is also important; Adrian and Jacob have established a very civil community and the community as a whole tries to keep it that way.
I think the biggest lessons Django can learn from other frameworks is in how to market itself. Because we’re still pre-v1.0, we haven’t really been aggressive about marketing ourselves. Some of the other frameworks have been selling themselves for a long time, so we are in a position to learn from their experience.

Shabda: One of the most requested features with Django is schema-evolution, which you are working on. What are the design goals for django-evolution? How production ready is this?

Russell: At the moment, Django Evolution handles the simple cases (add/delete/rename simple field) fairly well. There are some bugs in the more complex cases, particularly those involving foreign keys and many-to-many fields. It tends to work quite well for SQLite and Postgres; MySQL tends to be more problematic. At this time, I’d classify it as “production ready for capable developer”. As long as your schema evolution needs aren’t too advanced and you’re willing to do a bit of hand holding and carefully check the SQL that is produced, you can use Django Evolution right now. If you’re not as confident with raw SQL, you might want to wait a while. Obviously, the long term goal is to make it “production ready for everyone”, and support every possible change that you can make to a Django model. Completely aside from fixing bugs and supporting other databases, one of the big open issues is populating data into new columns. At present, we have some simple mechanisms to put in default data, but these are somewhat limited. Longer term, we’re aiming to be able to populate a new column with data from existing columns. This will require providing a subquery as initial data, and my hope is that the new Queryset Refactor features will make this sort of thing much easier.

Shabda: I asked this to Malcolm as well, but he was unwilling to make a prediction. So let me ask this slightly differently. What problems would you like Django to tackle after 1.0? What big features would you be most interested in having in Django, after it hits 1.0?

Russell: My efforts in prognostication have a nasty habit of going badly wrong… but here goes:
One of the recent shifts in in the Django project as a whole has been the move to allow user extension as much as possible. The ability to have user-defined field types, database backends, and management commands are all examples of the push to move Django development into the community as much as possible. I would be very happy to see the majority of Django v2.0 features be a merge of established community-based projects into the trunk, rather than projects tightly integrated with Django internals.
That said, some changes can’t be done as extensions. One such reoccurring feature request is for multi-database support. It isn’t something I have a huge need for myself, but it is a reoccurring request on the Django users mailing list, so it would be good to be able to support it.
Support for aggregation functions (sum, average, etc) in the ORM is a feature I have a use for, and I’d like to see - especially since the lack of aggregation functions is the reason I got involved with Django in the first place. I’m currently mentoring a GSOC project to implement this capability, but I expect that this won’t be complete until post v1.0.
I would also expect that the resources around the community will get some attention. There is plenty of room for new tutorials. There are also a few contrib services that need much better documentation (in some cases, any documentation). Given that v1.0 is a point of API stability, it also makes a good time to add lots of new tutorials and other documentation. Formalization and documentation of some of Django’s more notable internals (like the _meta model class and the various extension mechanisms) would also be nice to have.

Shabda: You have worked in many different areas of Django. What suggestion would you give to somebody who is just looking to start contributing to Django? Are there some areas, where the learning curve is flatter, and people can make meaningful contribution more easily?

Russell: My suggestion would be “don’t be timid - jump in”. Even the most complex parts of Django - like the query system - are relatively straightforward once you’ve spent a bit of time walking through them. If someone is looking to contribute to Django, my suggestion would be to do the same thing I did when I was starting out - pick a Django component with a bunch of bugs, pick one, write a test case for the bug (if there isn’t one already), then walk through the code until you can fix it. Rinse and repeat, but stay in the same component. Eventually, you’ll know that component as well as anyone in the world, and you’ll be able to work on more complex bugs or contribute a feature. Occasionally, tracking down one bug will lead you into another component; use that as an opportunity to learn how another part of Django works.
The other thing to keep in mind is that working on Django means more than just code - if you want your contributions to be taken seriously, you need to get in the habit of writing rigorous tests and comprehensive documentation. Don’t just fix the bug and then complain that we don’t commit your tickets - you need to do the whole job.

Shabda: A lot of Django people come from diverse backgrounds, You have studied Physics, Adrian studied journalism, Malcolm mathematics and James philosophy. Is there something about with Django which attracts people from diverse backgrounds (keep simple things simple …)? Has a lack of people with CS degree been a problem sometime, as in “We wish there was a CS guy to discuss this problem with”?

Russell: My undergraduate degree is in Physics, but my honours and PhD program were both in pure CS. Plus, for all their lack of CS degrees, Adrian, Malcolm and James have certainly proven they know how to cut a line of code, and there are plenty of people in the Django community with pure CS backgrounds that contribute ideas and code. I don’t think Django is suffering from a lack of CS experience.
However, I do think that having a multidisciplinary background does predispose you to a certain way of looking at the world. It has been my experience that some people with a narrow IT/CS focus lose sight of the fact that at the end of the day, a computer is a tool, most people don’t like them very much, and only put up with them to the extent that they help to make their lives better. I can appreciate the elegance of a neat algorithm as much as the next CS major, but at the back of my mind I always have the question “How can I use this to solve a real world problem?”
One of the reasons that I suspect Django has been a success is that it has never lost sight of the fact that it exists to solve problems. Internally, it does a lot of smart things, but outwardly it tries to be as simple as possible. You download the code, and it works - you don’t need to resolve dozens of dependencies. The development server just works. There is a simple and easily understood pipeline from request to rendering. Django identifies that building web applications is a problem that people need to solve, and tries to provide the simplest possible path to let that happen.

Shabda: You have worked with a defense startup, and defense industry is presumed to be very conservative. What can Django do to market itself to market itself to people outside web 2.0 niches, such as in large corporations or defense related software?

Russell: There is a running joke in the Australian military that the Army’s idea of the perfect technology advancement would be a self-sharpening grease pencil. It’s not quite that bad, but the military is very conservative. This is understandable, because lives are potentially on the line if a technology fails. However, it does breed some interesting limitations on new technology: in the circles I was working, IE6 on Windows 2000 was considered to be a very modern web platform.
The way to market to people outside of Web 2.0 niches is to stop trying to sell Web 2.0 as if it was an answer to anything. Web 2.0 is a technology. Web 2.0 may give you a competitive advantage. However, your customer doesn’t give a pair of fetid dingo’s kidneys about your competitive advantage. The way to sell your Web 2.0 idea is to forget all about how cool the technology is, or how great it is that you can do this as a RESTful web service. Instead, focus on how your solution solves a specific problem of your customer in a way that your customer is comfortable with.
A person in industry has a problem. They need a solution. They don’t particularly care if the solution is implemented using an alien slave hiding in a cardboard box connected to the server using electric spaghetti - as long as the solution is cheap, reliable, solves their problem, and doesn’t require them to fundamentally change the way they do business. If you can convince your customer that your product meets these criteria, you’ll have a long and profitable career in the military-industrial complex - regardless of whether you use Web 2.0 or not.

Shabda: What is one thing about Django which you absolutely love, and one thing which you think Django should do differently?

Russell: On a technical level: I think the decision to keep logic out of templates is 100% correct. Keeping logic out of templates forces the separation between content and presentation, and simplifies the interface for the design team, who often will not be gung-ho programmers. I think this is one of the best decisions that Django has made - I know there are many in the world that claim that Django’s templates are it’s biggest weakness, but I simply don’t agree. I take some solace from the fact that Google has tacitly agreed with this position by providing Django’s templating engine inside their App Engine.
One area that I think Django could do better is in error handling - especially in the way that exceptions get caught and rethrown. There are a number of ways that import errors or dangling named URL references can be misreported in stack traces or 500 pages; given that both of these are relatively common errors, it would be nice to provide better handling.
On a community level: I love the general disposition of the Django community. It’s polite and professional, and appreciates the value of good design.
If I had to pick a flaw, it’s probably the way that the core developers (myself included) sometimes let the community down in the way we communicate overall project direction. When plans are made (however informally), they aren’t always communicated effectively to the community. It also isn’t always clear (beyond the simple fact that we’re busy) why specific tickets with bug fixes don’t get committed to trunk in a timely fashion.
Even I get bitten by this sometimes - being based in Perth, Western Australia, the cost of travel prevents me from attending PyCon or the Sprints in person. As a result, I don’t get to participate in the in-person discussions - I sometimes have to read between the lines of blog posts and mailing list messages to work out what the plans are, sometimes confirming details using our secret core developer BatPhone. This is certainly an area where we could do better. Constructive suggestions are always welcome.

Shabda: Frameworks like Turbogears and Zope take a “best of breed” approach, and tie together a lot of existing components to build the full framework. In comparison, Django takes an integrated approach, and most of Django’s components are Django specific. For this reason, many people have accused Django of having a NIH. syndrome. What are the pros and cons of both these approaches? Why did Django rebuild most of its components?

Russell: Saying that Django rebuilt most of its components is one way of looking at it. The other way to look at it is to say that Django identified a core set of functions that need to be tightly integrated, and need to meet some specific design goals, and built tools that met those requirements. In some cases, it’s not even fair to say that components were rebuilt. Remember, Django went public in mid 2005, but existed as an in-house project at the Journal-World for some time before that. For some of these components, Django’s implementation predates the ‘existing component’.
Those that throw the NIH argument at Django are also ignoring two important points. Firstly, Django already uses a number of prebuilt components from elsewhere - the signal dispatcher, the JSON serializer, and any number of other components are taken from existing projects. We also have dependencies on external software - most notably in database backends, but also in serialization and markup engines.
Secondly, there is the presumption that plumbing together external projects requires no effort. This is patently false - you only have to look at the places that Django does rely on external projects to see why. The MySQLdb 1.2.2 prerelease bindings had all sorts of problems that require special handling; similar workarounds are required for unicode handling in recent versions of Markdown. Each version of these external components can potentially require different handling, sometimes in subtly incompatible ways. Once you introduce multiple interacting components, you introduce the problem of incompatibility between versions of those tools. Integration like this isn’t easy. The Django approach guarantees that everything works out of the box, and there is only one core configuration to document and test. It reduces the barrier to entry: making the first act of a new user a decision about which slightly different template engine will best meet their needs isn’t the best way to make a good impression. It also focuses the efforts of the community in testing, documentation, tutorials and advice. If improvements are required, the entire community is in a position to provide an opinion, not just the subset of the community that works with that component option. In a Balkanized world, there will always be a tendency for the question “How do I do X using tool Y” to be answered “don’t use tool Y, use Q instead”.
Building the components from scratch also guarantees that all the components fit - and I don’t just mean that they satisfy their mutual interfaces. There is also the issue that the core components should all ‘taste’ the same - that they follow similar design philosophies, apply similar assumptions, and so on. This sort of consistency is an invaluable resource, as it forms a kind of implicit documentation: once you have learned one part of the system, you automatically know how other parts of the system work. You don’t need to relearn the rules for each component.
One notable (and frequently made) argument against this approach is that you lose flexibility. While this is theoretically true, I don’t think it matters much in practice. For all the supposed inadequacies of the Django template engine, I challenge anyone to present a problem that can only be solved by using a different templating engine. On top of that - there’s absolutely nothing preventing you from using your own template engine, or your own database backend, or any other component if you don’t like Django’s offerings. If you don’t believe me, go look at Google App Engine.

Shabda: What are the focus areas Django testing framework? What utilities does this add too unittest or doctests? How does this fit in with browser based tools like Selenium?

Russell: Like most things in Django, the test framework aims to be as simple as possible, but no simpler. It provides an environment in which tests can be run, it provides the facilities for finding and invoking test cases, and provides a few helpful web-specific assertions that can be used during testing. Base Python unittests and doctests get a dummy database to work with, plus a dummy mail server; the Django TestCase extends unittest.TestCase to provide fixture loading and a dummy web client that can be used to programatically poke views. So far, the Django testing tools have concentrated on testing Django views as if they were ordinary Python methods - after all, that’s what they are. Selenium takes things another step, allowing for tests of how a web browser will behave when given the output of a view. I tend to think that testing the view itself is a more rigorous test, as it allows for direct inspection of view output. However, Selenium does have a place, especially when dealing with Javascript or other dynamic on-page interactions. Adding support for Selenium to Django is on the to-do list - it just requires someone to do the work.

Shabda: Thanks for the interview Russell.

This was the interview of Russell Keith-Magee, a longtime Django core contributor based in Perth, Australia. Next we interview Michael Trier, a longtime Django user and evangelist. Michael’s This Week in Django has been a great resource for the Django community, and we discuss Django, and how best to market Django. So stay tuned.

Oh and yes, 42topics.com is live now! (How are we different?) And we have a Django section. So regsiter let’s get rolling!

James Bennett is the release manager of Django, and a long time contributor. He works on Ellington, a CMS designed for news organizations. His book, Practical Django Projects, is being published by Apress, and is scheduled to hit bookshelves in June 2008. He graciously agreed to be interviewed at the 42topics.com blog. His blog, The B-List, can be found here.

Shabda: Would you tell something about yourself, how did you get started with Django, and what other OSS projects are you involved with?

James: I got into Django fairly soon after the initial public release; I’d been doing PHP and Perl work (mostly Textpattern on the PHP side, Scoop on the Perl side), and I was working on teaching myself Ruby and Rails because it looked interesting. But I’d always liked Python; it was just that there weren’t a whole lot of good options for Python web development at that point. You could do Zope, or you could do Twisted, but they both had pretty steep learning curves when compared to the type of work I was doing, so it just wasn’t worth it.
Django changed that; I think I did the tutorial the day it was released, and I just fell in love.
These days most of my open-source time is devoted to Django, and to various Django-based apps I’ve written and released. Back in the day I used to do the occasional bit of PHP hacking, and I wrote some plugins for blogging engines, but none of that stuff’s maintained anymore.

Shabda: What are your responsibilities as the release manager of Django? Who are the other core contributors of Django, and what are their current areas of focus?

James: My responsibility is largely bureaucracy. Within the project, I keep an eye on the various branches and their progress, try to stay on top of active/problem areas in the ticket tracker, and maintain a list of things that have to happen before 1.0.
Outside the project itself, I also stay in touch with people who are distributing Django — Linux distros which package it, etc. — and watch their Django-related bug reports. When there’s a security fix, I also send out an advance notice to them that we’re going to roll a release, and do the initial disclosure so they have some lead time to respond to that.
Outside of that, I also contribute to the documentation, as well as the occasional code patch, and I maintain the 0.91-bugfixes branch, which provides legacy support and security updates for some of the really ancient Django installs out there (our policy is to support the current release and two prior releases with security updates, which means 0.91, 0.95 and 0.96 right now).
The rest of the core team is just people everybody knows if they watch the developers’ list or the Trac timeline: Adrian and Jacob are the lead guys, obviously. Then there’s Malcolm, who you interviewed the other day, and who’s carved out a pretty big niche for himself: he did the bulk of the Unicode work and he’s doing queryset-refactor, he does a lot of maintenance on the i18n system, and he somehow still finds time to have a day job. I don’t know how he does that. Then there’s Russell Keith-Magee, (russelm in Trac), who’s contributed all sorts of useful stuff. He’s the go-to guy for testing and serialization, and just an all-around brilliant guy who fixes things left and right.
Gary Wilson and Joseph Kocherhans are also both big names you’ll recognize from commit messages; Joseph used to work at the next desk over at World Online, but now he’s up in Chicago with Adrian, working on EveryBlock. He’s done a lot of the grunt work on newforms and on laying groundwork for newforms-admin.
Ian Kelly and Matt Boersma keep our Oracle support working, and do an amazing job of helping to track down and solve obscure problems with that. And rounding out the core commit bits are Simon Willison and Luke Plant, who aren’t as active these days but can still be seen popping up every so often.
Out on the branches, we’ve had a lot of work done on newforms-admin lately by Brian Rosner, who’s been stepping up and helping to really get that sorted out, and there’s been a lot of unsung UI work by Christian Metts, who’s a designer at World Online but also a not-bad JavaScript programmer and can flaunt some Python when he feels like it. And then there’s the “gis” branch, GeoDjango, which is adding support for GIS — spatial querying — to the Django ORM.
Justin Bronn and Jeremy Dunck really kick-started that, Justin gave a nice demo of some of their work at PyCon this year.
And recently we’ve also been giving commit access to some of the translators, so they don’t have to go through as much bureaucracy to submit updated translation files; that’s sped some things up on the i18n front, because we’re blessed with a large number of people who are willing to pitch in and do that work.
Oh, and I can’t forget Wilson Miner; he doesn’t flaunt it that often, but he’s the guy who originally designed the Django admin, and he’s been known to make the occasional tweak to fix CSS stuff.
(hope I didn’t forget anybody there)

Shabda: Would GeoDjango be sometime merged with the trunk, or is it forever going to be a parallel branch to trunk?

James: The goal is that the GIS branch will merge, sometime after queryset-refactor lands, and it’s going to provide an application — django.contrib.gis — that you can use to enable spatial queries for your models. There’s a pretty good writeup on the wiki page of how that works, and they’ve been tracking queryset-refactor because it helps make some of the custom query construction easier.
I don’t know for certain if it’ll hit trunk before Django 1.0, but it will not be a branch forever; they’ve put in a ton of work on that, and I’m looking forward to getting to use it

Shabda: Once Django hits 1.0, would 0.91 be end of lifed, but support for .96 and .95 continue?

James: Like I said, the policy is we provide security fixes for the current release plus the previous two, so 0.91 would sort of fall off there after Django 1.0. But there are a lot of legacy installs out there that are perfectly happy for now, and I wouldn’t be surprised to see people unofficially continuing to submit patches and keep that maintained for as long as there are people who want to use 0.91.

Shabda: A little about you. You have majored in philosophy. I can think of one other, Paul Graham. Would you say, what you learnt from philosophy help with programming?

James: Well, I wouldn’t say there’s anything specific necessarily. But I think there’s a big place for people with liberal-arts backgrounds to come to programming, and I think philosophy’s a good path to do that. If you look at a typical philosophy program, you’re doing a lot of logic, a lot of critical analysis, a lot of abstract reasoning.
You have to get comfortable sooner or later with all sorts of formalisms that don’t necessarily have any practical meaning, and that’s very similar in a lot of ways to programming And when you get right down to it, as programmers, about 90% of what we’re paid to do is think: our job is to take a problem, analyze it, break it down into pieces and solve them.
And that’s not terribly different from what you spend four years doing in a philosophy program. I’ve actually joked about that a bit with some of my former professors, that I still get to argue as much as when I was doing philosophy, but the programming pays a lot better.
I do think, though, that there’s a big need for that sort of thing; we don’t really teach critical thinking anymore, and while it’s a vital skill to have no matter what you do for a living, it’s absolutely crucial to programming. So if you can get a good liberal-arts background where you’ve been taught how to look at things and pick them apart and analyze them, you can definitely do well as a programmer. Though it’d also be a good idea to take at least a few elective math courses…

Shabda: You have been working on Ellington, how is Ellington CMS different from, say, a customized version of Drupal?

James: Well, they have some things in common: both are modular CMS-style products, both are meant to be extensible.
But Ellington is really targeted from the ground up at news operations. There’s all sorts of specialized stuff in there that’s really optimized for the way a newsroom works, most of it culled from our experience as a newspaper, and of working with other papers.
So where with Drupal you’d really have to do a lot of customization because nobody’s really done this kind of “news all the way through” version of Drupal, with Ellington it’s there out of the box, and you hit the ground running.
I think we also have a very unique position because we are a news company, we’ve got a bunch of newspapers, some periodicals, TV news, etc.
And we work with those folks every day, we see how they do stuff, we hear about it when they run into problems, and so we’re in a good spot to see just what a newsroom staff really needs out of their online platform.

Shabda: Wordpress is PHP’s killer app, arguably Basecamp is ROR’s. What would you say Django’s killer app is?

James: Honestly I don’t know right now; I think the thing that Django wins on is that there doesn’t have to be the One Big App that everybody uses, instead there’s this huge blooming ecosystem of applications. In a way, it’s like asking what the “killer library” of the Python stdlib is; the killer feature is that you’ve got all that stuff available.
Though there are definitely some cool Django apps out there right now, and a lot more on the way. I think there’s a different mentality, though, because in general Python people seem to keep their heads down and just get stuff done. So it may be you don’t hear about some project until maybe they decide to do an open-space talk at PyCon or OSCON, and then you just get blown away.
There was a guy at PyCon who came up and showed me an app he’s been developing, and I won’t spoil it and give away what it is, but it made my jaw drop.
He’d taken something that’s a really common software niche that’s been dominated by these abominable products because it’s not really a sexy thing to be doing, and just absolutely nailed it. Guy’s probably gonna make millions. But I don’t know if there is a general-purpose “killer app” for Django right now, and I’m not really sure I want there to be one; people can see that sort of thing and think “oh, that’s all it does”. They look at, say, Ellington, and think “oh, this is only good for a newspaper-style CMS“, or they see Rails and Basecamp and think “oh, this is no good for me, I’m not doing a Web 2.0 thing”. So in a way I’m kind of glad we don’t have a “killer app” hanging over us and pigeonholing Django.
Though I should definitely give a shout out to Review Board; of the public Django apps I’ve seen, it’s probably the coolest, and again takes something that’s not usually sexy in terms of software and really nails it.

Shabda: As you said, “Python people seem to keep their heads down and just get stuff done”, do you think Django needs to a better job marketing itself. For example, I have pitched Django to a fair number of people, and I always have to start with with “Django what?”, as compared to say ROR, or PHP which seems to have a good brand recall.

James: Well, to some extent Django hasn’t done a whole lot of explicit marketing because we’re not at 1.0 yet, and I expect that after 1.0 it’ll both be a lot easier to do marketing and that there will be more of it going on. But at the same time, Django’s doing pretty well as it is; Google’s doing their App Engine stuff with Django bundled, there are startups using it to build the next big thing, and it’s even been quietly sneaking its way into some huge corporations. Plus it seems like every time I turn around there’s another book coming out.
I saw one yesterday at the bookstore downtown; I hadn’t heard about it until that moment, but I think that makes five or six books that’ll be out by the end of this year. So I think that’ll help. Digital Web magazine did a feature article on Django just recently, and I did an article for Sitepoint a while back as well. I like to view this as the phase where we build up momentum until eventually Django is an unstoppable juggernaut and everybody’s listening to gypsy jazz music ;).

Shabda: Everyone is pretty excited about your coming book. When can we have it in book stores? Would you give a brief overviews of what’s in it?

James: The book will, I think (and hope) be shipping around the end of June. It’s very much a hands-on introduction to Django: walking through building three applications, picking up progressively more advanced bits of Django as you go, and seeing some best practices in action. So you start out with just simple stuff, using the contrib apps and learning the basics of getting Django running. Then you do some simple customizations of admin templates, then start building some models and views, then on into building full-on applications.
There’s not room to cover every single thing you can do with Django, but I think it handles a pretty good spectrum of techniques from basics up to some advanced things that let you poke around and really get a feel for how stuff works.
And of course, you get periodic bouts of me up on my soapbox yelling at people about how to write reusable applications, because that’s what I do.

Shabda: You talk a little about App Engine in Batteries sold separately. What effect do you see of App Engine on python hosting ecosystem, on Python web applications, and on Django?

James: I honestly don’t know what to expect from App Engine. It’s a very different kind of thing from what anybody outside Google is used to, and it’ll probably be a while before it’s really shaken out and we get an idea of the impact it’ll have. I fully expect that Google’s going to start supporting other languages, so there won’t be this effect where you’ll always have to use Python to use App Engine. And the way they’ve sandboxed it is going to make it feel weird to Python people, I think. And that’s on top of getting used to the fact that you’re not using an RDBMS; I’ve been watching the blog flamewars about that, and that seems to be the big takeaway.
If I had to make a prediction now, I’d say that App Engine will bring some people to Python, but probably not in hordes, and that its big long-term effect is going to be to point out to a lot of folks that they’re not really using the “R” in “RDBMS“, and so maybe it’s OK to think about their applications in a different way. And that’s not Python-specific at all.

Shabda: What were the focus areas of the newforms-admin branch. How far have they been achieved. About when would the newforms-admin branch be merged? How backwards incompatible is this going to be?

James: Well, there are a couple goals running parallel. The first thing, obviously, is that with the oldforms package deprecated we’ve got to get stuff migrated to newforms, and the admin’s got to make that jump.
And because the admin does a lot of tricky and complex stuff with forms, it’s also been a fertile proving ground for designing some advanced newforms features that’ll find their way back into trunk and make it easier to do that tricky stuff on your own.
Another big thing is cleaning out the coupling issue where you declare a class inside a model to activate the admin interface; there’s no reason for that to happen, so instead there’s a class you’ll subclass and override stuff on to customize the behavior, and then you’ll register a model to have the admin interface you’ve set up that way.
Finally, there’s been a lot of stuff abstracted in newforms-admin; while I don’t think it’s really an official design goal, there’s been a lot of opportunity to provide hooks where people can go in and customize stuff. I looked at the code a couple months ago, and it was about 95% of the way to being able to run completely without django.contrib.auth, for example, because there are enough hooks where you can override something and replace defaults to make it use something else.
And that’s a huge deal, because it means you could do stuff like run the admin on HTTP auth, or run it in a corporate environment against an LDAP database, and you barely have to touch anything; you can already do an auth backend that handles most of the work, then you tweak a few things for the admin and you’re good to go with no hacking directly on Django code.
Plus, a lot of people will be happy to see that the methods you can go in and override all get the HttpRequest object as an argument, so even though it’s bad workflow a lot of the time you’ll be able to have the admin do stuff like automatically fill in a foreign key with the current user. And along the way there’ve been huge numbers of bug fixes; there’s a lot of stuff that’s always just been really fragile in the admin, like edit_inline, and newforms-admin is a good opportunity to just yank out all the things that made them troublesome and do it right.

Shabda: Time for a meme. What is the one think you absolutely love about Django, and one thing which Django should have done differently?

James: I love contrib.localflavor, and I want people to go look at it and adore it and use it. There’s so much useful stuff in there, you can validate checksums on Scandinavian social-security numbers, you can get lists of all the state in Brazil, it’s just this great gigantic resource for localizing your apps, and it kicks so much ass.
Something to do differently… I’d probably rewrite the tutorial to not do the apps-inside-projects thing. That trips a lot of people up because they come away thinking they have to do it that way, and so they really miss out on the benefits of being able to write an app once and just use it over and over and over.

Shabda: Before we leave, would you like to give a quick tip, or hard to find information about Django to our readers?

James: The best tip I can give is not to be afraid to look at code. We do our best to document all the useful stuff, but that’s a pretty huge area to cover, so there are always going to be neat little things hidden away that you either won’t see unless you read the code, or won’t see until some date in the far future when somebody gets time to document it.
Plus, if you find something cool you can write up a little tutorial and post it, and then people will start reading your blog. That’s pretty much what happened to me, where my blog was for a long time just me writing up these little articles to remind myself of stuff I’d learned.
And especially if you’re learning Python as you go, reading a significant piece of code like Django can really help to improve your understanding of the language. There’s really nothing better than that for assimilating the way a language works, and I think Django’s around the right size — it’s small enough you can carry a solid understanding of it around in your head, but it’s big enough that there’s lots of stuff, and lots of different kinds of stuff, that you can look at and learn from.

Shabda: Thanks a ton for this interview, and for sharing these useful information.

James: Sure.

This was James Bennett’s interview. I plan to be interviewing a few more leaders from Django community, so if you would like me to ask any question, write them in the comments.

Malcolm Tredinnick is a core contributor to Django, and was the driving force behind the Queryset-refactor branch of Django, which adds important capabilities such as model inheritance. He has a long association with OSS, and contributed significantly to GNOME and Django. He graciously agreed to be interviewed at 42topics blog. Malcolm’s blog, Defying Classification, can be read here.

Shabda: Would you tell a few things about yourself, how did you get involved with OSS and Gnome, and with Django?

Malcolm: Here are some recollections I’ve written about: 1 and 2
Short version; started using Linux in university as a poor undergrad; kept using it since then (started back in 1993).
Started using GNOME in 1999 (after trying a very early version of Qt and what became KDE), started contributing to GNOME about 12 months later (mid-2000).
Started using Django in October 2005, I guess (a few months after it was open sourced) and haven’t stopped. Started contributing patches more or less immediately and was given commit privileges about March of 2006, from memory.

Shabda: So what is one thing in Django which you absolutely love, and one thing which you think Django should have done differently?

Malcolm: I guess Django’s separation of responsibilities is something that has always been attractive to me. It’s very easy to keep the cross-application business logic and the persistent storage logic and the state control and the visual presentation separate.
I guess the piece that probably routinely annoys me the most is a slight inconsistency in the template tags: we often mix up how to use a literal string in template tags (sometimes it’s in quotes, sometimes not), which makes it very hard to later make that tag also take a template variable.
As somebody who handles bug reports a lot, something like that ends up taking up a lot of time in my life, which could probably be better spent elsewhere. Still, it’s almost impossible to change now, so that’s the way life goes.

Shabda: Queryset-refactor is almost done, what would were the major overarching goals for this branch?

Malcolm: A number of things…
1. Clean things up internally, so that future extensions are easier. A lot of the existing (trunk) query construction grew from something much smaller. It all mostly worked, but it was getting pretty hard to manage. Some bugs were almost impossible to fix in the trunk form too, which was the real motivation.
2. Organise things so that I could add model inheritance, which was the main thing that started me looking deeply at all that. This has been accomplished.
3. Make it easier to add backends like Oracle and MS SQL Server that don’t support, say, limit and offset in SQL. That meant providing a more object-based query construction so that they can tweak things before creating the SQL string.
In the process, we’ve made it a lot easier for people to extend this for backends and to add functionality to existing backends.
For example, adding new lookup types is now possible (not particularly easy, but possible). And the geo-django branch can do their stuff without having to patch the query construction code any longer.
(Justin Bronn, the geo-django maintainer has been tracking qs-rf very closely, so we know that geo-django works nicely with qs-rf).
I guess they’re the main points. The first one — a better/different internal organisation — was the main one and was really the bulk of the work. Getting everything to mostly work was relatively easy. Getting it all to work perfectly took a lot longer.

Shabda: So would this make, say, writing a backend for non relational DBs like Bigtable easier. Michael Trier was working on a Sqlachemy based ORM for Django, how would this be affected by qs-rf merge?

Malcolm: Well, that’s really two questions. So, one at a time…
Whether it would make a non-relation backend easier,… yes. The QuerySet class passes all the work of talking to the backend, whatever it is, off to a class called Query. Writing a different sort of backend would require writing a new Query class that knew how to take the function calls QuerySet makes and turn them into the right return types.
Mostly, QuerySet does not poke at the internals of Query: it uses “public” methods only. This is deliberate. It means that writing something like a Query class for a different sort of backend (RDF store, or LDAP or whatever) might very well be possible.
As for something like SQLAlchemy, I gather it’s made things easier. I know Michael and Brian have been writing their code against the qs-rf code and it seems to be working.
However, I haven’t looked at their code, so I’m not sure if they’re using the QuerySet/Query split very much or not. I gather they’re doing it slightly differently from how I might have done based on somethings they mentioned when I was talking to them yesterday — apparently a recent qs-rf commit “broke” their approach to something and that was a little surprising to me. But it wasn’t a showstopper apparently, so I’m not too worried.

Shabda: As you said, a commit sort of “broke” their approach. How backwards comaptible is qs-rf. Is it mostly backwards compatible, but some corner cases will break, or would any thing which worked previously, but does not now should be reported as a bug?

Malcolm: It’s almost 100% backwards compatible. The thing I “broke” (and I’m using quotes, because it wasn’t really a breakage, just a change) was a very internal thing. Obviously if you’re writing something like a new storage backend, you need to dive into the internal API of the ORM a bit and that’s what the django-sqlalchemy stuff is doing…
For normal user code, there should be very few changes required. There’s a list on the wiki page for the branch but they are mostly things that won’t affect normal code.
Only if you are doing something very “tricky” or slightly unsupported will changes be required. Of course, some people will see slightly different results in complex querysets because we have fixed a lot of bugs (over 60 that were reported to Trac plus some that never got that far) and some of those fixes change the results that were returned.
So some people may be relying on currently incorrect behavior, but I doubt that’s going to be too troublesome. Also, I’ve tried to make the error reporting a lot clearer so that when somebody does try to do something undefined, it should hopefully give an error message (e.g. introducing an infinite ordering loop in models)

Shabda: Speaking of Bigtable, Google’s App Engine seemed to be sort of a letdown from the initial hype. What are your opinions about the effect of App Engine on Django?

Malcolm: Well, I think the letdown came from people letting their expectations get ahead of their brains a bit. Google didn’t really seem to over-hype it or misrepresent it. Obviously when Google releases something, everybody gets excited. But you have to at least take the time to look at what was released.
For a particular class of applications, I think App Engine is probably ideal. It gives cheap hosting, great reliability and access to some Google’s storage stuff.
For example, if you have something that presents a read-only view onto a large bunch of data, App Engine would be very appropriate. Okay, you have to do data extraction from multiple models manually, since there’s no joining, but that’s not too hard to work around and people will write little helper functions for those cases. Remember that a lot of very successful, very popular websites are basically read-only: a lot of newspaper sites, things like EveryBlock (and, formerly, Chicago Crime) — all those sorts of sites. Even news.google.com.
There’s a great deal of information that wants to be “presented” to people. Not everything requires form entry and comments and the like and even those are possible on App Engine if you need them.
So I’m kind of glad the initial reaction phase has settled down a bit and people are starting to think about what is possible, rather than focusing on what isn’t possible. Some interesting things will come out of App Engine, I suspect. We just don’t know what they are yet because people are still writing them. As far as how it affects Django, that’s probably both good and bad.
If people view App Engine as being representative of everything Django, it’s not going to be too handy. However, I doubt that will be the prevailing opinion. People realise that Google have used portions of Django to implement portions of the App Engine experience. And those pieces of Django are quite handy. The URL dispatching, the general data flow, the templates — all will be nice for people to use.
It’s also getting a few people interested in the internals of Django as they try to work out whether they can move the extra bits that are ‘missing’ from App Engine into the fold.
In addition, we’ve also already received some direct positive benefits: some of the Google team, particularly Guido van Rossum, have been filing a few bugs against Django as they did the initial App Engine development. So there are a few bug fixes in things like newforms and the templating component that are a direct result of App Engine development inside Google.

Shabda: There is app-engine-django, which is trying to bridge the impedance mismatch between Django ORM, and App Engine ORM. Is there any effort to bridge this from Django side by making Django ORM play nice with Appengine. If such a request comes in what would Django’s response be?

Malcolm: shrug. I have no idea.
I think this is an ideal opportunity for some people to write stuff like django-appengine and work out what might be needed and come up with a concrete list of things that might be needed. Predicting the future is hard enough to be mostly futile and it’s not really worth speculating.
Right now, the core Django developers are fully focused on getting Django 1.0 released. That’s where our time is going. How other people spend their time is up to them and it’s definitely not for me to judge whether it’s worthwhile or not.
People need to experiment. It’s the way new software products are developed. So anybody who’s interested and working on this should be encouraged. That’s how open source development has always worked and I think it works quite well.
I guess I could hazard a guess that any intrusive changes just to support some App Engine style thing probably aren’t interesting prior to Django 1.0, since they’re not necessary to get that out of the door. But that’s as much of a prediction as I’m willing to make and anybody who actually attaches any value to that prediction is probably being foolish. I’m wrong more often than I’m right.

Shabda: Well this gets asked a lot on the mailing lists, qs-rf is almost done, Newforms admin should be done soon, so when can we expect Django 1.0, and what new goodies will this bring?

Malcolm: heh
Well, the real answer is “as soon as possible”.
Notice that this is a big step up from “when it’s ready”, which is the normal answer here. Hopefully people appreciate that the maintainers want to get 1.0 out the door as much as anybody else. It’s just that release 1.0 when it’s not at a point we’re happy with won’t be useful in the future.
More concretely, we have a few major things to do between now and 1.0 and they’re all getting pretty close. Then there’s a period of stabilizing and random bug squashing and triage to make sure we haven’t missed anything big.
The idea (and driving goal) is that whatever we release as 1.0 should remain backwards-compatible on an API level until, say, 2.0. So we need to get all the big infrastructure changes in before 1.0.
That isn’t the same as saying 1.0 will be bug free or any pipe dream like that, because it won’t be. But fixing bugs is an ongoing job and as long as we can feel comfortable that we can fix existing bugs without having to make major breakages, we can feel happy about 1.0.

Shabda: This has happened to me a few times, I am trying to pitch Django to a client, and then I mention Django has not hit 1.0 and we would be developing their site against a subversion checkout, and boom, I am against a brick wall. I am guessing this happens to a lot of people. Any suggestions on overcoming client’s reluctance on this point.

Malcolm: Well, this is the hard problem, of course. There are at least three issues here.
Firstly, it’s well known that in a lot of corporate situations, they only want to go with “released” versions, whatever that means. The fact that released versions are often a disaster (e.g. Microsoft Vista) seems to be forgotten, but that’s the way things go. However, it shouldn’t be the ultimate driving force in this equation. It’s one consideration only and having the software released faster just to meet some immovable, often-times unsupportable policy isn’t the answer.
The second aspect is to realise that there are a lot of successful sites built on Django 0.96 and even Django 0.91. So it’s not impossible to stick with the last release.
Now, of course ,that tends to show up the problem with the first point, since 0.96 has some bugs that have subsequently been fixed, so sticking to a release handicaps you there.
However, if we did more frequent releases, that adds a lot of burden to the release process. For example, distributions (Debian, Suse, Fedora, etc), now have a bigger problem in choosing the right version to ship since they also want some stability. Security releases either become harder (we have to patch more versions) or else we have to “end of life” earlier releases more frequently, which harms those people using those releases (again, it reduces stability for the userbase).
The third point is somewhat related to that: in order to make sure that a release can be relied upon, we sometimes need to make sure that we do all the necessary preliminary work so that we don’t immediately break any code that relies on this new release.
That is exactly the situation we’re in now. We release 0.96 as a sort of “stability point” for people to rely on. Almost immediately, we then started to introduce a bunch of necessary changes to things that will require code to be ported when people upgrade. When this round of churning has finished, we’ll make the next release. That happens to be 1.0 in this case.
If we released, say 0.97 at some point in the past few months, people would have a similar but slightly different sort of marketing problem: now they have 0.96 code that has to be ported if they want to move to 0.97. But it’s also just “more work” on some level, since they’ll have to do even more porting when 1.0 comes out. So we aren’t helping developers by providing them with a lot of stability.
It’s fairly well understood by most experienced developers that this is a tough path to walk. With open source software it’s even harder, since everybody is a volunteer and we have a largely unknown userbase to satisfy. Sometimes that difference is forgotten by people who argue that “no company ever works like this”. A corporation can set release deadlines and in exchange for everybody having to work to those deadlines, those people get paid. It’s called a salary. Open Source software doesn’t work like that.
I’m sure that once Django reaches 1.0 or perhaps a little later, we’ll slip into time-based releases so that things go a bit more smoothly. At the moment that simply isn’t practical because of all the changes we need to make. Every project goes through that phase. Before Linux had time-based releases, they had to get an amount of features in and a certain stability level reached. Ditto for GNOME (which only started doing time-based releases at 2.0 and there was a long gap between 1.4 and 2.0 precisely because we needed to be able to guarantee that at 2.0). Ditto for KDE and Ubuntu and …
So, yeah ,it’s slightly tough times for everybody at the moment. Partly Django is a victim of its own success here: everybody wants to use it and we’re trying to keep up.
Hopefully people realise that ultimately that’s a good thing. It’s an investment in the future in the sense that we’ll still be here in a year, two years, five years… because of this great use.

Shabda: What are the future plans for Django. Django has most of the things which I want in a framework, (import soul), and we would hate to have Django suffer from featuritis. After 1.0 what are the areas Django wants to tackle?

Malcolm: Again, this is asking for a predication and I don’t do those.
Partly because I don’t know and am not really in a position to say, in any case. (I’m just a contributor). Partly because having too many goals too far ahead of time is possibly going to restrict people. Django runs on its contributors. A lot of the ideas that are implemented are initially suggested and often developed in quite some detail by people other than the core developers. It might not always seem that way, because you sometimes have to see the common thread in a bunch of requests before you notice the lurking feature requirement ,but it’s true.
So what happens post 1.0 is going to depend a lot on how people use Django, on what people offer in the way of code and, particularly, what good ideas are suggested.
Who could have predicted Google AppEngine when Django 0.96 was released? What other things like that will appear as a result of Django 1.0? Who knows?
Of course, there are some things that we could probably say will be worked on (successfully or not is another question entirely). Multi-database support seems to be popular and one day somebody might do it; maybe there’ll be more front-end developments — e.g. somebody like you who keeps wanting more Ajax support will actually develop something that provides this support everybody seems to want but hasn’t ever specified.
We can sort of see a few things coming up again and again, so I guess they’ll be worked on. But it’s really going to be up to the people who write the code, who come up with the ideas, who write the websites that use Django, who try to teach Django to others. And right now, I have no idea what direction that will go in.

Shabda: Before we close, would you like to share a handy tip which you use a lot, but does not get used so much otherwise?

Malcolm: People possibly overlook both the {% debug %} template tag (best to use it as <pre>{% debug %}</pre> in a template) and the debug context processor django.core.context_processors.media.
Both of those are very useful for trying to see what’s going on when you’re passing information between a view function and a template.

Shabda: Thanks Malcolm. It was great talking to you.

I plan to interview Leaders from Django community, in coming few weeks, so if you would like me to ask any specific questions, put them in comments, and I will ask them when I interview.