Facebook: The New Google?

For many years now, I've thought of Google as top dog in the software world. But Facebook is challenging that. Yes, the frequent UI changes are odd, but the back-end tech is innovative and impressive. Talk about scaling: hundreds of millions of users. Daily number of writes is probably at least that. Daily number of reads is probably a couple of orders of magnitude higher. Hell, just getting all those email alerts out is impressive.

They're pragmatic: they use everything from PHP to C++ to Erlang (the billion-msg-a-day chat server), and they open-source a lot of stuff.

Freedom to choose the right tool instead of the popular one: priceless.

What's In A Server Name?

In the beginning, the admins named the servers. So the servers got geek-culture names like gandalf and sauron, or enterprise and voyager. And it worked pretty well. People remembered the names, and they remembered that gandalf was the development box, and sauron was the production Oracle database.

Then some suit decided that the geek-culture names were unprofessional (whatever that means), so IT management stepped in. They came up with server naming conventions that embedded all sorts of useful information: data center, run-time environment, software configuration, server number, etc. And out came monstrous server names like NYCP3STDZ09. Which is a challenge to say out loud, and hard to remember, and doesn't connote much of anything unless you know the original conventions. Which hardly anyone does after a few years of turnover.

And NYCP3STDZ09 is a hard name to get right in a phone call, and an easy name to mistype in an email, and that's just so much extra fun in the middle of a production problem.

Meanwhile, sauron comes across clearly in phone calls and emails, even when you're frazzled because your boss's boss's boss wants to know when the fuck the system will be back up, even when your data center is offshore and your admin's accent is very different from yours, and you're on a conference call that sounds like a cell phone in a hurricane. It's a great server name when people need to communicate.

But it's unprofessional, and disorderly, and doesn't tell us anything about the server's role in the organization. So we have a winner: NYCP3STDZ09. Just be careful what you type. Don't want to get the wrong box rebooted again!

Interviews 2.0

I've been doing some J2EE screening interviews for my company, over the phone. Some trends:

• Candidates google as they go, so feature questions like What's the difference between a HashMap and a HashTable? have almost no value. All you're testing is how quickly and quietly the candidate can type.


• Candidates have found a quasi-ethical way to embellish their resumes: describe a project in detail, but don't say which pieces you worked on. List every tool/technology used in the project, whether or not you were the one using it.
  
  Of course, this is pretty easy to blow apart. I've gotten some amusing explanations of Ajax from people who maybe used it once to clean a sink. Which brings us to...


• Don't just ask people to explain a technology. Ask them how they've used it on a project. Any candidate who answers with an accurate general description of the technology probably hasn't used it much. A candidate who can tell you what he's done with the technology, the problems he ran into, how he worked around them - he's telling you the truth. Bonus points if he gets animated while he's talking about it.
  

Experience is hard to fake, even over the phone with a browser open.

It's The Framework, Stupid

I'm looking at Scala. It uses static typing, albeit in a sophisticated, type-inferred-when-we-can way. But still: plenty of syntax, and rules, rules, rules. The last static language I learned was Java, about ten years ago. Since then, I've been much more interested in dynamic languages like Python and Ruby.

So why Scala? Because of Lift. I think we're at a point where framework trumps language, and I want to give Lift a try. Hence, Scala.

So why Lift? Because the value of a framework is in the code you don't write, and the worries you don't have. I've been learning/thinking about web security lately, and it's not trivial. Passwords, for example: pretty much everyone knows that passwords shouldn't be stored as plain text. But do you understand why adding a salt to the hash matters? Why you should use a different salt for each password? Why storing the salts as plain text right next to the passwords is not a risk? Which hash you should use? (If you want to dive into all this, here's a good starting point. Read the comments, too.)

There's a lot to it, and that's just passwords. There are plenty more ways to screw up web security, and I'd just as soon hand all that off to the framework. So, when I read this blog post by David Pollak, the creator of Lift, the stuff under the Lift offers unparalleled security heading got me all excited. It sounds like Lift has security baked in.

Contrast that with Rails, where there's still no canonical authentication/authorization solution. Plugins? Sure. But this functionality is critical. It belongs in core. That makes the code better, because you've got more people using it, looking at it, poking at it. And everyone gets the fixes automatically when they upgrade. Then there's the unescaped HTML thing. Rails still leaves it up to you, the developer. Over-all, I'm left feeling that security just isn't high on the list for the Rails core team. But leaving it to us application developers is dangerous.

Lift looks like a potential big win. It'll take me a little extra time to get the basics of Scala into my head, but I expect to make that time up pretty quickly. Within a project or two, if Lift works out for me. After that, it's all upside.

Cleaner and Cleaner

Re: yesterday's post, I read a little more of the excellent and free Django book this morning, and I realized that template includes don't have to be hard-coded. They can use variables.

Woohoo! This lets me move all my special-case logic into the Python code. Instead of nested {% if %} statements in the template, I'm doing a simple list lookup in Python. Then I use one {% include d.template %} to pull the results into the template. This Django stuff cleans up real nice.

The Template Made Me Do It

I've been doing a little Django lately. Django's template language isn't Python; it's a somewhat Pythonic template language, and it's limited by design. At first, it frustrated me; after a brief battle, it surprised me.

My template was pretty simple: generate a monthly calendar page. Every day gets the same HTML, except for three special cases: first Wednesday of the month, other Wednesdays, and all Fridays.

In my first version, I passed a list of datetime.date objects in the context. The template for-looped through the list, and did things like

{% ifequal d.isoweekday 3 %}

All by itself, that's mildly evil - the hard-coded magic number, though I can probably remember that isoweekday() starts counting at Monday == 1. Then I started nesting the if-else tags, and things got messy quickly. The Wednesday case, the Friday case, the first-Wednesday case: frustration. I wanted Python at my fingertips, not this crippled template language.

Which, of course, was the answer. I needed better abstractions for the template, and Python was the right place to create them. I subclassed datetime.date and added the methods I needed: is_wednesday(), is_first_wednesday(), is_friday(), is_special()

Which let me write reasonable template code like

{% if d.is_special %}
    {% if d.is_first_wednesday %}

And then the surprise: I realized that fixing the messy template had also made my Python code cleaner. I ended up with a nice little app-specific date class, unit tests and everything. That was the right thing to do, but I hadn't bothered till I needed it for the template. The limitations of the template language drove the refactoring.

And that makes me think that limiting the template language was a very good idea indeed.

Do Be Do Be Done

Do you ever feel like the big-name web frameworks have grown too much, that they're bloated from an endless diet of new features?

Here's fast relief for that bloated feeling: Sinatra, a lightweight Ruby web framework. Minimal docs, but $10 gets you traction from a couple of excellent screencasts.

Make An Effort(s)

I wrote some simple code today: a batch job that counts the number of items in a queue, then emails the count to a couple of people. My first version sent a message like this:

There are 3 item(s) in the queue.

Then I smacked myself and dove back into the code. Now the message looks like:

There are 3 items in the queue.

or:

There is 1 item in the queue.

Maybe ten minutes to code and test, and I feel better. The item(s) dodge is lazy, and looks so cheesy.

Lotus Domino - Sometimes Store And Forward Wins

In my day job, I babysit a fairly large Lotus Domino web application – call it ELF – at a very large financial services company. I do all the development and most of the production support.

Some parts of ELF work like this:

1. A customer request comes in through the web front end. (ELF handles about a dozen different kinds of requests);


2. The web server stuffs the request into a Domino database;


3. A batch job moves the request to a relational database.

At first glance, step 2 seems redundant. Why slow things down? Why not just go directly to the relational database? After all, data redundancy is a bad thing. It can lead to discrepancies, it's wasteful, it's needless complexity.

All true, but this basic store and forward architecture has a real advantage: it's more robust than the typical web application (call it TWA). TWA looks something like this:

• Web server running on one machine;


• Application server, a separate piece of software, running as a separate process on the same machine, or on another machine;


• Database server, a separate piece of software, almost certainly running on another machine.

That's three potential points of failure. Plus network risk. Maybe the application server is using web services to store the data – another potential point of failure. And there's configuration risk – more about that later.

Domino OTOH is a large, monolithic monster. It provides everything - the web server, the application server, the database server, even a batch scheduler. All in one server, on one machine. This eliminates a lot of the risks. If the web server is up, then the database server is up, and the web server can talk to it. If Domino can serve up pages, it can store the data that come back.

For ELF, this means independence from the back-end relational databases. If they crash, if they go off-line for maintenance, if there's a network problem, ELF doesn't care. It keeps accepting customer requests and storing them in Domino. Every couple of hours, it tries to reach the relational databases. If they're available, fine – it transfers the requests. If not, no worries, it tries again later. And the website stays up.

Here's a real-life example, from last week, that got me thinking about all this. I mentioned configuration risk. In TWA, the application server probably stores its database credentials – userid, password – in a secure configuration file. But what if some overzealous dolt disables the database userid? That kind of thing happens in big companies. And when it does, TWA stops working, till someone figures out what happened and gets it fixed. Again, in a big company, that can take a surprisingly long time.

But when this happened to ELF, it kept rolling along. We didn't even know there was a problem till the people on the relational database side noticed they weren't seeing any inserts. We looked into it, found the credentials problem, and got it fixed. But the userid went bad on a Friday, when I was out. By the time we got the problem fixed, it was Tuesday - five days. All that time, ELF accepted customer requests and stored them in Domino. Once the credentials were fixed, it transferred all the requests. No data loss, no unhappy website users.

TWA can't load-balance around a problem like this. But let's reshape TWA a bit. Let's have the application server write all the front-end data into a local store, on the same machine. Maybe it uses the filesystem, maybe sqlite. Whatever's fast and reliable. After that, let some batch job deal with it. Any problems reaching the “real” datastore are now in the background. The application server can still capture the user's data and send good news back to the browser.

Obviously, if your customers are trading stock, you don't want to do this. But if they're buying books or T-shirts, maybe you can get away with it. And it's a whole lot better than “Try again later.”

Clojure and MySQL, continued

I've been working through the Data Access section of the Clojure in the Wild chapter in Stuart Halloway's Programming Clojure. Here's a MySQL version of the last-created-id form:

(defn last-created-id
  "Extract the last created id. Must be called in a transaction
  that performed an insert. MySQL version."
  []
  (:last_created_id (first (sql-query "select LAST_INSERT_ID() as last_created_id"))))


Clojure and MySQL

Here's the magical connection incantation to get Clojure and MySQL talking:

(def *db* {:classname "com.mysql.jdbc.Driver"
  :subprotocol "mysql" :subname "//your-server/your-db"
  :user "your-user" :password "your-password"})

Assumptions - you're using

• Stephen C. Gilardi's clojure.contrib.sql library, and


• MySQL's Connector/J JDBC driver, with mysql-connector-java-5.1.7-bin.jar in your Clojure classpath.
  

Clojure is elegant. It's from the future, with features like support for software transactional memory. Meantime, it's a close personal friend of Java, so it plays well with more mundane tools. I get a buzz when that happens. I've flirted with lisps before, but this one feels like a keeper.

Incidentally, the Connector/J license defaults to GPL, so it may contaminate your application like ice-nine. Hail Bokonon!