I’m about 30 pages into William Zinsser’s excellent book On Writing Well when I’m struck by my realization.Over the past nine months, I’ve been writing about abstract ideas, day to day happenings and insights into the world around me, everything except my work. The short and simple reason for this is because I’m afraid.

When I started Bumblebee Labs, I made myself promise that I would be open about the work I was doing. But as soon as I started writing, it was always easier to write about anything but that. Writing about my own work felt embarrassing. The ideas and concepts that I had were so unclear in my own head that I never felt I could do them justice. And whenever I felt guilty about not writing, there were always other things to write about.

Well, from today, that’s going to stop. The pieces about ideas are still going to come but they’re going to be mixed with much more personal and relevant content. I’m going to conquer my fears of being a fool and start writing more of what made me start this blog.

Ugh, yet another media establishment is running the fallacious facebook redesign argument and acting all clever about it. Sadly, this time it’s one I actually respect.

Not statistically significant…

Most people have no idea what “Not statistically significant” means and I don’t see the media being too eager to fix this.

Say you read the following piece in a newspaper:

A study done at the University of Washington showed that, after controlling for race and socioeconomic class, there was no statistically significant difference in athletic performance between those who stretched for 5 minutes before running and those who did no stretching at all.

What do you conclude from that? Stretching is useless? WRONG.

Here’s what the hypothetical study actually was: I picked four random guys on campus and asked two of them to stretch and two of them not to. The ones who stretched ran 10% faster.

Why is this then not statistically significant? Because the sample size was too small to infer anything useful and the study was designed poorly.

All “not statistically significant” tells you is that you can’t infer anything from the study but word the study carefully enough and you can have people believe the opposite is true.

Have you ever heard the claim “There’s no statistically significant difference between going to an elite Ivy League school and an equally good state school?” Perhaps from here, here or even here?

Well, from this paper (via a comment in an Overcoming Bias post):

For instance, Dale and Krueger (1999) attempted to estimate the return to attending specific colleges in the College and Beyond data. They assigned individual students to a “cell” based on the colleges to which they are admitted. Within a cell, they compared those who attend a more selective college (the treatment group) to those who attended a less selective college (the control group). If this procedure had gone as planned, all students within a cell would have had the same menu of colleges and would have been arguably equal in aptitude. The procedure did not work in practice because the number of students who reported more than one college in their menu was very small. Moreover, among the students who reported more than one college, there was a very strong tendency to report the college they attended plus one less selective college. Thus, there was almost no variation within cells if the cells were based on actual colleges. Dale and Krueger were forced to merge colleges into crude “group colleges” to form the cells. However, the crude cells made it implausible that all students within a cell were equal in aptitude, and this implausibility eliminated the usefulness of their procedure. Because the procedure works best when students have large menus and most student do not have such menus, the procedure essentially throws away much of the data. A procedure is not good if it throws away much of the data and still does not deliver “treatment” and “control” groups that are plausibly equal in aptitude. Put another way, it is not useful to discard good variation in data without a more than commensurate reduction in the problematic variation in the data. In the end, Dale and Krueger predictably generate statistically insignificant results, which have been unfortunately misinterpreted by commentators who do not sufficient econometric knowledge to understand the study’s methods.

In other words, the study says no such thing, it simply says the study itself was not sufficient to prove that Ivy League educations made you more money because the data wasn’t good enough and yet the media has twisted this into a positive assertion that state schools do indeed make you as much money as Ivy Leagues.

I’m generously inclined to believe that most cases that I see of this error are caused by incompetence but it’s pretty trivial to see how this could be used for malice. Want the public to believe that Internet usage doesn’t cause social maladjustment? Just design a shitty study and claim “We found no statistical difference in social competence between heavy internet users, light internet users and non users”. Bam, half the PR work has already been don for you.

Controlling for…

Here’s another statistical gem I see all the time:

An analysis done at the University of Washington showed that there was zero correlation between race and financial attainment after controlling for IQ, education levels, socioeconomic status and gender.

Heartwarming right, it means if we put blacks and whites in the same situation, they should earn the same amount of money. WRONG.

The key here is to see that we’re looking for financial attainment and controlling for socioeconomic status. Those two things mean the same damn thing. Basically, all this study told us was that being rich causes you to be rich.

Most people view the “controlling for” section of statistical reporting as a sort of benign safeguard. Controlling for things is like… due diligence right, the more the better… It’s easy to numb people into a hypnotic lull with a list of all the things you control for.

But controlling for factors means you get to hide the true cause for things under benign labels. That’s why I’m always so wary of studies that control for socioeconomic status or education levels, especially when they don’t have to. Sure, socioeconomic status might cause obesity but what causes socioeconomic status.

Conclusion

When people do bother to talk about statistical manipulation, they usually focus on issues of statistical fact: Aggressive pruning of outliers, shotgun hypothesis testing and overly loose regressions. But why bother with having to sneak poorly designed studies past peer review when you can just publish a factually accurate study which implies a conclusion completely at odds with the data? That way, you sneak past the defenses of anyone who actually does know something about statistics.

Sometimes, I swear, the more statistically savvy a person thinks they are, the easier they are to manipulate. Give me a person who mindlessly parrots “Correlation does not imply causation” and I can make him believe any damn thing I want.

It’s a common complaint for the tech savvy that you need to go through the gauntlet of dumb questions (did you check if the computer is on?) before you get anywhere with customer service. What if a company made it a policy that a certain proportion of their customers could get “upgraded” into getting direct tier 2 support. Now customers have an incentive to be nice to the reps because the reps are able to reward them and reps have a better gauge of what level of technical sophistication the customer is.

Douglas Bowman, Google’s lead visual designer announced yesterday that he was leaving Google to join Twitter. At the root of it, Bowman’s decision to leave stems from a clash of cultures between the world of Interaction and Visual Design. The best way to understand this this clash of cultures is to listen to the ghost stories each field tells the young’uns.

In Interaction Design, around the campfires at night, it’s common to hear a variant of this chilling tale:

I heard, there was this company once, where they, like, got these totally great designers to build this user interface for them and they were all excited about it being the best thing since sliced toast until they tried to watch some people use it in the real world and it, like, totally sucked. The things everyone thought were easy to use were completely confusing. Luckily, they went through several iterations of redesign and testing the thing until it became something users loved.

Interaction designers are actively trained to filter out expert opinion as a justification for design decisions. The expert, no matter how qualified and trained they are, is ultimately, not the user and is ultimately, totally ineffectual and predicting what the user is like. The only way that design decisions can be justified is through feedback from actual users. Uttering the words “I prefer…” as justification for a design decision is the quickest way to move you from the potentially-an-ally category to dangerous-fool-who-must-be-neutralized category in the eyes of an interaction designer.

Over in the Visual Designer camp, a different ghost story is being passed round the campfire:

I heard, there was this company once who hired this, like, genius visual designer who built them this totally bold and brilliant design. But then, in an attempt to please everyone, the design was buried under so many focus groups and QA evaluations that  integrity of the design was destroyed and what was ultimately put up, like, totally sucked and ended up pleasing no one. Luckily, a more design friendly management was put into place and the original design was restored which ended up creating the emotional bond with the users that saved the company.

Visual designers are trained to keep their artistic integrity in the face of pressure and to be the keepers of the secret knowledge against the tide of the aesthetically ignorant. Uttering the words “consensus seeking” as justification for a design decision is the quickest way for you to become a dangerous-fool-who-must-be-neutralized in the eyes of a visual designer.

You can see both of these dynamics play out in the Google saga. Douglas Bowman’s characterization of the design process at Google:

Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such miniscule design decisions. There are more exciting design problems in this world to tackle.

The debate on border pixels dragged on because Bowman became a dangerous-fool-who-must-be-neutralized in the eyes of the interaction design team.

Similarly, on Marissa Mayer’s attempt to reach out towards the visual designers:

A designer, Jamie Divine, had picked out a blue that everyone on his team liked. But a product manager tested a different color with users and found they were more likely to click on the toolbar if it was painted a greener shade.

As trivial as color choices might seem, clicks are a key part of Google’s revenue stream, and anything that enhances clicks means more money. Mr. Divine’s team resisted the greener hue, so Ms. Mayer split the difference by choosing a shade halfway between those of the two camps.

Is so, tin-earred it’s cringe inducing. Like rich yuppies trying to connect with the less affluent by speaking the language of the “street”, Marissa reads the culture of visual design so wrong and her attempt and consensus and compromise ends up doing more harm than good.

The sad thing is, both of these viewpoints are perfectly justified and are the result of a counter-intuitive lesson learned. Both of these ghost stories are repeated precisely so the newbies in the field don’t end up making the same mistakes the pros once made. Unfortunately this means for both sides, the views of the other side look like ignorance.

Look, I was like you once, and then I learned better. So I’m just going to sit hear and wait for the other shoe to drop for you Mmmkay? Do you want to hear a ghost story while we’re waiting?

So what you end up getting is a staring contest where each side is waiting for the other to finally blink. Unfortunately, in this case, Douglas Bowman blinked first and both Douglas and Google were both impoverished for this.

PS: In anticipation of the criticism that I have no business talking about visual design when the design of my own site sucks so much, I know, it’s being fixed, be patient.

Wow, I’ve been talking about facebook a lot recently. I’ve been hearing this argument a lot about the redesign:

1. Here is some evidence that users hate the new design
2. But users have hated every redesign of facebook when it was first rolled out
3. And eventually users learned to love it

There’s an implicit logical fallacy in this: Just because some users hate good redesigns because they’re change-phobic, does not mean that hating it is evidence for it being a good redesign. Unfortunately, facebook as a culture has learned to ignore user feedback until it gets to the point of overwhelmingness for precisely this reason. Mark my words, this rebellion is not going to be quelled like the last few were because they really are pointing to systematic failures in the design.

Every time facebook does a redesign, the basic concept is sound but they manage to get the specifics wrong. I was enthusiastic when they moved to a real time streaming feed system but not so much after I discovered they removed the filtering algorithm at the same time which allows this sort of thing:

What gummed up Skinner’s reputation is that he developed a case of what I always call man-with-a-hammer syndrome: to the man with a hammer, every problem tends to look pretty much like a nail.

The Psychology of Human Misjudgment is an absolutely brilliant talk given by Charlie Munger (#2 at Berkshire Hathaway) that I still return to and read every year to gain a fresh perspective. There’s a lot of wisdom to be distilled from that piece but the one thing I want to talk about today is the man-with-a-hammer syndrome.

Man-with-a-hammer syndrome is pretty simple: you think of an idea and then, pretty soon, it becomes THE idea. You start seeing how THE idea can apply to anything and everything, it’s the universal explanation for how the universe works. Suddenly, everything you’ve ever thought of before must be reinterpreted through the lens of THE idea and you’re on an intellectual high. Utilitarianism is a good example of this. Once you independently discover Utilitarianism you start to believe that an entire moral framework can be constructed around a system of pleasures and pains and, what’s more, that this moral system is both objective and platonic. Suddenly, everything from the war in the middle east to taking your mid-morning dump at work because you need that 15 minutes of reflective time alone with yourself before you can face the onslaught of meaningless drivel that is part of corporate America but feeling guilty about it because you were raised to be a good Randian and you are not providing value from your employers so you’re committing and act of theft can be fit under the Utilitarian framework. And then, hopefully, a few days later, you’re over it and Utilitarianism is just another interesting concept and you’re slightly embarrassed about your behavior a few days prior. Unfortunately, some people never get over it and they become those annoying people write long screeds on the internet about THE idea.

The most important thing to realize about man-with-a-hammer syndrome is that there’s absolutely no possible way to avoid having it happen to you. You can be a well seasoned rationalist who’s well aware of how man-with-a-hammer syndrome works and what the various symptoms are but it’s still going to hit you fresh with each new idea. The best you can do is mitigate the fallout that occurs.

Once you recognize that you’ve been struck with man-with-a-hammer syndrome, there’s a number of sensible precautions you can take. The first is to have a good venting spot, being able to let your thoughts out of your head for some air lets you put them slightly in perspective. Personally, I have a few trusted friends to which I expose man-with-a-hammer ideas with all the appropriate disclaimers to basically ignore the bullshit that is coming out of my mouth. One thing I’m experimenting with is a less public portion of my blog to put that kind of stuff on (which will be made much clearer after the sorely needed redesign).

The second important thing to do is to hold back from telling anyone else about the idea. Making an idea public means that you’re, to a degree, committed to it and this is not what you want. The best way to prolong man-with-a-hammer syndrome is to have other people believing that you believe something.

Unfortunately, the only other thing to do is simply wait. There’s been nothing I’ve discovered that can hasten the recovery from man-with-a-hammer syndrome beyond some minimum time threshold. If you’ve done everything else right, the only thing left to do is to simply out wait it. No amount of clever mental gymnastics will help you get rid of the syndrome any faster and that’s the most frustrating part. You can be perfectly aware that you have it, know that everything you’re thinking now, you won’t believe in a weeks time and yet you still can’t stop yourself from believing in it now.

Man-with-a-hammer syndrome can destroy your life if you’re not careful but, if handled appropriately, is ultimately nothing more than an annoying and tedious cost of coming up with interesting ideas. What’s most interesting about it to me is that even with full awareness of it’s existence, it’s completely impossible to avoid. While you have man-with-a-hammer syndrome, you end up living in a curious world in which you are unable to disbelieve in something you know to be not true and this is a deeply weird state I’ve not seen “rationalists” fully come to terms with.

It seems like how I use facebook is radically different from how facebook thinks I use facebook. For me, events are one of the killer apps that facebook provides, it’s basically the driver for my social calendar. But facebook has oddly seemed to relegate it to the red headed stepchild of it’s feature list. I talked a while ago about how you could infer company priorities through their mobile offerings and it’s quite telling how facebook regards events. Let’s review the status quo:

• Facebook for the iPhone app: events are completely missing. Theres no way to see them, there’s no way to interact with event news on the news feed, it’s almost soviet in it’s denial that events exist.
• Facebook website for the iPhone: Events are linked to prominently on the front page but the event description page is missing several crucial pieces of information. Address is listed but not location, both host and description are missing and, most importantly, the attendee list is missing. Photos for events is devoted an entire tab despite the fact that maybe 5% of events I go to ever upload any photos.
• Facebook Mobile: This is the only mobile offering that actually has a usable event interface but it’s also the least rich in user interface and most annoying to navigate to.
• Facebook main website: Post redesign, events are getting an incredibly short shrift on the new main website. Sure, you can see what everyone else is going to but trying to figure out how to get to a list of my events for the next week took me twenty minutes of poking around (click on the event app in the bottom left). I’m not sure anyone but a power user can still figure out how to see what events they committed to more than two days in advance.

I have to admit, I’ve quite puzzled by facebook’s attitude on this given that not only do I regard events as an essential part of facebook, I regard it as pretty much the killer app of facebook mobile. When I’m away from my computer, the chances are better than even that I’m heading to an event, at an event or leaving an event. Facebook needs to recognise the opportunity it’s been missing by neglecting events and return them back to their rightful place in the facebook ecosystem.

I’m a huge tetris geek and so when I discovered Torus, my first reaction was “That’s so obvious, why the hell didn’t I think of that?” which lead me to thinking about how almost every person has their own personal theory about how they think innovation happens and yet they are so rarely inclined to put that theory under empirical scrutiny.

Some people believe that innovation is technology limited and that as soon as a new product becomes practical, someone will build it. Often, the critical technological factor might not be the most obvious one. Looking at torus, their reaction would be that sure, it would have been technically feasible 20 years ago but such a variant never would have spread without the viral power of the internet. Because there’s so many entrepreneurs working on so many different approaches to the problem, one of them is bound to hit on a good idea eventually.

Other people believe that innovation is a matter of luck, talent and persistance. MP3 players as good as the iPod and search engines as good as Google were perfectly possible well before they came out but it took the genius of the designers at Apple/Google to finally show people what an MP3 player/search engine could be.

There are still others who believe innovation is a social process driven by fads and fashions. People innovated in social networking because social networking was what’s hot. Now, they’re innovating in iPhone apps. Driving innovation is largely a matter of pushing trends.

In truth, all of these explainations are more or less valid in different areas and every sophisticated person holds a complex mix of all these views but I think it’s interesting and useful to articulate your own view so that you can determine whether it’s correct or not.