Being on the bleeding edge of progress means you see new technologies come out all the freaking time. Some of them are truly worthless and can be safely ignored. Most of them will be intriguing but ultimately what you would expect. A very few of them have the potential to surprise the hell out of you and those are the ones worth keeping an eye on.

About 20 years ago, the surprising thing of the day was using commodity hardware to build supercomputers. Before that point, the way to make supercomputers better was to utilize every hertz of processing power through custom hardware and clever software. The revolution of commodity hardware was not in the engineering, it was in the shift in thinking. The new way to solve hard problems was to just design simple, less efficient algorithms and throw more hardware at the problem. That shift changed not only the types of applications that could be built but also the way we think about building apps. The reason Mechanical Turk is worth keeping an eye on is because its about to do something similar for labor.

When Amazon released its iPhone app my entire understanding of what was possible changed. You load up the app, snap a picture of an object, Amazon will use Mechanical Turk to find the closest Amazon equivalent and, within about 5 minutes, you can buy it for one click. The application itself was a beautiful usage of Mechanical Turk but more interesting is how a shift in thinking had to occur before it could even be imagined. That Amazon is releasing this app for free but paying for human labor means  their business model relies on human labor being cheap enough to hide in the margins. At the same time, the user experience is only compelling because the search results come back before you’ve left the store so Amazon needs to assumes the pool of available labor as essentially infinite to deliver that experience.

Once you’re able to get over that hump of believing that human labor can only be an expensive, limited resource, an entire vista of compelling applications open up. Here’s one I came up with today in a conversation with a collegue: Calorie tracking sucks because of the data entry problem. You need to manually enter in every single thing you ate and that requires far more organization than most people have. Why not just snap a photo with your iPhone and let a Mechanical Turker figure out what you ate? How do you solve the reliability problem? Have every picture looked at by at least three Turkers and only accept it if at least two agree. When labor becomes that cheap, its smarter to be dumb and throw more human hardware at the problem.

Does that mean Mechanical Turk will do to human labor what the commodity hardware & cloud computing did to server farms? Of course not, the analogy is instructive, not a direct mapping. What it does mean is that we as a society are going to experience several “everything we knew was wrong” type moments and that the labor market of 2039 will look as different from today as supercomputers did in 1979 and those who are the first to recognise this change will be the ones who have the best chance of exploiting it.

One reflection of my time at CHI this year is the luxury that academics have on thinking deeply about a particular domain. In the startup/Web 2.0/corporate world, there’s a certain hummingbird like intensity in which people flit from topic to topic and apply only the lightest and most shallow analysis on everything they touch. There’s always more work to be done and new things that need to be grokked and so many of the people I meet are jack of all trades but master of none. Academics, by contrast, are expected to know one domain of knowledge well and, while it can lead them to becoming comically out of touch with how technology is being deployed and used in the real world, also gives them a perspective and history which is interesting to engage with.

The cause of this is understandable, thinking deeply is a luxury and often somewhat of a guilty pleasure. There’s always something pressing to be done or yet another domain that needs to be mastered. Yes, there’s a lot of criticism that academia becomes an ivory tower but I think there’s value in deliberately cultivating an environment in which you’re forced to think deeply.

However, one persistant criticism I have about academia is it’s failure to engage with the larger discourse that’s happening on the web. My first example of this was actually after my first CHI in 2006 in Montreal. At CHI, a young graduate student called Anand Agrawala presented a neat little system called BumpTop (it launched as a commercial product almost exactly 3 years later while I was at CHI 2009). After the conference, Anand put the BumpTop video online and it quickly became the #1 viewed video on youtube. Working in the field of tabletops at that time, I had some understanding of that space but, being a first year graduate student, I didn’t feel like I could adequately comment on it. But after looking at reams of commentary about the system from a variety of different sources, what I continually failed to see was the insightful and grounded critique I was used to seeing in Academia. Everyone commenting on it approached it from a complete vacuum, ignoring the important work that had gone before it and the hard won lessons of the field.

From that point, I’ve seen, time after time, interesting HCI systems make a larger splash within the general public but no voices of informed critique there to educate and contextualise the news. Part of the reason I made a commitment to blog about CHI this year was to help provide people access to this world of deep thinking that exists within the academic community and to make industry people aware of this immense, untapped resource. I don’t know what the solution to this bridge is or even if there is a solution. I’m certainly not the first or the last to bemoan the gulf between these two worlds. But I think, because this gulf exists, anyone willing to take advantage of it can often profit hugely.

I’ve never liked sites that replicate the paper-based conventions for footnoting. The entire point of the web is that it’s non-linear and multi-dimensional. Does anyone know of any good tools (or website) that have footnotes in-place in the text which dynamically expand when you click on them?

I’m about 30 pages into William Zinsser’s excellent book On Writing Well when I’m struck by my realization.Over the past nine months, I’ve been writing about abstract ideas, day to day happenings and insights into the world around me, everything except my work. The short and simple reason for this is because I’m afraid.

When I started Bumblebee Labs, I made myself promise that I would be open about the work I was doing. But as soon as I started writing, it was always easier to write about anything but that. Writing about my own work felt embarrassing. The ideas and concepts that I had were so unclear in my own head that I never felt I could do them justice. And whenever I felt guilty about not writing, there were always other things to write about.

Well, from today, that’s going to stop. The pieces about ideas are still going to come but they’re going to be mixed with much more personal and relevant content. I’m going to conquer my fears of being a fool and start writing more of what made me start this blog.

Not statistically significant…

Most people have no idea what “Not statistically significant” means and I don’t see the media being too eager to fix this.

Say you read the following piece in a newspaper:

A study done at the University of Washington showed that, after controlling for race and socioeconomic class, there was no statistically significant difference in athletic performance between those who stretched for 5 minutes before running and those who did no stretching at all.

What do you conclude from that? Stretching is useless? WRONG.

Here’s what the hypothetical study actually was: I picked four random guys on campus and asked two of them to stretch and two of them not to. The ones who stretched ran 10% faster.

Why is this then not statistically significant? Because the sample size was too small to infer anything useful and the study was designed poorly.

All “not statistically significant” tells you is that you can’t infer anything from the study but word the study carefully enough and you can have people believe the opposite is true.

Have you ever heard the claim “There’s no statistically significant difference between going to an elite Ivy League school and an equally good state school?” Perhaps from here, here or even here?

Well, from this paper (via a comment in an Overcoming Bias post):

For instance, Dale and Krueger (1999) attempted to estimate the return to attending specific colleges in the College and Beyond data. They assigned individual students to a “cell” based on the colleges to which they are admitted. Within a cell, they compared those who attend a more selective college (the treatment group) to those who attended a less selective college (the control group). If this procedure had gone as planned, all students within a cell would have had the same menu of colleges and would have been arguably equal in aptitude. The procedure did not work in practice because the number of students who reported more than one college in their menu was very small. Moreover, among the students who reported more than one college, there was a very strong tendency to report the college they attended plus one less selective college. Thus, there was almost no variation within cells if the cells were based on actual colleges. Dale and Krueger were forced to merge colleges into crude “group colleges” to form the cells. However, the crude cells made it implausible that all students within a cell were equal in aptitude, and this implausibility eliminated the usefulness of their procedure. Because the procedure works best when students have large menus and most student do not have such menus, the procedure essentially throws away much of the data. A procedure is not good if it throws away much of the data and still does not deliver “treatment” and “control” groups that are plausibly equal in aptitude. Put another way, it is not useful to discard good variation in data without a more than commensurate reduction in the problematic variation in the data. In the end, Dale and Krueger predictably generate statistically insignificant results, which have been unfortunately misinterpreted by commentators who do not sufficient econometric knowledge to understand the study’s methods.

In other words, the study says no such thing, it simply says the study itself was not sufficient to prove that Ivy League educations made you more money because the data wasn’t good enough and yet the media has twisted this into a positive assertion that state schools do indeed make you as much money as Ivy Leagues.

I’m generously inclined to believe that most cases that I see of this error are caused by incompetence but it’s pretty trivial to see how this could be used for malice. Want the public to believe that Internet usage doesn’t cause social maladjustment? Just design a shitty study and claim “We found no statistical difference in social competence between heavy internet users, light internet users and non users”. Bam, half the PR work has already been don for you.

Controlling for…

Here’s another statistical gem I see all the time:

An analysis done at the University of Washington showed that there was zero correlation between race and financial attainment after controlling for IQ, education levels, socioeconomic status and gender.

Heartwarming right, it means if we put blacks and whites in the same situation, they should earn the same amount of money. WRONG.

The key here is to see that we’re looking for financial attainment and controlling for socioeconomic status. Those two things mean the same damn thing. Basically, all this study told us was that being rich causes you to be rich.

Most people view the “controlling for” section of statistical reporting as a sort of benign safeguard. Controlling for things is like… due diligence right, the more the better… It’s easy to numb people into a hypnotic lull with a list of all the things you control for.

But controlling for factors means you get to hide the true cause for things under benign labels. That’s why I’m always so wary of studies that control for socioeconomic status or education levels, especially when they don’t have to. Sure, socioeconomic status might cause obesity but what causes socioeconomic status.

Conclusion

When people do bother to talk about statistical manipulation, they usually focus on issues of statistical fact: Aggressive pruning of outliers, shotgun hypothesis testing and overly loose regressions. But why bother with having to sneak poorly designed studies past peer review when you can just publish a factually accurate study which implies a conclusion completely at odds with the data? That way, you sneak past the defenses of anyone who actually does know something about statistics.

Sometimes, I swear, the more statistically savvy a person thinks they are, the easier they are to manipulate. Give me a person who mindlessly parrots “Correlation does not imply causation” and I can make him believe any damn thing I want.

What gummed up Skinner’s reputation is that he developed a case of what I always call man-with-a-hammer syndrome: to the man with a hammer, every problem tends to look pretty much like a nail.

The Psychology of Human Misjudgment is an absolutely brilliant talk given by Charlie Munger (#2 at Berkshire Hathaway) that I still return to and read every year to gain a fresh perspective. There’s a lot of wisdom to be distilled from that piece but the one thing I want to talk about today is the man-with-a-hammer syndrome.

Man-with-a-hammer syndrome is pretty simple: you think of an idea and then, pretty soon, it becomes THE idea. You start seeing how THE idea can apply to anything and everything, it’s the universal explanation for how the universe works. Suddenly, everything you’ve ever thought of before must be reinterpreted through the lens of THE idea and you’re on an intellectual high. Utilitarianism is a good example of this. Once you independently discover Utilitarianism you start to believe that an entire moral framework can be constructed around a system of pleasures and pains and, what’s more, that this moral system is both objective and platonic. Suddenly, everything from the war in the middle east to taking your mid-morning dump at work because you need that 15 minutes of reflective time alone with yourself before you can face the onslaught of meaningless drivel that is part of corporate America but feeling guilty about it because you were raised to be a good Randian and you are not providing value from your employers so you’re committing and act of theft can be fit under the Utilitarian framework. And then, hopefully, a few days later, you’re over it and Utilitarianism is just another interesting concept and you’re slightly embarrassed about your behavior a few days prior. Unfortunately, some people never get over it and they become those annoying people write long screeds on the internet about THE idea.

The most important thing to realize about man-with-a-hammer syndrome is that there’s absolutely no possible way to avoid having it happen to you. You can be a well seasoned rationalist who’s well aware of how man-with-a-hammer syndrome works and what the various symptoms are but it’s still going to hit you fresh with each new idea. The best you can do is mitigate the fallout that occurs.

Once you recognize that you’ve been struck with man-with-a-hammer syndrome, there’s a number of sensible precautions you can take. The first is to have a good venting spot, being able to let your thoughts out of your head for some air lets you put them slightly in perspective. Personally, I have a few trusted friends to which I expose man-with-a-hammer ideas with all the appropriate disclaimers to basically ignore the bullshit that is coming out of my mouth. One thing I’m experimenting with is a less public portion of my blog to put that kind of stuff on (which will be made much clearer after the sorely needed redesign).

The second important thing to do is to hold back from telling anyone else about the idea. Making an idea public means that you’re, to a degree, committed to it and this is not what you want. The best way to prolong man-with-a-hammer syndrome is to have other people believing that you believe something.

Unfortunately, the only other thing to do is simply wait. There’s been nothing I’ve discovered that can hasten the recovery from man-with-a-hammer syndrome beyond some minimum time threshold. If you’ve done everything else right, the only thing left to do is to simply out wait it. No amount of clever mental gymnastics will help you get rid of the syndrome any faster and that’s the most frustrating part. You can be perfectly aware that you have it, know that everything you’re thinking now, you won’t believe in a weeks time and yet you still can’t stop yourself from believing in it now.

Man-with-a-hammer syndrome can destroy your life if you’re not careful but, if handled appropriately, is ultimately nothing more than an annoying and tedious cost of coming up with interesting ideas. What’s most interesting about it to me is that even with full awareness of it’s existence, it’s completely impossible to avoid. While you have man-with-a-hammer syndrome, you end up living in a curious world in which you are unable to disbelieve in something you know to be not true and this is a deeply weird state I’ve not seen “rationalists” fully come to terms with.

Finally figuring out that you’re abnormal is akin to coming home from work and realizing that your shirt was on backwards the entire day, you wonder how many other people noticed it and didn’t say anything. I get that same feeling when I figure out that something I thought was totally normal and that everyone did was actually something that only a very few people possessed.

When I was small, I always imagined there was another me, floating roughly 3 feet behind me and above my head, looking down on me. This wasn’t an imaginary friend or an out of body experience or anything like that, it was just the part of me that lived outside of my body and could observe the world from a different perspective. The inside me would be worried about how I was feeling and my own emotional reactions but the outside me was completely indifferent to my well being and was the hyper-logical, hyper-rational being who would crunch the numbers and present the truth in an impersonal manner. Inside me and outside me would argue a lot and outside me would never win but I would at least respect his opinions. It took me until I was maybe 17 or 18 before I realized that the reason why my appeals to the outside them was not working for most people was because their outside them didn’t exist and they were unable to consider an argument from anything but a personal perspective. Because I had an outside me, I thought everyone else must have an outside them as well.

One recent skill I finally figured out not many people had is the ability to learn about a subject without subscribing to the orthodox view and it took me the recent financial crisis to hammer that home to me. When I first learned economics, I came into it enthusiastic but well aware of it’s limitations and simplifications. Surely, if I could see how economics was limiting from a high school class, these professionals with their fancy educations must be very well versed in just what the flaws and deficiencies in their subject was. It wasn’t until this crisis hit that it finally hit home to me the level of intellectual shallowness and mindlessness most economic professionals were operating under. Sure, they knew the theory and how to apply it but they also believed that, because they knew it, it was true.

Knowing that you have a skill is useful but finding out that other people don’t have that skill is infinitely more powerful because it makes you unique. However, finding such things out generally requires a jolt out of your normal circumstances. I’m really curious to see if any other people have stories about discovering unusual abilities?

I’m reading a book right now which I’ll name in a later post which contains, roughly according to my estimate, about 80% wisdom to 20% bullshit and it strikes me that this is the absolutely worst ratio possible. In fact, making a mental list of wisdom:bullshit ratios made me think how highly non-linear my personal scale is. So here it is from best to worst:

• 90:10 – These are the books I most enjoy reading as the amount of bullshit is just enough for you to calibrate your filter and be able to extract out the deep wisdom hidden within. I would say Freakanomics would fit squarely into this category.
• 100:0 – Although, intuitively, you would think a book with no bullshit is better than a book with some bullshit, it’s actually not because you now no longer have a scale of comparison. Did you not detect bullshit because it wasn’t there or because it was too advanced for you to understand? I don’t think I’ve read any books that fall squarely in this category, at least not for a long time.
• 0:100 - Simple, skim a couple of pages and throw it away, your life remains unaffected. The Secret would be a good book of this category.
• 50:50 – These books are challenging and require you to have your bullshit filter on full force. You’re forced to comb through every sentence and carefully consider each statement. Such books can often cause you to radically shift your thinking or at least inject doubt into your process. The 48 laws of power was a book like that for me.
• 30:70 / 40:60 / 60:40 /70:30 – For the most part, this will be largely everything you read. Most mass media falls into this comfortable middle range and, although it’s challenging to parse properly, you become so used to doing it that it’s routine.
• 10:90 / 20:80 – These books are frustrating because there’s a hidden core of wisdom surrounded by so much utter crap that they’re emotionally draining to read and yet the small insights urge you to keep going. Atlas Shrugged was definitely in that category for me. I hated it so much, I inexplicably read it again a few years later, gritting my way through the entire book.
• 80:20 – The 80:20 book is the book that derails people’s lives. Mixed in among all that wisdom is just a few key mistakes that can lead you far, far astray. This is how cults recruit their members and smart financial wizards managed to argue their way into this financial crisis. The wisdom is compelling enough that you let down your guard and don’t realize how much bullshit you let in until it’s too late. I’ll be providing a review of the current book I’m reading in a few days with the case of why I believe it to be an 80:20 book.

In the meantime, I’d like to hear your examples of books you think fit into any of the above categories…

I sometimes catch myself thinking something and then try and trace back my chain of thought to the original thought that sparked it…

Also, note to self: build blog integration into Am I Normal or Not…

I was reading Paul Graham’s Startups in 13 sentences and came along this interesting piece of advice:

7. You make what you measure.

I learned this one from Joe Kraus. [3] Merely measuring something has an uncanny tendency to improve it. If you want to make your user numbers go up, put a big piece of paper on your wall and every day plot the number of users. You’ll be delighted when it goes up and disappointed when it goes down. Pretty soon you’ll start noticing what makes the number go up, and you’ll start to do more of that. Corollary: be careful what you measure.

It had been meshing with my thinking a lot about visualising valued work and community building. I think the same thing applies in both contexts, if you display each user’s post count under their username, then people are going to start posting a lot. If you implement a karma score, then people will try and do things that maximize karma. Not only exposing information about valued work important, not exposing information can also be an important design strategy. Unfortunately, for most online community organisers, there’s little thought about the effects of a feature, features are often turned on just because they’re there.

But chewing over this a little bit before going to sleep last night, I realised that inventives can be thought of as a context free instantiation of values. To have values means that you believe that certain actions are either right or wrong stemming from some base moral reasoning. What incentives do is it replaced that base moral reasoning with a much simpler system of rewards. In other words, you switch from doing something because it’s right to doing it because it’s good for you.

Because incentives are context free, they’re much easier to scale, both up and out. Every person you meet has their own rich tapestry of past experiences and beliefs which influence their value system but every person is going to get exactly +1 to their post count when they make a post.

However, the downside of an incentive system is that they never incentivize precisely what you want to reward and, whenever you get humans into the mix, you’re going to get gaming of the system. Incentives are also inflexible in the face of novelty compared to values because values stem from the motivation rather than the result of reasoning.

Values are more effective but harder to implement, incentives are easier but less subtle. One of the unique advantages that startups have is that they’re still small enough that they can make the effort to instill a strong value system and this is one of their unique competitive advantages. Some companies nowadays are starting to get the picture and have come out full force in the expression of their values but too many still try and ape large companies and hide behind a bland moral ambiguity. More startups need to realise their true values are a massive asset, not a liability and that they won’t have the luxury of having them for much longer.