Log in

No account? Create an account

Previous Entry | Next Entry

Errors vs. Bugs and the End of Stupidity

"A pianist has to believe in telekinesis.  You have to believe you have the power to move your fingers with your mind."

I learned that from Phil Cohn, my piano teacher's piano teacher.  Once in a while, when I was in high school, she'd arrange for me to have a master class with him.  He was a diminutive man who looked exactly like Dr. Strangelove, and had a gentle way of guiding your hands and body while you played.  He was very interested in the physicality of piano; he liked to tell stories about students of his who could play any piece upside down, or cross-handed, or one-fingered like Chico Marx.  He had a lot of theories about the process by which we can learn to control our muscle movements.

I wasn't an exceptional pianist, and when I'd play my nocturne for him, there would be a few clinkers.  I apologized -- I was embarrassed to be wasting his time.  But he never seem to judge me for my mistakes.  Instead, he'd try to fix them with me: repeating a three-note phrase, differently each time, trying to get me to unlearn a hand position or habitual movement pattern that was systematically sending my fingers to wrong notes.

I had never thought about wrong notes that way.  I had thought that wrong notes came from being "bad at piano" or "not practicing hard enough," and if you practiced harder the clinkers would go away.  But that's a myth.

In fact, wrong notes always have a cause. An immediate physical cause.   Just before you play a wrong note, your fingers were in a position that made that wrong note inevitable. Fixing wrong notes isn't about "practicing harder" but about trying to unkink those systematically error-causing fingerings and hand motions.  That's where the "telekinesis" comes in: pretending you can move your fingers with your mind is a kind of mindfulness meditation that can make it easier to unlearn the calcified patterns of movement that cause mistakes.

Remembering that experience, I realized that we really tend to think about mistakes wrong, in the context of music performance but also in the context of academic performance.

A common mental model for performance is what I'll call the "error model."  In the error model, a person's performance of a musical piece (or performance on a test) is a perfect performance plus some random error.  You can literally think of each note, or each answer, as x + c*epsilon_i, where x is the correct note/answer, and epsilon_i is a random variable, iid Gaussian or something.  Better performers have a lower error rate c.  Improvement is a matter of lowering your error rate.  This, or something like it, is the model that underlies school grades and test scores. Your grade is based on the percent you get correct.  Your performance is defined by a single continuous parameter, your accuracy.

But we could also consider the "bug model" of errors.  A person taking a test or playing a piece of music is executing a program, a deterministic procedure.  If your program has a bug, then you'll get a whole class of problems wrong, consistently.  Bugs, unlike error rates, can't be quantified along a single axis as less or more severe.  A bug gets everything that it affects wrong.  And fixing bugs doesn't improve your performance in a continuous fashion; you can fix a "little" bug and immediately go from getting everything wrong to everything right.  You can't really describe the accuracy of a buggy program by the percent of questions it gets right; if you ask it to do something different, it could suddenly go from 99% right to 0% right.  You can only define its behavior by isolating what the bug does.

Often, I think mistakes are more like bugs than errors.  My clinkers weren't random; they were in specific places, because I had sub-optimal fingerings in those places.  A kid who gets arithmetic questions wrong usually isn't getting them wrong at random; there's something missing in their understanding, like not getting the difference between multiplication and addition.  Working generically "harder" doesn't fix bugs (though fixing bugs does require work). 

Once you start to think of mistakes as deterministic rather than random, as caused by "bugs" (incorrect understanding or incorrect procedures) rather than random inaccuracy, a curious thing happens.

You stop thinking of people as "stupid."

Tags like "stupid," "bad at ____", "sloppy," and so on, are ways of saying "You're performing badly and I don't know why."  Once you move it to "you're performing badly because you have the wrong fingerings," or "you're performing badly because you don't understand what a limit is," it's no longer a vague personal failing but a causal necessity.  Anyone who never understood limits will flunk calculus.  It's not you, it's the bug.

This also applies to "lazy."  Lazy just means "you're not meeting your obligations and I don't know why."  If it turns out that you've been missing appointments because you don't keep a calendar, then you're not intrinsically "lazy," you were just executing the wrong procedure.  And suddenly you stop wanting to call the person "lazy" when it makes more sense to say they need organizational tools.

"Lazy" and "stupid" and "bad at ____" are terms about the map, not the territory.  Once you understand what causes mistakes, those terms are far less informative than actually describing what's happening. 

These days, learning disabilities are far more highly diagnosed than they used to be. And sometimes I hear the complaint about rich parents, "Suddenly if your kid's getting B's, you have to believe it's a learning disability.  Nobody can accept that their kid is just plain mediocre.  Are there no stupid people left?"  And maybe there's something to the notion that the kid who used to be just "stupid" or "not a great student" is now often labeled "learning disabled." But I want to complicate that a little bit.

Thing is, I've worked with learning disabled kids.  There were kids who had trouble reading, kids who had trouble with math, kids with poor fine motor skills, ADD and autistic kids, you name it.  And these were mostly pretty mild disabilities.  These were the kids who, in decades past, might just have been C students, but whose anxious modern-day parents were sending them to special programs for the learning disabled. 

But what we did with them was nothing especially mysterious or medical.  We just focused, carefully and non-judgmentally, on improving their areas of weakness.  The dyslexics got reading practice.  The math-disabled got worksheets and blocks to count.  Hyperactive kids were taught to ask themselves "How's my motor running today?" and be mindful of their own energy levels and behavior.  The only difference between us and a "regular" school is that when someone was struggling, we tried to figure out why she was struggling and fix the underlying problem, instead of slapping her a bad report card and leaving it at that.

And I have to wonder: is that "special education" or is it just education?

Maybe nobody's actually stupid.  Maybe the distinction between "He's got a learning disability" and "He's just lousy at math" is a false one.  Maybe everybody should think of themselves as having learning disabilities, in the sense that our areas of weakness need to be acknowledged, investigated, paid special attention, and debugged.

This is part of why I think tools like Knewton, while they can be more effective than typical classroom instruction, aren't the whole story.  The data they gather (at least so far) is statistical: how many questions did you get right, in which subjects, with what learning curve over time?  That's important.  It allows them to do things that classroom teachers can't always do, like estimate when it's optimal to review old material to minimize forgetting.  But it's still designed on the error model. It's not approaching the most important job of teachers, which is to figure out why you're getting things wrong -- what conceptual misunderstanding, or what bad study habit, is behind your problems.  (Sometimes that can be a very hard and interesting problem.  For example: one teacher over many years figured out that the grammar of Black English was causing her students to make conceptual errors in math.)

As a matter of self-improvement, I think it can make sense not to think in terms of "getting better" ("better at piano", "better at math," "better at organizing my time").  How are you going to get better until you figure out what's wrong with what you're already doing?  It's really more an exploratory process -- where is the bug, and what can be done to dislodge it?  Dislodging bugs doesn't look like competition, and sometimes it doesn't even look like work.  Mr. Cohn was gentle and playful -- he wasn't trying to get me to "work harder," but to relax enough to change the mistaken patterns I'd drilled into myself. 


( 40 comments — Leave a comment )
Apr. 30th, 2012 03:38 pm (UTC)
you're on fire lately, in terms of insightful posts!

I haven't played piano regularly in a long time, though I took lessons regularly for about 8 years. Never actually enjoyed it until I stopped taking lessons and joined an informal rock band. After reading this post I think part of the problem was the drills and exercises actually forcefully programmed in some buggy behavior. The last teacher I had though (a russian grad student) made an observation that really helped me: after fumbling on the left hand part on Stairway to Heaven, hitting lots of clashing notes, she said [paraphrasing] "why is this so hard for you, the chords are written right there!"
I won't say I never hit a wrong note again, but music suddenly went from a rigid process of pressing the keys when it says to on the paper to a frame of informal rules (the chord structure and rhythm) with other peices fit in (melody, harmonies, counterpoints, etc). This jived completely with my preference for the music of folk tradition and improvisation. This was really solidified when I started playing the accordion, because its a 'schizophrenic' implementation of chords & rhythm and melody/harmonies (at least the way I play!). I am eternally impressed by classically trained musicians but am finally ok that this will never be me... other skills such as playing by ear are much more useful.

One final observation: I have an electric piano at my parent's home and, after a while a few of the keys became broken, such that hitting them makes a loud, obnoxious buzzing noise. Even when playing other pianos I tend to avoid these keys (supporting your telekinetic finger model).

I think there are, in fact, stupid people (or to be politically correct - a variance in intelligence that has genetic and environmental causes) though you're right that framing it in that way is not helpful to anyone - not people making those judgments and definitely not for the people who are supposed to be learning when those judgments are being placed on them (this is well supported by the research)

Edited at 2012-04-30 03:39 pm (UTC)
Apr. 30th, 2012 04:43 pm (UTC)
"I think there are, in fact, stupid people" -- I think, by your definition, I don't disagree. Certainly genetics and environment affect people's capabilities.

But I was actually inspired by reading activist Amanda Baggs' blog (http://ballastexistenz.wordpress.com/). She's autistic and has a number of other disabilities, some cognitive. She can't speak, and she needs a lot of help for household tasks. And yet my impression from reading her was "clearly very intelligent." When she describes being unable to do something (sometimes writing is difficult) she makes it sound more like being overwhelmed by fatigue or overstimulated. And I realized: "Huh. Because she has a specific reason for why she can't do X, I don't think of it as "stupidity," I think of it as fatigue, overstimulation, etc. I only think of an incapacity as 'stupidity' if I don't know the direct cause."
May. 1st, 2012 04:54 am (UTC)
thanks for the read.. fascinating stuff.
May. 28th, 2012 04:12 pm (UTC)
This is a fascinating discussion, and I'd like to share it if I can figure out, are y'all using "schizophrenic" in the deprecated split personality use (now called disassociative identity disorder), or in the current psychological diagnosis sense?
Apr. 30th, 2012 04:15 pm (UTC)
Would you consider making this into a public post? I'd like to share it with others.

(This is not the first time that I've had a "oh, I'd like to link to this post, but it's protected" reaction to one of your posts, BTW, though this was the first time that I asked.)

Edited at 2012-04-30 04:17 pm (UTC)
Apr. 30th, 2012 08:18 pm (UTC)

If you ever find something you want to make public again, let me know.
Apr. 30th, 2012 08:48 pm (UTC)
Thanks! Shared ([1], [2]).

Will do. I think the previous post where I considered asking was when you commented on Girls, but I figured that you might consider that one too personal for it.
Apr. 30th, 2012 10:33 pm (UTC)
No, it's okay. Shared.

This started out as more diary-style, but I've found that my natural writing form is the essay, and I may gradually transition into public blogging again. This LJ doesn't really have a theme, but I figure as life takes shape I may develop common threads that make for writing that's suitable to share.
Jan. 26th, 2013 11:36 am (UTC)
That was the progression that my LJ took. I haven't even logged in in years, but decided to in order to say thanks for this post (found via Less Wrong).
May. 1st, 2012 12:08 am (UTC)

Okay, you have inspired me to type in a favorite passage from one of my favorite novels, Frank Conroy's Body and Soul. I got done with it and it was so long I felt weird about leaving it here as a comment, so I put it on my LJ.

May. 28th, 2012 04:07 pm (UTC)
Oooh, thanks for that passage! Maybe I should read that book.

Here via firecat.
May. 2nd, 2012 05:03 am (UTC)
(Quick note: according to the confirmation page after my last comment, it was marked as spam - I'm guessing because of the link. Sorry about the inconvenience!)
May. 2nd, 2012 01:48 pm (UTC)
"Sometimes that can be a very hard and interesting problem. For example: one teacher over many years figured out that the grammar of Black English was causing her students to make conceptual errors in math.)"

Why are you blaming Black English and not the teacher's inability to learn the language of her students?

May. 2nd, 2012 04:38 pm (UTC)
Re: Causality
Her argument, not mine. I think the premise was that quantity words are used inconsistently in Black English, but I don't know if that's true.
May. 9th, 2012 04:45 am (UTC)
Re: Causality
It might be more fair to say that an unexamined mismatch between black and white English was causing her students to make errors. I don't know whether any other comparable work has been done on figuring out the mismatch.

The book is Twice As Less.

Edited at 2012-05-11 03:38 pm (UTC)
May. 2nd, 2012 02:01 pm (UTC)
I'm really glad you did turn this public (and also that a friend linked me to it). One of the best reads I've had this year.
May. 2nd, 2012 04:08 pm (UTC)
Nice post!

Re: Knewton, I think you are making a false dichotomy. Do you actually know what kind of analysis they are doing? My understanding is that they do what might be called "statistical bug detection" (among other types of analyses). I would say that this is one of the holy grails of psychometrics, which few people can approach because there is a high hurdle in terms of knowledge engineering. e.g. you need a graph to tell you which production rules (whether good or buggy) lead to which kinds of behavior.
Thom Blake
May. 2nd, 2012 09:11 pm (UTC)
A must-read
I'm telling everyone I know to read this. I've been claiming for years that saying "I'm stupid" is one of the biggest causes of stupidity, but I didn't have theory to back it up - now I do. Thanks!
May. 28th, 2012 01:48 pm (UTC)
Excellent. I have written a post of my own about it.
May. 28th, 2012 03:00 pm (UTC)
Here via andrewducker -- superb!
May. 28th, 2012 04:44 pm (UTC)
*makes a note of this sensible essay*
May. 29th, 2012 07:11 pm (UTC)
I think of statistical knowledge as one of the lowest forms of knowledge, what you use to describe a phenomenon when you do not have or cannot trust more specific insights. A Gaussian error is often just a sum of a moderate number of specific effects, most or all of which can be comprehended. Peter Thiel recently made some related observations in his Startups class.

Thinking of my mistakes as bugs has been second nature for me for as long as I can remember; it did not occur to me that this was unusual. I wonder how common this actually is. "Maybe nobody's actually stupid" strikes me as literally false but usefully provocative; a teacher should calibrate the speed of teaching to the student's abilities--so that systematic debugging is possible--and then work with the student on the actual debugging. Ideally, the student learns to and gets in the habit of debugging their own mistakes most of the time.

I wonder about practical considerations. I've seen a lot of pushback against the "small class sizes = good" conventional wisdom, but this approach does seem to require a small class size, and good teachers of course.

Edit: The debugging model has wide application, but there are quite a few educational problems outside its scope. E.g. if someone is performing poorly in a foreign language class, a teacher might be able to suggest a few study habit adjustments, but there's a significant amount of raw memorization that has to occur, and it's often rational for the student to spend that time on something else even if the foreign language class is required. It's also only partially applicable to creative endeavors like English composition and arguably mathematical proofs. Still, it's a big step forward from what we have now.

Edited at 2012-05-30 04:19 am (UTC)
(Deleted comment)
May. 31st, 2012 09:48 am (UTC)
Aug. 3rd, 2012 09:55 pm (UTC)
Tags like "stupid," "bad at ____", "sloppy," and so on, are ways of saying "You're performing badly and I don't know why."

Stupid's also shorthand for wilfully ignorant, as with Douglas Adams' oft-cited example of laughing at boffins not making planes out of the same material as black boxes. A lot of humour relies on stupidity and desperation to belong, and people are to some extent what they do.

Debugging's a great metaphor, particularly taken to some conclusions -- eventually a particular subroutine's reasonably debugged, there may be limitations of the language that can't be worked around, and setting further targets for refactoring can become counter-productive and it's time to move on to other areas. All of which is generally easier to tackle with independent study than in a class environment with thirty students to juggle.
Sijin Joseph
Sep. 24th, 2012 10:53 am (UTC)
Thank you - Very insightful
Thank you for taking the time to write this. This has was one of the most insightful articles I've read in a long time. I had been thinking along these lines for a while now mostly prompted by my efforts to teach Math and Reading to my 5 year old son. It was nice to see everything thought through in your article and helped me to understand my own thoughts.
Sep. 24th, 2012 11:26 am (UTC)

When giving me a new piece to try, my old drum teacher would make me play it reallllly slowly. He did this because, although he didn't know the name of the phenomenon, he knew that once your muscles and brain 'learn' something it's very difficult to unlearn it. Most of the mistakes I made as a young drum player were due to my muscles "just doing it" even though my brain said other wise, because I'd previously made mistakes whilst rushing through things that then became set in stone.

Lesson: Take it slow and learn it correctly the first time. This applies to everything from drums to politics.
Jorge Cañizales Díaz
Sep. 24th, 2012 05:58 pm (UTC)
Please keep sharing your thoughts.
That was a very very interesting read. I found it because someone submitted it to www.reddit.com/r/programming .
Sep. 24th, 2012 06:50 pm (UTC)
This is a good article, but it would be better if it didn't use "schizophrenia", which is an actual condition that affects some people, as a metaphor for something else. I don't have schizophrenia, but I am bipolar, and I feel othered when people describe themselves as "a little bipolar about bagels" (or whatever) because they like bagels sometimes and dislike them at other times. To them, it's just a metaphor, but to me, it's my life, which they are taking and trivializing for their own point.

I realize that this is part of a quote, but I think you could use the quote while also bracketing the problematic nature of using disabled people's lives as metaphors to prove points that aren't about disability (at least, if you think it's problematic).

I'd like to be able to link this, but I don't feel like I can do so while it begins in an ableist way. Which is too bad, since I think the main point (that it's more useful to think about debugging one's issues than to think of oneself as irreparably "stupid") is an anti-ableist one.
Oct. 3rd, 2012 08:59 pm (UTC)
Ok, good point. It was a distracting word and there's really no reason for me to trivialize real problems over a metaphor.

Oct. 3rd, 2012 09:02 pm (UTC)
Ken Catchpole
Sep. 27th, 2012 01:47 am (UTC)
Human Error Theory

This is a really great post.

There's a whole science behind why accidents happen that is precisely about this; about understanding what trips you up every day, and rather than "trying harder" working out what goes wrong and building a better way of doing it.

I work with Doctors on precisely this sort of thing, because when things go really wrong, it ends up with the death of the patient and possibly the end of the Doctors career, even though they may have been set up to fail by their training, the equipment they use, or a whole range of other things.

I've just spent the day analysing 1800 very small errors associated with about 90 trauma care patients, looking at trends and common causes. We know that little problems can build to bigger ones, and that telling the Docs to do better won't work - but giving them better information, better communication tools and skills, and new methods to do the same things really does make a difference to saving lives.

A good place to start with the academic side of this is a 1990 book by James Reason called "Human Error".
Keith Albion Borgholthaus
Sep. 27th, 2012 04:09 am (UTC)
A bug in your theory
I got here from one of your friends. I actually enjoy your analysis, however I see a bug. What if something is actually a gift and it is just called a disability?

Most people I know work with what is called the left side of the bell jar. Those who have disabilities and they stop the person from being able to function in society are treated while those with gifts are looked at in funny ways and not really analyzed. I believe that many students with disabilities are actually gifted they just haven't figured out how to use it.

My best example is Sherlock Holmes. He has a habit of looking at everything, and getting bored easily. In our minds this is because his intellect is so high of course it would be bored. The instant I change it to ADHD we view him as having a disability. Before you say that he is fictional Holmes is based on the skills and instincts of an investigator at Scotland yard. Sir Doyle simply added adventure to the character.

So, you mention the people you treat and in the same sentence as autism you mention ADHD. It seems so odd to me. The bug I see is that we are placing some very intelligent people on the left side of the bell curve because we haven't studied the right side well enough.

I enjoyed your article and believe you are very close to figuring out the answer, but I saw this bug and hope you can fix the programming for it. It's missing a bracket or something.
Nov. 21st, 2012 11:19 am (UTC)
I found your LJ through Yvain's. I like your comments on his journal and your entries are pretty interesting. If some of your posts are friends-only, it'd be nice to be added.
Feb. 17th, 2013 09:58 pm (UTC)
Thank you!
This was a wonderful application of systems thinking to education and learning, and the music analogy and how your music teacher thought was a fantastic analogy. Really very good insights. These are the kinds of things that I think apply to nearly every field and every person, and if understood, would help make the world a better place. So by writing this, and by extension hopefully helping some people understand it, thank you for making the world better.
Feb. 12th, 2014 12:58 pm (UTC)
As a math professor (who used to take piano lessons, but is now very rusty), I LOVE this entry! I'm tempted to give my students excerpts (with credit, of course). Would that be OK with you?
(Deleted comment)
Feb. 14th, 2014 03:55 pm (UTC)
This really hits the nail on the head. Either that, or I'm succumbing to the tendency I frequently criticise of choosing the model that is metaphorically derived from our most advanced technology ;-)

BTW, with regarding to learning music, recorder teacher had an alternative approach. He said to me "You're practising like an amateur, not a professional. An amateur musician plays the piece, then tries to correct their mistakes. A professional musician plays the piece so slowly that there are no mistakes, then gradually increases their speed one BPM at a a time." I suppose that would be not so much debugging as making sure bugs don't creep in in the first place.
R. J. Zimmerman
Nov. 13th, 2014 05:28 pm (UTC)
What a pleasure to read.
Dec. 21st, 2014 10:37 pm (UTC)
I wish I could share this a bazillion times. Thank you.
( 40 comments — Leave a comment )