Log in

No account? Create an account

Previous Entry | Next Entry

Errors vs. Bugs and the End of Stupidity

"A pianist has to believe in telekinesis.  You have to believe you have the power to move your fingers with your mind."

I learned that from Phil Cohn, my piano teacher's piano teacher.  Once in a while, when I was in high school, she'd arrange for me to have a master class with him.  He was a diminutive man who looked exactly like Dr. Strangelove, and had a gentle way of guiding your hands and body while you played.  He was very interested in the physicality of piano; he liked to tell stories about students of his who could play any piece upside down, or cross-handed, or one-fingered like Chico Marx.  He had a lot of theories about the process by which we can learn to control our muscle movements.

I wasn't an exceptional pianist, and when I'd play my nocturne for him, there would be a few clinkers.  I apologized -- I was embarrassed to be wasting his time.  But he never seem to judge me for my mistakes.  Instead, he'd try to fix them with me: repeating a three-note phrase, differently each time, trying to get me to unlearn a hand position or habitual movement pattern that was systematically sending my fingers to wrong notes.

I had never thought about wrong notes that way.  I had thought that wrong notes came from being "bad at piano" or "not practicing hard enough," and if you practiced harder the clinkers would go away.  But that's a myth.

In fact, wrong notes always have a cause. An immediate physical cause.   Just before you play a wrong note, your fingers were in a position that made that wrong note inevitable. Fixing wrong notes isn't about "practicing harder" but about trying to unkink those systematically error-causing fingerings and hand motions.  That's where the "telekinesis" comes in: pretending you can move your fingers with your mind is a kind of mindfulness meditation that can make it easier to unlearn the calcified patterns of movement that cause mistakes.

Remembering that experience, I realized that we really tend to think about mistakes wrong, in the context of music performance but also in the context of academic performance.

A common mental model for performance is what I'll call the "error model."  In the error model, a person's performance of a musical piece (or performance on a test) is a perfect performance plus some random error.  You can literally think of each note, or each answer, as x + c*epsilon_i, where x is the correct note/answer, and epsilon_i is a random variable, iid Gaussian or something.  Better performers have a lower error rate c.  Improvement is a matter of lowering your error rate.  This, or something like it, is the model that underlies school grades and test scores. Your grade is based on the percent you get correct.  Your performance is defined by a single continuous parameter, your accuracy.

But we could also consider the "bug model" of errors.  A person taking a test or playing a piece of music is executing a program, a deterministic procedure.  If your program has a bug, then you'll get a whole class of problems wrong, consistently.  Bugs, unlike error rates, can't be quantified along a single axis as less or more severe.  A bug gets everything that it affects wrong.  And fixing bugs doesn't improve your performance in a continuous fashion; you can fix a "little" bug and immediately go from getting everything wrong to everything right.  You can't really describe the accuracy of a buggy program by the percent of questions it gets right; if you ask it to do something different, it could suddenly go from 99% right to 0% right.  You can only define its behavior by isolating what the bug does.

Often, I think mistakes are more like bugs than errors.  My clinkers weren't random; they were in specific places, because I had sub-optimal fingerings in those places.  A kid who gets arithmetic questions wrong usually isn't getting them wrong at random; there's something missing in their understanding, like not getting the difference between multiplication and addition.  Working generically "harder" doesn't fix bugs (though fixing bugs does require work). 

Once you start to think of mistakes as deterministic rather than random, as caused by "bugs" (incorrect understanding or incorrect procedures) rather than random inaccuracy, a curious thing happens.

You stop thinking of people as "stupid."

Tags like "stupid," "bad at ____", "sloppy," and so on, are ways of saying "You're performing badly and I don't know why."  Once you move it to "you're performing badly because you have the wrong fingerings," or "you're performing badly because you don't understand what a limit is," it's no longer a vague personal failing but a causal necessity.  Anyone who never understood limits will flunk calculus.  It's not you, it's the bug.

This also applies to "lazy."  Lazy just means "you're not meeting your obligations and I don't know why."  If it turns out that you've been missing appointments because you don't keep a calendar, then you're not intrinsically "lazy," you were just executing the wrong procedure.  And suddenly you stop wanting to call the person "lazy" when it makes more sense to say they need organizational tools.

"Lazy" and "stupid" and "bad at ____" are terms about the map, not the territory.  Once you understand what causes mistakes, those terms are far less informative than actually describing what's happening. 

These days, learning disabilities are far more highly diagnosed than they used to be. And sometimes I hear the complaint about rich parents, "Suddenly if your kid's getting B's, you have to believe it's a learning disability.  Nobody can accept that their kid is just plain mediocre.  Are there no stupid people left?"  And maybe there's something to the notion that the kid who used to be just "stupid" or "not a great student" is now often labeled "learning disabled." But I want to complicate that a little bit.

Thing is, I've worked with learning disabled kids.  There were kids who had trouble reading, kids who had trouble with math, kids with poor fine motor skills, ADD and autistic kids, you name it.  And these were mostly pretty mild disabilities.  These were the kids who, in decades past, might just have been C students, but whose anxious modern-day parents were sending them to special programs for the learning disabled. 

But what we did with them was nothing especially mysterious or medical.  We just focused, carefully and non-judgmentally, on improving their areas of weakness.  The dyslexics got reading practice.  The math-disabled got worksheets and blocks to count.  Hyperactive kids were taught to ask themselves "How's my motor running today?" and be mindful of their own energy levels and behavior.  The only difference between us and a "regular" school is that when someone was struggling, we tried to figure out why she was struggling and fix the underlying problem, instead of slapping her a bad report card and leaving it at that.

And I have to wonder: is that "special education" or is it just education?

Maybe nobody's actually stupid.  Maybe the distinction between "He's got a learning disability" and "He's just lousy at math" is a false one.  Maybe everybody should think of themselves as having learning disabilities, in the sense that our areas of weakness need to be acknowledged, investigated, paid special attention, and debugged.

This is part of why I think tools like Knewton, while they can be more effective than typical classroom instruction, aren't the whole story.  The data they gather (at least so far) is statistical: how many questions did you get right, in which subjects, with what learning curve over time?  That's important.  It allows them to do things that classroom teachers can't always do, like estimate when it's optimal to review old material to minimize forgetting.  But it's still designed on the error model. It's not approaching the most important job of teachers, which is to figure out why you're getting things wrong -- what conceptual misunderstanding, or what bad study habit, is behind your problems.  (Sometimes that can be a very hard and interesting problem.  For example: one teacher over many years figured out that the grammar of Black English was causing her students to make conceptual errors in math.)

As a matter of self-improvement, I think it can make sense not to think in terms of "getting better" ("better at piano", "better at math," "better at organizing my time").  How are you going to get better until you figure out what's wrong with what you're already doing?  It's really more an exploratory process -- where is the bug, and what can be done to dislodge it?  Dislodging bugs doesn't look like competition, and sometimes it doesn't even look like work.  Mr. Cohn was gentle and playful -- he wasn't trying to get me to "work harder," but to relax enough to change the mistaken patterns I'd drilled into myself. 


( 38 comments — Leave a comment )
Page 1 of 2
<<[1] [2] >>
Apr. 30th, 2012 03:38 pm (UTC)
you're on fire lately, in terms of insightful posts!

I haven't played piano regularly in a long time, though I took lessons regularly for about 8 years. Never actually enjoyed it until I stopped taking lessons and joined an informal rock band. After reading this post I think part of the problem was the drills and exercises actually forcefully programmed in some buggy behavior. The last teacher I had though (a russian grad student) made an observation that really helped me: after fumbling on the left hand part on Stairway to Heaven, hitting lots of clashing notes, she said [paraphrasing] "why is this so hard for you, the chords are written right there!"
I won't say I never hit a wrong note again, but music suddenly went from a rigid process of pressing the keys when it says to on the paper to a frame of informal rules (the chord structure and rhythm) with other peices fit in (melody, harmonies, counterpoints, etc). This jived completely with my preference for the music of folk tradition and improvisation. This was really solidified when I started playing the accordion, because its a 'schizophrenic' implementation of chords & rhythm and melody/harmonies (at least the way I play!). I am eternally impressed by classically trained musicians but am finally ok that this will never be me... other skills such as playing by ear are much more useful.

One final observation: I have an electric piano at my parent's home and, after a while a few of the keys became broken, such that hitting them makes a loud, obnoxious buzzing noise. Even when playing other pianos I tend to avoid these keys (supporting your telekinetic finger model).

I think there are, in fact, stupid people (or to be politically correct - a variance in intelligence that has genetic and environmental causes) though you're right that framing it in that way is not helpful to anyone - not people making those judgments and definitely not for the people who are supposed to be learning when those judgments are being placed on them (this is well supported by the research)

Edited at 2012-04-30 03:39 pm (UTC)
Apr. 30th, 2012 04:43 pm (UTC)
"I think there are, in fact, stupid people" -- I think, by your definition, I don't disagree. Certainly genetics and environment affect people's capabilities.

But I was actually inspired by reading activist Amanda Baggs' blog (http://ballastexistenz.wordpress.com/). She's autistic and has a number of other disabilities, some cognitive. She can't speak, and she needs a lot of help for household tasks. And yet my impression from reading her was "clearly very intelligent." When she describes being unable to do something (sometimes writing is difficult) she makes it sound more like being overwhelmed by fatigue or overstimulated. And I realized: "Huh. Because she has a specific reason for why she can't do X, I don't think of it as "stupidity," I think of it as fatigue, overstimulation, etc. I only think of an incapacity as 'stupidity' if I don't know the direct cause."
(no subject) - ledflyd - May. 1st, 2012 04:54 am (UTC) - Expand
(no subject) - selki - May. 28th, 2012 04:12 pm (UTC) - Expand
Apr. 30th, 2012 04:15 pm (UTC)
Would you consider making this into a public post? I'd like to share it with others.

(This is not the first time that I've had a "oh, I'd like to link to this post, but it's protected" reaction to one of your posts, BTW, though this was the first time that I asked.)

Edited at 2012-04-30 04:17 pm (UTC)
Apr. 30th, 2012 08:18 pm (UTC)

If you ever find something you want to make public again, let me know.
(no subject) - xuenay - Apr. 30th, 2012 08:48 pm (UTC) - Expand
(no subject) - celandine13 - Apr. 30th, 2012 10:33 pm (UTC) - Expand
(no subject) - maradydd - Jan. 26th, 2013 11:36 am (UTC) - Expand
May. 1st, 2012 12:08 am (UTC)

Okay, you have inspired me to type in a favorite passage from one of my favorite novels, Frank Conroy's Body and Soul. I got done with it and it was so long I felt weird about leaving it here as a comment, so I put it on my LJ.

May. 28th, 2012 04:07 pm (UTC)
Oooh, thanks for that passage! Maybe I should read that book.

Here via firecat.
May. 2nd, 2012 05:03 am (UTC)
(Quick note: according to the confirmation page after my last comment, it was marked as spam - I'm guessing because of the link. Sorry about the inconvenience!)
May. 2nd, 2012 01:48 pm (UTC)
"Sometimes that can be a very hard and interesting problem. For example: one teacher over many years figured out that the grammar of Black English was causing her students to make conceptual errors in math.)"

Why are you blaming Black English and not the teacher's inability to learn the language of her students?

May. 2nd, 2012 04:38 pm (UTC)
Re: Causality
Her argument, not mine. I think the premise was that quantity words are used inconsistently in Black English, but I don't know if that's true.
Re: Causality - nancylebov - May. 9th, 2012 04:45 am (UTC) - Expand
May. 2nd, 2012 02:01 pm (UTC)
I'm really glad you did turn this public (and also that a friend linked me to it). One of the best reads I've had this year.
May. 2nd, 2012 04:08 pm (UTC)
Nice post!

Re: Knewton, I think you are making a false dichotomy. Do you actually know what kind of analysis they are doing? My understanding is that they do what might be called "statistical bug detection" (among other types of analyses). I would say that this is one of the holy grails of psychometrics, which few people can approach because there is a high hurdle in terms of knowledge engineering. e.g. you need a graph to tell you which production rules (whether good or buggy) lead to which kinds of behavior.
Thom Blake
May. 2nd, 2012 09:11 pm (UTC)
A must-read
I'm telling everyone I know to read this. I've been claiming for years that saying "I'm stupid" is one of the biggest causes of stupidity, but I didn't have theory to back it up - now I do. Thanks!
May. 28th, 2012 01:48 pm (UTC)
Excellent. I have written a post of my own about it.
May. 28th, 2012 03:00 pm (UTC)
Here via andrewducker -- superb!
May. 28th, 2012 04:44 pm (UTC)
*makes a note of this sensible essay*
May. 29th, 2012 07:11 pm (UTC)
I think of statistical knowledge as one of the lowest forms of knowledge, what you use to describe a phenomenon when you do not have or cannot trust more specific insights. A Gaussian error is often just a sum of a moderate number of specific effects, most or all of which can be comprehended. Peter Thiel recently made some related observations in his Startups class.

Thinking of my mistakes as bugs has been second nature for me for as long as I can remember; it did not occur to me that this was unusual. I wonder how common this actually is. "Maybe nobody's actually stupid" strikes me as literally false but usefully provocative; a teacher should calibrate the speed of teaching to the student's abilities--so that systematic debugging is possible--and then work with the student on the actual debugging. Ideally, the student learns to and gets in the habit of debugging their own mistakes most of the time.

I wonder about practical considerations. I've seen a lot of pushback against the "small class sizes = good" conventional wisdom, but this approach does seem to require a small class size, and good teachers of course.

Edit: The debugging model has wide application, but there are quite a few educational problems outside its scope. E.g. if someone is performing poorly in a foreign language class, a teacher might be able to suggest a few study habit adjustments, but there's a significant amount of raw memorization that has to occur, and it's often rational for the student to spend that time on something else even if the foreign language class is required. It's also only partially applicable to creative endeavors like English composition and arguably mathematical proofs. Still, it's a big step forward from what we have now.

Edited at 2012-05-30 04:19 am (UTC)
May. 31st, 2012 09:48 am (UTC)
Aug. 3rd, 2012 09:55 pm (UTC)
Tags like "stupid," "bad at ____", "sloppy," and so on, are ways of saying "You're performing badly and I don't know why."

Stupid's also shorthand for wilfully ignorant, as with Douglas Adams' oft-cited example of laughing at boffins not making planes out of the same material as black boxes. A lot of humour relies on stupidity and desperation to belong, and people are to some extent what they do.

Debugging's a great metaphor, particularly taken to some conclusions -- eventually a particular subroutine's reasonably debugged, there may be limitations of the language that can't be worked around, and setting further targets for refactoring can become counter-productive and it's time to move on to other areas. All of which is generally easier to tackle with independent study than in a class environment with thirty students to juggle.
Sijin Joseph
Sep. 24th, 2012 10:53 am (UTC)
Thank you - Very insightful
Thank you for taking the time to write this. This has was one of the most insightful articles I've read in a long time. I had been thinking along these lines for a while now mostly prompted by my efforts to teach Math and Reading to my 5 year old son. It was nice to see everything thought through in your article and helped me to understand my own thoughts.
Page 1 of 2
<<[1] [2] >>
( 38 comments — Leave a comment )