by Scott Young
This type of feedback is often the easiest to get, and research shows that even getting this feedback, which lacks a specific message about what you need to improve, can be helpful. In one study, feedback for a task involving visual acuity facilitated learning, even when it was delivered in blocks that were too large to get any meaningful information about which responses were correct and which were incorrect.4 Many projects that wholly lack feedback can easily be changed to get this broad-scale feedback. Eric Barone, for instance, provided a development blog to publish work on his game and solicit feedback from early drafts. It couldn’t provide him with detailed information about what exactly to improve and change, but his simply being immersed in an environment that provided feedback at all was helpful.
Outcome feedback can improve how you learn through a few different mechanisms. One is by providing you with a motivational benchmark against your goal. If your goal is to reach a certain quality of feedback, this feedback can give you updates on your progress. Another is that it can show you the relative merits of different methods you’re trying. When you are progressing rapidly, you can stick to those learning methods and approaches. When progress stalls, you can see what you might be able to change in your current approach. Although outcome feedback isn’t complete, it is often the only kind available and can still have a potent impact on your learning rate.
Informational Feedback: What Are You Doing Wrong?
The next type of feedback is informational feedback. This feedback tells you what you’re doing wrong, but it doesn’t necessarily tell you how to fix it. Speaking a foreign language with a native speaker who doesn’t share a language with you is an exercise in informational feedback. That person’s confused stare when you misuse a word won’t tell you what the correct word is, but it will tell you that you’re getting it wrong. Tristan de Montebello, in addition to the overall assessment of his performance by audience members at the end of a speech, can also get live informational feedback about how it’s going moment to moment. Did that joke work? Is my story boring them? This is something you can spot in the distracted glances or background chatter throughout your speech. Rock’s stand-up experiment is also a type of informational feedback. He can tell when a certain joke lands or doesn’t, based on the reaction of the audience. However, they can’t tell him what to do to make it funnier—he’s the comedian, not them.
This kind of feedback is easy to obtain when you can get real-time access to a feedback source. A computer programmer who gets error messages when her programs don’t compile properly may not have enough knowledge to understand what she’s doing wrong. But as errors increase or diminish, depending on what she does, she can use that signal to fix her problems. Self-provided feedback is also ubiquitous, and in some pursuits it can be almost as good as feedback from others. When painting a picture, you can simply look at it and get a sense of whether your brushstrokes are adding to or detracting from the image you want to convey. Because this kind of feedback often comes from direct interaction with the environment, it often pairs well with the third principle, directness.
Corrective Feedback: How Can You Fix What You’re Doing Wrong?
The best kind of feedback to get is corrective feedback. This is the feedback that shows you not only what you’re doing wrong but how to fix it. This kind of feedback is often available only through a coach, mentor, or teacher. However, sometimes it can be provided automatically if you are using the right study materials. During the MIT Challenge, I did most of my practice by going back and forth between assignments and their solutions, so that when I finished a problem, I was shown not only whether I had gotten it right or wrong but exactly how my answer differed from the correct one. Similarly, flash cards and other forms of active recall provide corrective feedback by showing you the answer to a question after you make your guess.
The educators Maria Araceli Ruiz-Primo and Susan M. Brookhart argue, “The best feedback is informative and usable by the student(s) who receive it. Optimal feedback indicates the difference between the current state and the desired learning state AND helps students to take a step to improve their learning.”5
The main challenge of this kind of feedback is that it typically requires access to a teacher, expert, or mentor who can pinpoint your mistakes and correct them for you. However, sometimes the added edge of having corrective over merely informational feedback can be worth the effort needed to find such people. Tristan de Montebello worked with Michael Gendler to help him with his public speaking performance, and that helped him spot subtle weaknesses in his presentations that would have gone unnoticed by himself or by a less experienced audience member giving broader feedback.
This type of feedback trumps outcome feedback, which can’t indicate what needs improving, and informational feedback, which can indicate what to improve but not how. However, it can also be unreliable. Tristan de Montebello would often get conflicting advice after delivering a speech; some audience members would tell him to slow down, while others said to speed up. This can also be a situation in which paying for a tutor can be useful, because that person can spot the exact nature of your mistake and correct it with less struggle on your part. The self-directed nature of ultralearning shouldn’t convince you that learning is best done as an entirely solitary pursuit.
Further Notes on Types of Feedback
A few things are worth noting here. First, you need to be careful when trying to “upgrade” feedback from a weaker form to a stronger form if it’s not actually possible. To switch from outcome feedback to informational feedback, you need to be able to elicit feedback on a per element basis of what you’re doing. If instead the feedback is being provided as a holistic assessment of everything you’re doing, trying to turn it into informational feedback can backfire. Game designers know to watch out for this, because asking play testers what they don’t like about a game can often return spurious results: for example, they don’t like the color of the character or the background music. The truth is, the players are evaluating the game holistically, so they often can’t offer this kind of feedback. If their responses come from using it as a whole, not from each aspect individually, asking for greater specificity may lead to guesses from those giving feedback.
Similarly, corrective feedback requires a “correct” answer or the response of a recognized expert. If there is no expert or a single correct approach, trying to turn informational feedback into corrective feedback can work against you when the wrong change is suggested as an improvement. De Montebello noted to me that the advice most people gave him wasn’t terribly useful, but the consistency of it was. If his speech elicited wildly different reactions each time, he knew there was still a lot of work to do. When the speech started to get much more consistent comments, he knew he was onto something. This illustrates that ultralearning isn’t simply about maximizing feedback but also knowing when to selectively ignore elements of it to extract the useful information. Understanding the merits of these different types of feedback, as well as the preconditions that make them possible, is a big part of choosing the right strategy for an ultralearning project.
How Quick Should Feedback Be?
An interesting question in the research on feedback is how quick it should be. Should you get immediate information about your mistakes or wait some period of time? In general, research has pointed to immediate feedback being superior in settings outside of the laboratory. James A. Kulik and Chen-Lin C. Kulik review the literature on feedback timing and suggest that “Applied studies using actual classroom quizzes and real learning materials have usually found immediate feedback to be more effective than delay.”6 Expertise researcher K. Anders Ericsson agrees, arguing in favor of immediate feedback when it assists in identifying and correcting mistakes and when it allows one to execute a corrected version of their performance revised in response to the feedback.7
Interestingly, laboratory studies tend to show that delaying the presentation of the correct response along with the original task (delay
ed feedback) is more effective. The simplest explanation of this result is that presenting the question and answer again offers a second, spaced exposure to the information. If this explanation were correct, all it would mean is that that immediate feedback is best paired with delayed review (or further testing) to strengthen your memory compared with a single exposure. I’ll cover more on spacing and how it impacts your memory in the next chapter on retention.
Despite the superficially mixed results on the timing of feedback from the scientific literature, I generally recommend faster feedback. This enables a quicker recognition of mistakes. However, there’s a possible risk that this recommendation might backslide into getting feedback before you’ve tried your best to answer the question or solve the problem at hand. Early studies on feedback timing tended to show a neutral or negative impact of immediate feedback on learning. In those studies, however, experimenters often gave subjects the ability to see the correct answer before subjects had finished filling out the prompt.8 That meant subjects could often copy the correct answer rather than try to retrieve it. Feedback too soon may turn your retrieval practice effectively into passive review, which we already know is less effective for learning. For hard problems, I suggest setting yourself a timer to encourage you to think hard on difficult problems before giving up to look at the correct answer.
How to Improve Your Feedback
By now you see the importance of feedback to your learning efforts. I’ve explained why feedback, especially when delivered to others, can sometimes backfire. I’ve also showed how the three types—outcome, informational, and corrective—have different strengths and the preconditions that need to be in place in order to make them effective. Now I want to focus on some concrete tactics you can apply to get better feedback.
Tactic 1: Noise Cancellation
Anytime you receive feedback, there are going to be both a signal—the useful information you want to process—and noise. Noise is caused by random factors, which you shouldn’t overreact to when trying to improve. Say you’re writing articles that you post online, trying to improve your writing ability. Most of them won’t attract much attention, and when they do, it’s often because of factors outside of your control; for example, just the right person happens to share it, causing it to spill across social networks. The quality of your writing does drive these factors, but there’s enough randomness that you need to be careful not to change your entire approach based on one data point. Noise is a real problem when trying to improve your craft because you need to do far more work to get the same information about how to write well. By modifying and selecting the streams of feedback you pay attention to, you can reduce the noise and get more of the signal.
A noise-cancelling technique used in audio processing is filtering. Sound engineers know that human speech tends to fall within a particular range of frequencies, whereas white noise is all over the spectrum. They can boost the signal, therefore, by amplifying the frequencies that occur in human speech and quieting everything else. One way to do this is to look for proxy signals. These don’t exactly equal success, but they tend to eliminate some of the noisy data. For blog writing, one way to do so would be to use tracking code to figure out what percentage of people read your articles all the way to the end. This doesn’t prove your writing is good, but it’s a lot less noisy than raw traffic data.
Tactic 2: Hitting the Difficulty Sweet Spot
Feedback is information. More information equals more opportunities to learn. A scientific measure of information is based on how easily you can predict what message it will contain. If you know that success is guaranteed, the feedback itself provides no information; you knew it would go well all along. Good feedback does the opposite. It is very hard to predict and thus gives more information each time you receive it.
The main way this impacts your learning is through the difficulty you’re facing. Many people intuitively avoid constant failure, because the feedback it offers isn’t always helpful. However, the opposite problem, of being too successful, is more pervasive. Ultralearners carefully adjust their environment so that they’re not able to predict whether they’ll succeed or fail. If they fail too often, they simplify the problem so they can start noticing when they’re doing things right. If they fail too little, they’ll make the task harder or their standards stricter so that they can distinguish the success of different approaches. Basically, you should try to avoid situations that always make you feel good (or bad) about your performance.
Tactic 3: Metafeedback
Typical feedback is performance assessment: your grade on a quiz tells you something about how well you know the material. However, there’s another type of feedback that’s perhaps even more useful: metafeedback. This kind of feedback isn’t about your performance but about evaluating the overall success of the strategy you’re using to learn.
One important type of metafeedback is your learning rate. This gives you information about how fast you’re learning, or at least how fast you’re improving in one aspect of your skill. Chess players might track their Elo ratings growth. LSAT studiers might track their improvements on mock exams. Language learners might track vocabulary learned or errors made when writing or speaking. There are two ways you can use this tool. One is to decide when you should focus on the strategy you’re already using and when you should experiment with other methods. If your learning rate is slowing to a trickle, that might mean you’re hitting diminishing returns with your current approach and could benefit from different kinds of drills, difficulties, or environments. A second way you can apply metafeedback is by comparing two different study methods to see which works better. During the MIT Challenge, I’d often split up questions from different subtopics before testing myself on an exam and try different approaches side by side. Does it work better to dive straight into trying to answer questions or to spend a little time to try to see that you understand the main concepts first? The only way you can know is to test your own learning rates.
Tactic 4: High-Intensity, Rapid Feedback
Sometimes the easiest way to improve feedback is simply to get a lot more of it a lot more often. This is particularly true when the default mode of learning involves little or infrequent feedback. De Montebello’s strategy of improving public speaking relied largely on getting far more frequent exposure to the stage than most speakers do. Lewis’s language immersion exposes him to information about his pronunciation at a point when most students still haven’t uttered a word. High-intensity, rapid feedback offers informational advantages, but more often the advantage is emotional, too. Fear of receiving feedback can often hold you back more than anything. By throwing yourself into a high-intensity, rapid feedback situation, you may initially feel uncomfortable, but you’ll get over that initial aversion much faster than if you wait months or years before getting feedback.
Being in such a situation also provokes you to engage in learning more aggressively than you might otherwise. Knowing that your work will be evaluated is an incredible motivator to do your best. This motivational angle for committing to high-intensity feedback may end up outweighing the informational advantage it provides.
Beyond Feedback
Receiving feedback isn’t always easy. If you process it as a message about your ego rather than your skills, it’s easy to let a punch become a knockout. Though carefully controlling the feedback environment so it is maximally encouraging may be a tantalizing option, real life rarely affords such an opportunity. Instead, it’s better to get in and take the punches early so that they don’t put you down for the count. Though short-term feedback can be stressful, once you get into the habit of receiving it, it becomes easier to process without overreacting emotionally. Ultralearners use this to their advantage, exposing themselves to massive amounts of feedback so that the noise can be stripped away from the signal.
Feedback and the information it provides, however, is useful only if you remember the lessons it teaches. Forgetting is human nature, so it is not
enough to learn; you also need to make the information stick. This brings us to the next principle of ultralearning, retention, in which we’ll discuss strategies that will ensure the lessons you learn aren’t forgotten.
Chapter X
Principle 7
Retention
Don’t Fill a Leaky Bucket
Memory is the residue of thought.
—Daniel Willingham, cognitive psychologist
In the small Belgian city of Louvain-la-Neuve, Nigel Richards has just won the World Scrabble Championships. On its own, this isn’t too surprising. Richards has won a championship three times before, and both his prowess with the game and his mysterious personality have made him something of a legend in competitive Scrabble circles. This time, however, is different: instead of the original English-language version of the famous crossword game, Richards has won the French World Championship. This is a much harder feat: most English dictionary versions have roughly 200,000 valid word entries; French, with its gendered nouns and adjectives and copious conjugations, has nearly double that with around 386,000 valid word forms.1 To pull off such a feat is quite remarkable, even more so due to one simple fact: Richards doesn’t speak French.
Richards, an engineer born and raised in Christchurch, New Zealand, is an unusual character. With his long beard and retro aviator sunglasses, he looks like a cross between Gandalf and Napoleon Dynamite. His skills at Scrabble, however, are no joke. A late starter to the game, his mother encouraged him to start in his late twenties, saying “Nigel, since you’re no good at words, you won’t be good at this game, but it will keep you occupied.”2 From those inauspicious beginnings Richards has gone on to dominate the competitive Scrabble scene. Some people even argue that he may be the greatest player of all time.