Learning from Total Failure: Why Do Impossible Tests Boost Learning? 2017-2021

Hollins, Timothy and Mitchell, Christopher and Wills, Andrew and Seabrooke, Tina (2021). Learning from Total Failure: Why Do Impossible Tests Boost Learning? 2017-2021. [Data Collection]. Colchester, Essex: UK Data Service. 10.5255/UKDA-SN-855137

In education, a test is usually used to measure learning. However, the last decade has seen an explosion of research demonstrating that tests can also dramatically improve learning - the testing effect. Most recently, a surprising discovery has been made that a test can enhance learning even when it is given before the material has been taught. Hence, when students are tested completely unfamiliar material (e.g., foreign language vocabulary), and will inevitably get all the test questions wrong, subsequent learning of that topic is enhanced. This effect has very significant implications for educational settings, and we seek to understand why the effect occurs.
The first demonstrations of the testing effect involved 3 phases. Participants first studied the material (e.g. a text). Next, one group took a test on the material, while a second group simply studied the correct answers. A final test assessed how much learning had taken place. Taking the interim test led to better final performance than restudying the material, and later research showed the effect was further enhanced if initial answers were corrected with feedback.
One possible explanation for the testing effect is that after thinking of a (wrong) answer, people are highly motivated to learn the correct answer. This particular explanation suggests that testing might be helpful even before the first encounter with the to-be-studied material, as has recently been observed. For example, if you were asked to guess the meaning of a rare English word such as "roke" before ever being told its true meaning (mist), then you would be especially good at remembering that meaning on a later test. It is this benefit of initial tests prior to learning (known as test-potentiated learning - TPL) that is the focus of the current proposal.
The Current Project: We will test a number of potential explanations for the effects of initial tests (TPL) in three strands of research. Strands 1 and 2 will use unfamiliar word pairs and face-word pairs. The former are foreign language items (Finnish nouns and their English language meanings), and the latter are unfamiliar faces, and facts associated with those faces (e.g. name/occupation). The Finnish vocabulary is used because it has clear implications for foreign language learning. Also, Finnish words specifically are not similar to English words, which guarantees that the answers to the initial test will be incorrect. Face-name learning has implications for more social and work-place situations. In the final Strand 3, more complex word-based materials (texts and general knowledge) will be used to extend the findings from Strands 1 and 2 to a range of classroom situations. Participants will know nothing about these materials in advance.
In a prototypical experiment using Finnish vocabulary, all trials will start with the presentation of a Finnish word. In the "test" condition, participants will be asked to guess the meaning of the word before being given the true meaning. A "study" condition, in which no guess is made, will serve as the control. It is expected, given previous research in our laboratory, that guessing will enhance memory for the true meaning.
Strand 1 will explore the extent to which initial tests benefit learning precisely because participants make errors, and so they are surprised by the true answer. Strand 2 will look at the extent to which people are more motivated to study, or likely to change their study strategies following a guess. That is, Strand 1 examines potential "low-level" mechanisms (e.g., error correction) of learning whereas Strand 2 looks at more "high-level" strategic processes that might result from being tested. The experiments in Strand 3 will test the generality of the findings from Strands 1 and 2 to more complex tasks such as general knowledge learning. This strand is designed to broaden the scope of more applied research that might be conducted in the future.

Data description (abstract)

The project concerns the effect of an unsuccessful pre-test (effectively a guess), on the subsequent learning of information, relative to studying that information with no-initial guess. The focus of the work has been the development of a theoretical understanding of when pre-testing is or is not beneficial to subsequent learning, with a view to developing applications of the technique to educational practice. Consequently, each experiment compared the effects of studying versus guessing (and receiving feedback to study) on subsequent memory for the material, with each experiment varying in other aspects (e.g. the nature of the material, or nature of the final test). A total of 26 experimental studies have been completed. Thirteen of these experiments have been published in four outputs and for each of these, the relevant data are published in Open Science Framework (OSF) repositories, as detailed below. A further 6 studies form parts of papers that are either under review, or have been reported at conferences (or both). The remaining studies have not yet been output, but are included in manuscripts in preparation. OSF repositories for the unpublished work will be made available upon acceptance for publication.

All data were collected from volunteer participants who were either undergraduates participating for partial course credit, or members of the public who received a small financial payment. Prior to March 2019 all work was completed in person at the University of Plymouth, but thereafter we moved to online testing using Prolific due to the impact of the global pandemic.

Output 1 examines the impact of pre-testing on different aspects of the event, tested through different criterion memory tests across 5 experiments. The main conclusion from this output is that pre-testing boosts availability of targets (measured through recognition), but not cue-target associations (measured through recall, or associative recognition). Output 2 tested potential accounts of the pre-testing effect are that either guessing increases a person’s motivation to know the answer (before it arrives), or that the discrepancy between the guess and the actual answer induces surprise which drives learning. We tested these ideas in two experiments, and found that pre-testing increases self-reported motivation to learn a fact before it is revealed, but not surprise in the answer after it is revealed. Output 3 demonstrated that the differential pattern for recall and recognition reported in output 1 also applies to learning of related- and unrelated word pairs, and so challenges previously accepted theories of the pre-testing effect. Output 4 explored whether learning from a pre-test was related to the magnitude of the error made. Across 3 experiments, participants guessed the meaning of foreign-language words that came from one of two semantic categories. Contrary to some popular learning theories, a greater pre-testing effect was found if the initial error was closer to the target answer, rather than further away. Output 5 examines the impact of pre-testing upon memory for incidental details of the presented answer. Across two experiments we showed that while pre-testing reliably boosted memory for what the answer was, it had no impact on memory for what colour the answer was presented in (Experiment 1), or when it was presented (Experiment 2). Output 6 was a conference presentation of subset of two experiments from a larger set of 7 experiments that are in preparation for a paper submission. Collectively these studies sought to explain the discrepancy previously seen between the effects of pre-testing on recognition and recall (Outputs 1 and 3). These experiments demonstrate that pre-testing is reliably observed when tested by recognition, but for the pattern for recall depends upon the degree of similarity between different study items, a factor previously overlooked in the pre-testing literature. Output 7 followed up the findings reported in Output 2, and explored whether the curiosity elicited by pre-testing is specific to the answer being sought, or represents a generalised state such as increased attention or arousal that will boost memory for incidental material. Two experiments demonstrated that the pre-testing effect is highly specific.

List of outputs and associated OSF repositories where published are available on the Read-me document.

Downloads

Altmetric