• Presentations
• Photos

Timelines

• IALS: A Timeline
• Major Publications on IALS

Think Papers

• IALS and Writing in the Workplace, by John Vorhaus. Describes a new study on the real life tasks and practices of writing as observed among small businesses in England

• Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for  Policy Development, by Stephen Reder. A look at questions of validity, program impact and policy development related to the IALS proficiency measures

Country Stories

For the Institute we asked some of our participants to write research pieces about the experiences of their countries with IALS and its successors.

• A Slow Burn: The Impact of IALS in Australia, by Juliette Mendelovits
• An Examination of IALS and its Effect on Canada, by Kjell Rubenson
• A Tale of IALS’s Influence (or not) in the UK, by Greg Brooks
• IALS in Norway, by Egil Gabrielsen
• Groundhog Day? The 2009 re-run of the 1996 IALS in Scotland, by Ralf St. Clair
• IALS in the US: A Rippling Impact on Literacy Policy, by Ron Pugsley

The following questions were posed by participants at the ends of Days 1 and 2.   Given their technical nature, we asked Marylou Lennon and Kentaro Yamamoto from Educational Testing Service (ETS) to offer answers.  We thank them both.

The cut-off points for the IALS levels represented critical, shifting points – what are those skills and what makes them critical?   What are the thresholds that these levels mark?

The five (5) levels used for reporting IALS results, with the cut-offs at particular points along the scale, were determined based on a combination of expert judgment and research.  The research focused on understanding what factors seemed to distinguish the difficulty of tasks along each scale and what common characteristics seemed to be shared among tasks that fell along the same points on the scale.  Kentaro Yamamoto noted in his presentation at the IALS Institute that this research focused on the features of the tasks as well as on the materials and the level of processing required to successfully complete the tasks.  This is different from readability formulas which attempt to characterize the overall complexity of an entire text or large segments of texts. The approach in the literacy surveys recognizes that not all tasks require an individual to read and fully understand an entire text.  Therefore, the purpose for reading plays an important role in contributing to the overall difficulty of a task.  In other words, a task in the survey represents the interaction between the question that is asked (the “purpose”) and the text that must be read and processed.

For more information:  See a paper written by Irwin Kirsch, entitled “The International Adult Literacy Survey: Understanding What was Measured”  at www.ets.org/Media/Research/pdf/RR-01-25-Kirsch.pdf.   It explains these variables in detail.

Additional explanations are also included in Kirsch, I. S., Jungeblut, A., & Mosenthal, P. (1998). The measurement of adult literacy.  This chapter appears in S. Murray, I. Kirsch, & L. Jenkins (Eds.), Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey. Washington, DC: National Center for Educational Statistics.

How do you test for the progression of the skills?

Testing for the progression of skills, in terms of the large-scale international assessments (IALS, ALL and PIAAC) is done by using the models mentioned in the previous answer and explained in Irwin Kirsch’s paper. Items in these assessments are distributed along each scale as the result of the responses by various populations in each participating country.  Thus, they represent the relative difficulty that individuals with varying abilities have in responding correctly.  This distribution is reflected in the item maps that are in the various reports for these assessments.  Research over many years attempts to explain the progression of tasks along each scale from easy to more difficult based on the task presented in the questions, the characteristics of the stimulus material, and the relationship between the task and stimulus material.  For example, the research shows that a task can be easier or harder depending on a number of characteristics that include:  the structure and characteristics of the text; the type of information requested; the number of conditions that must be satisfied; and, the process that the respondent must use (e.g., must the respondent locate one or more pieces of information, integrate information to make a comparison, or respond based on the information provided plus some outside knowledge).  As we discussed in the Institute session about the PDQ instructional system, if the stimulus is a document, it can be more or less difficult based on its structure (simple list, combined list, etc.)  Prose materials also vary in difficulty based on structural characteristics such as length, organization, and use of headings or other visual cues.  Finally, the relationship between the task and stimulus can make a task more or less difficult based on the match between wording in the question and the stimulus (e.g., whether the question wording exactly matched words or phrases in the stimulus or not).  Again, this is discussed in detail in the papers referenced above, both of which provide a more complete context for this information.

Can you explain the difference between RP67 and RP80? Knowing this will be important to interpret and understand the results of PIACC.

RP stands for response probability.  The scale point assigned to each task is the point at which individuals with that proficiency score have a given probability of responding correctly. In IALS, a response probability of 80% (RP80) was used. This means that individuals estimated to have a particular scale score are expected to perform tasks at that point on the scale correctly with an 80% probability. It also means they will have a greater than 80% chance of performing tasks that are lower on the scale. It does not mean, however, that individuals with given proficiencies can never succeed at tasks with higher difficulty values; they may do so some of the time. It does suggest that their probability of success is “relatively” low—that is, the more difficult the task relative to their proficiency, the lower the likelihood of a correct response.  An analogy might help clarify this. The relationship between task difficulty and individual proficiency is much like the high jump event in track and field, in which an athlete tries to jump over a bar that is placed at increasing heights. Each high jumper has a height at which he or she is proficient—that is, the jumper can clear the bar at that height with a high probability of success, and can clear the bar at lower heights almost every time. When the bar is higher than the athlete’s level of proficiency, however, it is expected that the athlete will be unable to clear the bar consistently.  In this sense the idea of consistency is reflected in the notion of proficiency.

So what is the impact of changing the response probability (RP) from 80% to 67%?

As we decrease the RP value from 80% to 67%, the estimated likelihood that someone with this level of proficiency will succeed on a given task decreases.  This is because we are adding more uncertainty about whether or not we expect individuals at particular points along the scale to respond correctly to items with various characteristics that are defined by each level.   For example, using an RP value of .67, the correct inference would be that someone at the midpoint of a particular level would be expected to get 67 percent of items that are defined by that level correct.  Someone at the bottom of a given level would have less than 67 percent chance of responding correctly to this set of items while someone at the top of a level would have greater than 67 percent chance.  Using an RP value of .80 adds more certainty to estimates.   With an RP value of .80, someone in the middle of a level would be expected to correctly respond to 80 percent of the items drawn randomly from that level.   Someone at the bottom of a level would be expected to score less than 80 percent on that pool of tasks while someone at the top of the level would be expected to score at more than 80 percent on tasks from that level.

It is important to keep in mind that the selection of a response probability comes after estimation of the item parameters and ability.  Thus, the choice of an RP value does not impact either the statistical characteristics of the items or the estimation of proficiency along the scale.  The RP value also does not impact the precision of measurement along a scale .  The same items are used to define the underlying scale regardless of which RP value is selected. The selection of the RP value does impact where a particular item falls along the scale.  As will likely be the case with PIAAC, which is keeping the cut points for levels that were used in IALS (e.g. Level 1 is 0 -225, Level 2 is 226-275, etc), this determines the Level in which some items are  located.  However, the selection of the RP value has no impact on the ability distribution or the percentage of people who fall within a particular level.

How were the words selected for the reading test – (job/sky) passage comprehension that Marylou Lennon described? Are thinking types considered?

For the comprehension section of the Reading Components test, passages were written that varied in difficulty and in text type (narrative, persuasive, expository).  Then words within those passages were selected as targets, or correct choices.  The incorrect choice next to the target was a high-frequency word in the same part of speech – so if the target word was a noun (as in the example) the associated incorrect choice was also a noun.  Research has shown that this type of item is correlated with how well individuals are able to comprehend certain types of texts.  The methodology is more commonly used with prose passages of narrative and expository texts.

Evaluation of digital information – Can you give an example of what is considered entry level?

If this is in reference to the digital materials in PIAAC – the digital texts in the Literacy domain and the digital materials in Problem Solving in Technology Rich Environments – these materials are, by design, intended to measure skills at the higher end of the ability distribution.  Currently, we do not have very simple, or entry level, digital materials in the assessments.  Additionally,  because “evaluation tasks”, on average, are more difficult than basic “locate tasks” or simple “integrate tasks” such as comparing two easy-to-locate pieces of information, applying this more complex cognitive task in the context of using digital materials is most likely to lead to tasks that are quite difficult.  This will be determined empirically using data from the PIAAC Main Survey. If you are interested, you might want to investigate some of the work that Don Leu has done on what is referred to as “new literacies”, or the skills that readers need when using digital information.  In terms of evaluation, one of Leu’s colleagues has identified “at least five different types of evaluation that take place during online reading comprehension:

1) Evaluating understanding: Does it make sense to me?

2) Evaluating relevancy: Does it meet my needs?

3) Evaluating accuracy: Can I verify it with another reliable source?

4) Evaluating reliability: Can I trust it? and

5) Evaluating bias: How does the author shape it?”

For more information, see: http://www.newliteracies.uconn.edu/pub_files/What_is_new_about_new_literacies_of_online_reading.pdf)    ·

Has the fact that we process words on paper differently than on computer been considered?

We have developed two types of tasks for PIAAC.  One type is delivered in a paper-and-pencil mode or a computer-based assessment mode.  For these tasks we have developed adaptation procedures so they can be delivered in either mode with no significant impact on the difficulty of the task.  Data collected so far has demonstrated that these items are, in fact, performing consistently across those two modes.  Other tasks, however, have been specifically developed to reflect the changing nature of how digital texts are constructed and used in various environments.  These items can only be delivered through computer-based assessments. Thus the electronic Literacy items as well as tasks in the Problem-Solving in Technology Rich Environments domain are designed to measure ways in which digital texts put different demands on traditional reading skills.

Is PDQ still available?

The PDQ instructional system is currently not available, although we are investigating how we might be able to disseminate some of the instructional and support materials.  You might want to check some of the work done by Mike Hardt (who attended the Institute) as well as materials developed by SkillPlan (Lynda Fownes attended the Institute)– both are based on the  framework used to develop PDQ.   From time to time, we also discuss internally at ETS whether or not there would be interest in and support for bringing the approach up to date in terms of delivery platform and content.  Egil Gabrielsen noted in his country story that Norway is working with ETS to use PDQ.

John Strucker of World Education reviewed the methodology and results of a study* that compared IALS Level 1 and Level 2 adults to more proficient Level 3 adults in terms of their abilities in reading components (e.g. word decoding, vocabulary, etc.) and whether different patterns of adult strengths and needs in reading could be identified using components assessments. He focused on the implications of this work for literacy assessments such as PIAAC and for the delivery of adult literacy instruction.

see *Strucker, Yamamoto, and Kirsch (2007) “Components and IALS…” http://www.ncsall.net/fileadmin/resources/research/report_29_ials.pdf.

IALS Measurement Validity, Program Impact and Logic Models for Policy Development

Steve Reder of Portland State University discussed research on adult literacy development that suggests how adult basic education programming can best facilitate adult literacy and numeracy growth and increase IALS scores, drawing on  studies of program impact to examine how programs affect adult literacy and numeracy growth. He concluded that programs need to adopt a new logic model to connect participation, literacy growth, and increases in proficiency scores such as IALS.

An overview of an instructional system developed in the late 1980’s by ETS, organized around the literacy frameworks developed for the early large-scale assessments and based on the research findings from those surveys. The early multi-media system integrated sound, video, still images and graphics in lessons and interactive exercises.

Translating IALS into the World of Instructional Practice: Two Examples - Charles Pankratz, Bow Valley College

Two Alberta projects provide examples of adapting IALS to instruction. The first project presents interesting considerations in adapting IALS levels and methodology in the publication of a formative, informal classroom test series called ReadForward. The second project involved translating the IALS levels and approach into the Alberta Reading Benchmarks, a framework for instructors and practitioners

An Overview of IALS-based Research and Tools – Scott Murray, DataAngel Policy Research

• Readings recommended by participants
• Issues identified by participants
• Glossary of survey- related terms

John Vorhaus described a set of reading and writing tests used in an influential study on adult literacy that were led by Professor Greg Brooks. He discussed the tests, the test forms and the administration of tasks, and explains what the students were able to do at each literacy level, noting the relationship between the items used in this study and items used in IALS. A final section discussed the difficulty and importance of also assessing writing.

Findings and Recommendations from NIACE’s National Inquiries into Literacy and Numeracy – Sue Southwood, the National Institute of Adult Continuing Education (NIACE)

Sue Southwood summarized findings and recommendations from NIACE’s National Inquiries into Literacy and Numeracy Learning, and “Skilled for Work”, a survey of the skills of the UK’s largest public sector trade union.

Jaleh Behroozi Soroui talked informally about the significance of participation in the PIAAC for the US adult literacy field, focusing on specific items and issues that have been included in the US PIAAC background questionnaire to provide a deeper demographic picture.

PIAAC in Canada: A brief overview
Presenter: Patrick Bussière, Human Resources and Skills Development Canada (HRSDC)

Patrick Bussière explained what PIAAC will look like in Canada, why so many partners have joined in the effort to make it happen, and HRSDC’s specific interest in PIAAC.

Sue Southwood presented and discussed the findings and recommendations from NIACE’s recent National Inquiries into Literacy and Numeracy Learning, as well as “Skilled for Work”, a survey of the skills of the UK’s largest public sector trade union.

Moving from Assessment to Instruction: Teaching Prose, Document and Quantitative Literacy
Presenter: Marylou Lennon, Educational Testing Service (ETS)

In the late 1980’s, ETS developed an instructional system organized around the literacy frameworks developed for the early large-scale assessments and based on the research findings from those surveys. Designed for use by a teacher with a small group of learners, this multi-media system integrated sound, video, still images and graphics in lessons and interactive exercises. Marylou Lennon will provide an overview of the system and results from pilot studies of the materials.