Judging and Applying the Standards
An important element of the examination procedures is the process of interpreting students’ examination performances in terms of a set of pre-specified course achievement ‘standards’. The achievement standards in each course take the form of a hierarchy of six described levels referred to as ‘bands’. Band 1 represents a level of achievement below the minimum standard expected in the course. Bands 2, 3, 4, 5 and 6 represent increasing levels of course achievement and are described in terms of the kinds of knowledge, skills and understandings typically displayed by students achieving at those levels.
Each year, the standards-setting process is used to determine the mark on that year’s examination corresponding to each of the course achievement standards. For example, the standards-setting process determines the minimum examination mark a student must achieve to be assigned to Band 6. This raw examination mark will not necessarily be the same in different years because examinations differ slightly in difficulty from year to year.
The standards-setting process is conducted for each examination by a group of ‘judges’ drawn from the examination markers for that course. They make judgements about how students at the borderlines between the course achievement standards are likely to perform on each of the examination questions.
The standards-setting process in each course is undertaken by a group of markers who also are appointed as judges for the standards-setting exercise. The selection of judges is made from recommendations provided by the Supervisor of Marking based on applicants’ experience in teaching and marking the course, and on their experience in making judgements against standards.
The outcome of the judging process is a set of five cut-off marks. These are the minimum marks students must achieve on a particular examination to be assigned to Bands 2, 3, 4, 5 and 6 in the course. Once these cut-off marks are established, each student with an examination mark is assigned to one of the six achievement bands.
The judging process involves three stages, giving judges several opportunities to review and refine their earlier decisions.
To inform their decisions, the judges review statistical data showing how students at varying levels of attainment in the course typically performed on each examination question. They also review samples of students’ responses to each question.
Judges are trained for this task and are given a copy of the course band descriptions, a copy of the examination paper and specially prepared recording sheets.
Working independently, the judges study the band descriptions and develop an ‘image’ of the kinds of knowledge and skills characterising students in each band. Having done this, they then develop an image of students at the borderlines between bands.
Still working independently, each judge considers the examination questions/tasks one at a time and judges and records the mark that they believe a student at the borderline between Bands 5 and 6 is likely to receive on that question/task. Each judge’s marks are then added across all examination questions to give a total examination mark corresponding to the borderline (or cut-off) between Band 5 and Band 6 for that judge.
The cut-off marks between Bands 5 and 6 determined by all the judges in the team are then averaged to provide the first estimate of the minimum examination mark required by a student to be assigned to Band 6.
The judges follow these same procedures to determine the cut-offs for the other bands.
The judges then meet to discuss the individual judgements they have made. To assist their discussion they are given a statistical report showing how students at different levels of attainment in the course performed on each question in the examination. The judges work through and discuss this information. These statistical reports are referred to as ‘item-student scales’. Judges also receive a report showing how students responded to the various alternatives of each multiple-choice question. During this process each judge has the opportunity to modify any decisions they made during Stage 1. In this way the team starts to develop a shared image of students at the borderlines between bands.
The judges’ recording sheets are again collected and processed as in Stage 1, resulting in a new set of band cut-off marks.
A sample of student examination scripts with total marks equal to or near each cut-off mark is then retrieved and judges review and discuss these scripts.
Where a student at the borderline has been awarded the mark on a question that judges expected of students at that borderline, the student’s response is shown to judges who are asked to confirm that it is typical of what they would expect of students at that level. During this process judges have the opportunity to further refine their band cut-off marks.
At the completion of this process, judges recommend to the HSC Consultative Committee a set of examination marks corresponding to the cut-offs between bands.
Role of consultative committee
The HSC Consultative Committee is appointed by the Board of Studies to review the distribution of marks in each course. The committee is an expert technical committee made up of leading authorities in educational measurement in NSW.
The role of the Consultative Committee in the Higher School Certificate is to ensure the integrity of the standards-setting process and the integrity of the final decisions. The Committee seeks advice from judges and, where necessary, makes adjustments within the natural variability of judges’ decisions. On behalf of the Board, the Consultative Committee also determines appropriate action if the standards-setting process has not been applied appropriately or if there are other anomalies or problems in the process.
Following the standards-setting process, the Consultative Committee meets and considers the results of the standards setting process for each course. The role of the Committee is to:
- ensure that the standards-setting process followed by each team of judges was in accordance with the Board’s requirements;
- identify any issues that may have impacted on the effectiveness of the standards-setting process;
- receive advice as to what adjustments, if any, should be made to the distributions of marks for optional questions; and
- receive from the judges their recommended band cut-off marks.
The Consultative Committee then:
- determines whether the recommendations submitted by each team of judges are appropriate or whether minor amendments are necessary; and
- approves the final band cut-off marks for each course.
Once the cut-off marks are approved by the HSC Consultative Committee, the cut-off examination mark between Band 5 and Band 6 is assigned a value of 90. The cut-off mark between Band 4 and Band 5 is assigned a value of 80; between Band 3 and Band 4, a value of 70; and so on. Examination marks between the cut-off marks are adjusted in a linear manner.
Adapted from Masters, G.N. (2002) Fair and Meaningful Measures? A review of examination procedures in the NSW Higher School Certificate, p42