- The process
- Appointment of markers
- Preparation for marking
- Marking schemes and benchmarks
- The marking process
- Review and evaluation
Higher School Certificate examinations are marked each year by several thousand experienced teachers in marking centres across country New South Wales and the Sydney metropolitan area.
Supervisors of Marking for each course or component of a course are appointed to manage the process which is designed to ensure that each student’s performance, project or written response is marked accurately and fairly. The Board has developed a range of quality assurance processes, including the training and briefing of markers, pilot (practice) marking and the monitoring and review of the marking process and its outcomes.
The major stages in the Board’s marking process include the selection and training of markers; the hand-over of the marking guidelines; the development and pilot testing of a marking scheme and benchmark responses; the briefing of markers; and the marking process itself.
Appointment of markers
The Board appoints various categories of markers who play different roles in the marking process. In any given course, the Supervisor of Marking has responsibility for overseeing marking in that course. Under the direction of the Supervisor of Marking, senior markers lead and supervise groups of markers. The appointment of markers generally is based on their level of experience in teaching the course, the recency of that experience and their general teaching experience.
Supervisors of Marking have administrative responsibility for the marking operation for a course or course component. They are responsible for recommending the appointment of markers and senior markers, for developing and implementing a marking plan, for managing the operation of the marking centre, and for ensuring the accurate and timely marking of all examination responses.
Supervisors of Marking are appointed following interview by senior Board officers and usually have extensive experience both as a marker and senior marker.
Assistant Supervisors of Marking may be appointed to assist the Supervisor of Marking in courses with very large candidatures or where other circumstances make this necessary.
Senior Markers are appointed on the basis of their marking experience and demonstrated leadership skills. Their role is to lead and supervise groups of up to ten markers in the marking of one section or question on a paper, depending on the size of the candidature and the nature of the examination. Senior markers are responsible for the briefing and monitoring of markers, check marking, the analysis of group results and the resolution of discrepant markings.
Pilot Markers begin marking earlier than other markers and assist senior markers to establish initial understanding of the marking guidelines and to assess the calibre and range of responses received from students
Preparation for marking
Following their appointment, Supervisors of Marking take part in a range of training and preparation activities.
Most Supervisors of Marking are directly involved in the development of the course marking guidelines as part of the prior development of the examination. All are responsible for signing-off an approval of the marking guidelines as capable of being implemented in the marking centres.
Prior to marking, Supervisors of Marking submit a range of marking management plans for approval. These include:
- a resource plan;
- a selection of markers and senior markers;
- senior marker briefing notes;
- marker briefing notes; and
- mark sheet sign off.
Marking schemes and benchmarks
The next step in the process is the development of a more detailed ‘marking scheme’ that is later used by marking teams. The marking scheme, based on the Examination Committee’s marking guidelines, is developed by the senior markers working with the Supervisor of Marking. While the marking guidelines identify and describe performances at the various mark levels, the marking scheme adds detail and elaborates particular components of anticipated student responses.
As an initial step, senior markers inspect a range of student responses to test the guidelines, and may select illustrative ‘benchmark’ samples of student responses and develop annotated marking schemes if required. In some cases, a marking scheme takes the form of additional notes to accompany the marking guidelines; in other cases the marking schemes take the form of documented ‘benchmark’ performances, especially in practical examinations.
For example, in Visual Arts, each expressive form has a marking scheme that includes photographs of student work at each mark range, and a detailed description of the elements that have contributed to the award of those marks.
In written examinations, benchmark scripts are sometimes provided as examples. In other courses, such as Software Design and Development, benchmark scripts are incorporated into a booklet of information with notes on the elements of the marking guidelines demonstrated in each of the benchmark examples. This booklet forms the marking scheme provided to markers.
Pilot testing of marking guidelines
Pilot testing occurs concurrently with the finalisation of guidelines, marking schemes and associated materials. In this process senior markers test the guidelines and scheme against a representative range of responses. This pilot testing provides an initial test and assists in refining the selection of benchmark scripts. In some courses senior markers are assisted by particularly experienced markers brought in specifically for this process.
Alterations to marking guidelines
It is anticipated that the pilot marking process will lead to elaboration of the marking scheme and the identification of a broader range of examples to illustrate the marking scheme. Pilot marking also may result in proposed changes to the marking guidelines. Alterations to marking guidelines proposed by the Supervisor of Marking as a result of pilot marking are endorsed by the Chief Examiner. Changes are made to the master copy submitted to the Office of the Board following marking.
The marking process
Following development of the marking scheme and pilot testing of the marking guidelines and related materials, the marking process itself commences. Markers are first briefed and various procedures are then used to monitor the progress of the marking.
During the briefing process, markers are introduced to the marking guidelines, detailed marking schemes (where they are necessary) and ‘benchmark’ responses at a range of performance levels described in the marking guidelines. The materials are discussed at length to ensure that all markers have a shared understanding of the requirements of each performance level.
The Board provides briefing notes for all markers and senior markers covering all aspects of the setting and marking process.
Pilot (practice) marking
In this process, markers apply the guidelines and other materials to a range of responses drawn from a number of representative examination centres. Typically, markers discuss in small groups the marks they have awarded. This discussion provides a further opportunity to achieve consensus and to consider responses that test the breadth of the marking guidelines. Pilot marking continues until senior markers are confident that the guidelines are being applied accurately and consistently. This activity sometimes is referred to as ‘pencil marking’ because the marks awarded in this process are not final marks, and responses are returned to the pool for marking at a later time.
As a result of pilot marking, changes may be made to marking guidelines to ensure that they adequately capture evidence of the outcomes and content that individual questions are designed to assess.
Once marking has commenced, a proportion of the scripts marked by each marker is passed to a senior marker to check mark. The senior marker marks each student’s response and checks that the mark awarded by the marker is accurate. If a pattern emerges showing that marking guidelines are not being applied correctly by a particular marker or group of markers, then re-briefing occurs to redress this anomaly.
Check marking generally begins with a high proportion of scripts being check marked. As consistency and confidence develop, the proportion of check marking is reduced.
Statistical reports also may identify a need to focus check marking on a particular marker. Check marking is subject to audit by the Supervisor of Marking.
Marking common (control) scripts
Supervisors of Marking and senior markers also use common (control) scripts to identify markers who are not applying the marking guidelines or scheme consistently.
The use of common scripts can vary from one subject to another. Generally a script from each question being marked within the centre is photocopied and distributed to all markers of that question at least once per marking session.
Results on common scripts are compared with previous sessions and group data to identify patterns in the marking process and to either confirm satisfactory marking patterns or to identify inconsistencies. Markers showing inconsistency or lack of appropriate application of the marking guidelines receive individual attention to redress the problem.
Statistical checks form part of the Board’s quality assurance processes for marking. Marks awarded by each marker are tallied and processed at regular intervals. A report identifying markers who are marking significantly above or below the average for the marker group is provided to the Supervisor of Marking. While this report may simply reflect the standard of a particular group of responses, it also can point to a marker who is unusually lenient or unusually harsh. Early identification ensures that re-briefing can be given where necessary.
Statistical reports also identify markers using an unusually narrow range of marks. Again, this may simply indicate a group of average students’ responses, or it may indicate conservatism or lack of confidence on the part of a particular marker.
A report on the rate of marking for each member of the marker group is used to adjust the allocation of resources within a marking centre, and allows the Supervisor of Marking to monitor the overall progress of the marking operation.
Supervisors of Marking use statistical reports to highlight perceived anomalies between the anticipated target range of marks (as indicated by the Examination Committee’s mapping grid) and the spread of marks resulting from the application of the marking guidelines.
These various statistical reports can be used by the Supervisor of Marking to identify areas that need finetuning. Their use throughout marking provides a monitoring mechanism for the Supervisor of Marking that contributes to the quality assurance of the marking process.
In subjects in which students provide responses that must be judged subjectively, the Board uses a system of double marking in which two markers make independent judgements of a student’s response. Each marker allocates a mark in accordance with the approved marking guidelines, and is unaware of the other marker’s judgement.
Double marking usually is applied to questions requiring an extended response (eg, essays, creative writing, projects and performances) and not to short answer questions of the kind commonly used in mathematics and the sciences.
When the marks assigned to a double-marked question differ by more than the maximum acceptable difference set by the Board, this difference is considered a ‘discrepancy’ and a third or possibly fourth marking of the student’s response is undertaken. The Supervisor of Marking or another senior marker then takes all these independent markings into consideration and uses professional judgement to determine the most appropriate mark for the student’s response.
Resolution of atypical responses
Atypical responses include answers that are outside the range of answers described by the marking guidelines, but which appear to be valid alternative interpretations of a question. They also include non-serious and offensive responses. Supervisors of Marking are required to have strategies in place to deal with atypical responses, all of which must be brought to the attention of a senior marker.
Valid alternative responses are discussed with the senior marker (and Supervisor of Marking if required). Senior markers and the Supervisor of Marking together determine and allocate a mark based on the quality of the response.
In the case of a non-serious attempt, senior markers may allocate a mark of zero, or may refer the matter to the Supervisor of Marking. Significant concerns are referred to the Director, Assessment and Curriculum for consideration. The Board’s Examination Rules Committee considers cases of alleged breach of examination rules and malpractice.
Responses that exceed stated parameters
In some examinations, particularly projects and performances, there are limits placed on students’ work. This limit may be a limit on the size, time or word length of the work. Where a student’s response exceeds the stated parameters, a mark penalty may be imposed.
In written examinations, some questions indicate a limit on the length of the response. Overly long responses generally are not penalised because the student is considered to have already imposed a limit on the time available to answer other questions.
In subjects in which single marking applies (due to the objective nature of the questions), revision marking is used to provide further quality assurance.
At the end of the marking process, a report is run to identify students whose total examination marks are significantly different from their school assessment marks. The responses of these students are extracted and a group of senior or other experienced markers is recalled to re-mark these students' papers. This process confirms or alters the marks received by students.
Review and evaluation
Following the completion of each year’s marking, an evaluation is undertaken of various aspects of the marking process, and reports are provided to the Director, Curriculum and Assessment. Evaluations are conducted of:
- the training of markers;
- the management of the marking;
- examination paper issues;
- marking guideline issues;
- quality control processes in marking;
- the marker reliability operation;
- marking logistics;
- marking resource plans;
- marker appointments; and
- the handling of special cases.