As someone who designs exams, Emma Jones weighs in on the classic statement.
I hear this all the time. As a Chief Examiner, I design and develop approaches to assess individuals to determine if they meet a required standard.
The approach to assessment depends on what is being assessed and to what standard. Obviously, the assessment and standards expected of a brain surgeon will differ greatly from that of a project manager.
What is being assessed?
According to revised Bloom’s taxonomy, within the cognitive domain (knowledge-based), there are six levels of objective:
Having determined the learning outcomes, and written a definitive syllabus, there are numerous approaches to assess the same thing.
For example, the following learning outcome - pertaining to the Praxis Framework™ Certification, could be assessed via a written exam question or a multiple choice exam question, but the required knowledge is the same in either construct.
Learning outcome: Identify the expansion of the triple constraint within the Praxis Framework.
As a written exam, the test may state:
In addition to the triple constraint of scope, time and cost, identify the other three components of delivery within Praxis.
As a multiple-choice exam, the test may state:
In addition to the triple constraint of scope, time and cost, what other three fundamental components of delivery does Praxis identify?
a) Productivity, Pace and Process
b) Resource, Risk and Change
c) Change, Configuration and Contingency
d) People, Processes and Practices
Is it easier to answer in multiple-choice format?
You might argue that someone could guess correctly in multiple-choice format, but that is also true in written format.
Both formats test the same learning objective – can the candidate identify the expansion of the triple constraint in the Praxis Framework?
The test is not whether the candidate can physically write their response (or that the examiner can interpret the candidate’s handwriting!).
OK, so that’s a simple example at the basic level of recall.
You may then argue that ‘Multiple-choice exams can’t assess application of understanding in the same way that a written exam can.’
So let’s look at an example at the application level.
Learning outcome: Able to identify an appropriate P3 management team structure for a given P3 scenario.
In any objective test, the information provided to the candidate should enable them to derive the required answer and demonstrate application of the understanding being assessed. This should not be a reading test, in that the answers are not provided in the given scenario. The candidate is expected to derive their answers by applying their understanding of the roles and responsibilities in a P3 management team structure.
As a written exam, the test may state:
Identify an appropriate organization structure for the given project scenario, explaining why the individuals you have selected are appropriate for the role(s) you have allocated to them.
As a multiple-choice exam, there may be numerous questions addressing the learning outcome. One test may state:
Each of the following questions includes only true statements about an individual from the E-commerce Initiative P3 organization. Only one statement explains why, in the context of a Praxis organization structure, roles and responsibilities, the individual is appropriate for that role in the P3 organization.
| 1 |
Which statement explains why the Chief Operating Officer would be appropriate as the Sponsor?
A. He knows the organization very well having worked there for over 25 years. B. He is a Board member with overall responsibility for Brook Bicycle's IT, Sales and Marketing, and Operations departments. C. He has experience of managing numerous projects and programmes. D. He is retiring in six months and has time to commit to the initiative. |
As with the recall example, the same understanding is being assessed in both question formats.
For a written exam, the marking scheme would identify the appropriate individuals for each role, along with the rationale for their selection (and the examiner would have to try and find the key elements of understanding in the candidates submission).
Is it easier to answer in multiple-choice format? Again, you might argue that someone could guess correctly in multiple-choice format, but that is also true in written format.
Both formats test the same learning objective – can the candidate identify an appropriate organization structure for a given scenario?
What is the expected standard?
Standard setting is arguably one of the most important aspects of test development.
Standard setting has been defined as ‘the task of deriving levels of performance on educational or professional assessments, by which decisions or classifications of persons (and corresponding inferences) will be made’ (Cizek, 1993, cited in Cizek, 2001, p.3).
In essence, the process is about encapsulating the kind of performance or ability sufficient for someone to be considered as a passing or minimally competent candidate, and mapping that on to the outcomes of the examination.
A number of approaches to standard setting have been developed over the years, including those by Angoff, Ebel, and Nedelsky.
It is important to note that all of these are variations on the same theme: attempting to conceptualize the performance of a candidate on the borderline between two groups (e.g. pass and fail).
Cizek articulates this well as ‘the point at which the best performance that still deserves to fail becomes the worst performance that still deserves to pass’.
Provided we have correctly established the correct pass mark to reflect the required standard, multiple-choice exams should not be easier than written exams.
I think the issue is actually around the perception of ‘acceptable pass rates’, which is commercially or culturally driven rather than an agreed standard driving the pass mark.
