Examinations 2010 - can we separate fact from fiction?

* KATHLEEN LETSHABO | Friday November 19, 2010 00:00

A seemingly benign and yet controversial use of these types of examinations is that they yield information which is used to pick winners from losers, thereby earning themselves a description of 'highstakes'. 

Highstakes examinations have important consequences for examinees, the most important consequence being that the opportunity to participate in the next cycle of education is extended to some learners, while such opportunity is denied others.

In our case the transition from Form 3 to Form 4, or to gain admission into UB and other institutions of Higher Learning depends on the examinees' performance in the JCE and the BGCSE.  This is the 'validity' of examinations central to any assessment activity.  Loosely speaking, validity speaks to the extent to which results of individuals reflect their real performance, and not error on the part of those who are trying to measure that performance.

Because of the high stakes the nature of most educational testing and the consequences of using test information in peoples' lives, their professional conduct is highly guarded.  It is regulated by laws and standards.

The most commonly used instruments are Standards for Educational and Psychological Testing, Code of Fair Testing Practice in Education, Code of Professional Responsibilities in Educational Measurement.  These were jointly produced by three professional bodies, namely the American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME).

As professional standards and ethics go, these have become industry standard, with adaptations and/or domestication found in examinations bodies worldwide.  Just to illustrate the type of issues that these standards address, the Code of Professional Responsibilities is predicated on seven responsibilities of assessment practitioners, to:

* protect the health and safety of all examinees; * be knowledgeable about, and behave in compliance with, state and federal laws relevant to the conduct of professional activities;

* maintain and improve their professional competence in educational assessment; * provide assessment services only in areas of their competence and experience, affording full disclosure of their professional qualifications;

* promote the understanding of sound assessment practices in education;* adhere to the highest standards of conduct and promote professionally responsible conduct within educational institutions and agencies that provide educational services; and,

* perform all professional responsibilities with honesty, integrity, due care, and fairness.

The Code addresses competence of those who develop, select and administer educational assessments, as well as test bias, conflict of interest, due process, among others.  The different sections of the Code enumerates professional responsibilities of person who work in eight major areas of assessment activity, namely:* Those who develop assessments * Those who market and sell assessments* Those who select assessments * Those who administer assessments * Those who score assessments * Those who interpret, use, and  communicate assessment results * Those who educate others about assessment * Those who evaluate programmes and conduct research on assessments So, what is the point in telling you all this? Botswana National Examinations, 2010, has been the most bizarre examination cycle to date. 

There have been serious allegations and counter-allegations, the most serious being allegations of examinee abuse, lapses in exam security, and invigilation of examinations being conducted by people who are not professionally fit to discharge that mandate.

Whether or not invigilators any of these occured should not be a matter of personal opinion, and if I can paraphrase the words famously attributed to US Senator, Daniel Patrick Moynihan, while everyone may be entitled to their opinion about these exams, no one is entitled to their own facts!  So, what are the facts about the conduct ofAExaminations 2010?

Botswana Examination Council (BEC) derives its mandate from the BEC Act No 11 of 2002.  The Council, established under Section 3 of the BEC Act, is empowered to 'conduct school examinations and any other examinations for the Ministry of Education and issue certificates in respect of the examinations (Article 5.1)'.

The Council is also an oversight body, empowered by Article 5.2(g) to 'maintain standards in the system of examinations'.  Since the law is clear on what the Council has to do, I anticipate that it has instituted standards and/or codes similar to the ones presented above, to guide the conduct of examinations, and to regulate conduct of their employees.

Assuming that those standards do exist, I have no doubt they will have been distilled from the best thinking in educational testing professional practice, and cover similar issues as are raised in the Code above.

The BEC Act also gives the Minister of Education powers to appoint individuals into the Council.

The men and women that constitute the Council are appointed, presumably, because they have the necessary information to provide oversight in the running of examinations, or at the very least, to ask the right questions of those who execute day-to-day duties of the Council.

I have a lot of questions on my mind about the Examinations 2010, but if was a Council member I would ask whether or not BEC has acted in the 'best interest of every child' who sat for exams and afforded her/him the maximum opportunity to succeed.

I would further ask whether the integrity of the examinations was preserved, and whether the public should trust selection decisions that will be made using information from the 2010 examinations.

First, on whether or not best interest of every child' was taken into consideration.  Because of their high stakes nature, the PSLE, JCE and BGCSE are stressors for all candidates, even those candidates who have done the necessary preparations.

Hence receiving media reports that children were reprimanded, let alone assaulted, is the last thing you want to hear in an exam situation. If it is true that some invigilator did assault examinees, then the responsibility to 'protect the health and safety of all examinees' will not have been communicated to such invigilators.

To the extent that the offending invigilators were ill-prepared for the task, those who have ultimate accountability for appointing and preparing invigilators should be held accountable.

We also received media reports that in one instance a group of examinees was reduced to tears as they came out of an examination room where they were supposed to write a listening comprehension paper that was broadcast to them in a manner where they could not make out what was said, let alone comprehend it.

How then, can we expect optimal performance from children who are subjected to such conditions?

Second, whether or not the integrity of the examinations was preserved depends partly on hiring people that are qualified to do the job.

Section 4.2 of the Code above states that individuals should 'administer only those assessments for which they are qualified by education, training, licensure, or certification'. 

Similar standards from BEC can be used to judge the appropriateness of individuals who were appointed as invigilators.

But using this standard, a lot of individuals who are trained educators do qualify, even though they might not be practicing teachers. Other than that, I am unable to think of another category of individuals who are 'qualified by education' to administer assessments.

The third question is whether or not the public should trust the decisions that will be made using information from the 2010 examinations.  We have already mentioned reports that in some instances children's emotional states were not such that you can expect optimal performance from them.

This examination cycle was awash with revelations of invigilators distributing one paper, only to have to substitute it with another; a revelation from the Minister's facebook page of one paper having leaked, only for a BEC official to claim ignorance of the leak; examinees prepared and expecting to write one paper, only for the exam to be cancelled for one reason or the other. 

In all these instances, substitute papers had to be written, and it is not beyond the realm of possibility that examiners were brought in to set new parallel papers.

Many examinees, parents of examinees, and member of the public are grappling with these issues.  So are some members of the Botswana Examination Council.

They need to have their fears allayed.  They need to hear that the system works.  They need to hear that the system works for all children. 

An admission that there were a few difficulties with the PSLE is not good enough, especially when such an admission is not accompanied by full disclosure of the nature of the problem, and how it was resolved.  Even if that problem affected one child that is one child too many.

All the questions raised so far speak to the validity of the examinations, and to the voracity of claims to be made about success or failure of learners. I am not arguing that the assessments are necessarily or decidedly invalid because of these occurrences.  Rather, I am proffering an argument that information about validity checks and corrections has to be shared with the public.

So, if I was a Council member I would propose a public enquiry into the controversy surrounding the examinations, to separate fact from fiction, and to take corrective action, not excluding holding individuals accountable for their decisions if can be found that they misdirected themselves.

This is mainly because the stakes are high for young Batswana learners who have legitimate hopes of having successful academic/educational careers. 

So, not only is my request for an enquiry reasonable, it is the sensible thing to do.  It will demonstrate the Council takes accountabilities seriously - their own accountability and that of their employees.

Lastly, in the interest of full disclosure, let me mention that closer to the time of the inception of BEC, I was considered a possible candidate for the management committee that later became the first Council.  Apparently my candidature was torpedoed by my politics.

The then appointing authority did not believe that someone could dabble in opposition politics and still have the ability to offer sound professional service and/or advice in the Council.  Obviously I respect that decision now, as I did then. 

But just in case anyone wants to ascribe a sinister motive for my engagement with the issue because of this little bit of history - my response is simply that as an educational measurement professional, I have a responsibility to educate others about assessment, including providing 'fair and balanced perspectives on assessment issues being discussed by policymakers, parents, and other citizens' (Section 7.9 of the Code). 

That is to say that mine is not a lay opinion - it is not the kind of social commentary that I proffer from time-to-time on different subjects. 

It is a professional opinion, borne out of solid academic preparation in the area of educational testing and measurement; an internship at the highly regarded Education Testing Service (ETS) in Princeton, New Jersey; practice and experience as an examiner and testing officer in Botswana; experience working as a consultant in educational assessment in a number of countries in Africa; and, membership of professional bodies such as above-mentioned NCME. 

All of this was made possible by education paid for by Botswana taxpayers.

So yes, I deliberately choose not to inject myself in labour relations end of this debate.  Rather, I challenge BEC to institute an enquiry into their own procedures and practices in Examination 2010, to examine what they could have done differently to preserve the integrity of the exams, and to restore public trust. 

After all they still have to come back to the public with the results of Examinations 2010 - to tell us who among the examinees they are promoting to the next class, and who did not make the mark.  We would love to believe them when they do.

* Kathleen Letshabo writes in her personal capacity.  The full text of the standards and code referenced above can be found in the following weblinks: 

*Standards for Educational and Physchological Testing at http://www.apa.org/science/programs/testing/standards.aspx

*Code of Professional Responsibilities in Educational Measurements at http://www.ncme.org/resources/code.cfm 

*Code of Fair Testing Practise in Education at http://www.ncme.org/pubs/pdf/CodeofFairTestingPractices.pdf.