Thursday, December 18, 2008

TESTING & EVALUATION WORKSHOP - the best ever !

TITLE : TESTING & EVALUATION

VENUE : De Palma Hotel, Ampang

DATES : 18-23/06/08, 16-19/08/08, 13-17/12/08


Alhamdulillah, I’ve just completed a 120-hours Testing & Evaluation (T&E) course cum workshop. The main objective of this workshop is to provide a basic knowledge and practical exposure in the development of quality, valid and reliable test items. My 4-member team comprised of Mohd Asri (KB), Norhisham (BP), and Mohd Zulfikar (BP) and yours truly. Honestly, this has been the best course I’ve ever attended in my 20-odd years in service with KPTM.


So, what’s so great about this course/workshop? Now, I’m sure each and every one of us has in our own way an experience taking a test. Your reason for taking the test could have been for a selection, classification, intelligence, an aptitude or may be simply for a placement purpose.


But how do we know that a particular test item set has any quality? What about their validity and reliability? Are we not concern at all?


Simply put, TESTING is a procedure for gathering valid and reliable information in evaluating the mastery level of our students on certain selected topics. It is to see whether they have learned what they are supposed to learn through the course of study.


To ensure this objective is met, a good quality test must therefore be constructed. But what is a good quality test? Simply put, a test of good quality will test what it is supposed to test on. Extra care and caution should therefore be taken to ensure that the test items were not confusing or misleading, what more if it is out of topic.


ITEM DEVELOPMENT PROCESS


The very first step in constructing test items is to plan its content area and the skills required for the students to be tested. To ensure that the test to be constructed will have a high content validity, references to the Learning Objectives (LO) and learning outcomes of the syllabus must be made when developing the Table of Specification (TOS).


Once you’re done with the writing process, then follow up with a reviewing process to see whether:


  • The items constructed are according to TOS?
  • There any repeats of the same question?
  • The contents and key of the items are of high validity?
  • The distracters can really discriminate between good and weak students?
  • The sentence arrangement, format and layout of the items were clear, easily understood and not confusing?


WHAT I’VE LEARNT FROM THIS WORKSHOP ?


Constructing a good test item is not as easy as most people would think. It’s not just about having the right format and layout, but also requires going through various process of planning, selection, analyzing, abiding to certain standards and guidelines. All these are to ensure that the test items we construct will actually test what it is supposed to test on.


The quality of a particular test depends on the quality of each individual item that we construct. Selection and construction of items for the final form should be based on their TOS and item analysis, focusing on the following criteria:


  • Does the item matches the LO?
  • Is there any wrong key? – whether the key used is confusing and/or incorrect?
  • What about ambiguity? – look into discrimination power (Point Bi-serial) or the r-value of the items. Good students should be able to answer correctly, and weak students answer wrongly.
  • Is there any blind guessing? – to see whether the examinees responded blindly.
  • The difficulty level (P-value).
  • What is the percentage of students who answer each item correctly? – from here we can know who/what is not good : either the lecturer, the student or is it the question itself ?
  • Effectiveness of the distracters used – whether they are good/bad. Do they function as expected?


A test is considered reliable only when it measures what it is supposed to measure, and consistently too ! And one need to remember also that our test scores may contain error that may lowers its reliability. These errors could have been due to several reasons:


  • The test taker – perhaps the candidate had a bad day like a cold/flu or an argument with someone special.

  • The test itself – the questions were confusing/ unclear. Some students failed in their exams not because they are weak or poor in the subject matter but mainly because the question items constructed were of low quality – vague, misleading and blatantly confusing! We have to be fair to them.

  • The test condition – there may be distractions (like noise or even unsuitable lighting or room temperature) in the exam hall that affect his attention and concentration level.


Frankly, before attending this workshop, issues on validity and reliability have not been my main concern. But now, they will definitely become my top priority whenever constructing the test items.


The Kuder-Richardson formula-20 (KR20) reliability coefficient used in our exercises showed that the less number of items the test has, the less reliable the test is. That means to say, to get a more reliable test, we have to construct more number of items to it.


To reduce blind guessing, we have to have more than 4 options for the Multiple Choice Question (MCQ) items. Example: if we have 4 options, 1 divide by 4 equals 0.25, and with 5 options, we have 1 divide by 5 equals 0.20. That shows the more options we have, the smaller the percentage of guessing it will be (ie. Only 20% possibility of guessing for an item with 5 options, as compared to 25% for a 4 options item).


Overall, I do faced a bit of a challenge to keep up with the momentum and continuity of this workshop as it was split into three parts, delivered on three separate sessions spanning from June to December 2008. Fuyooohh ! 120 jam tu beb !


Nevertheless, a very practical and activity-based approach used by our “Mr. Cool” T&E expert, Dr Quek Bong Cheang (OUM) was able to keep our attention and spirit high, right from the beginning till the end. His lectures were really an “eye-opener”, very relevant and practical indeed. Terima Kasih CikGu Quek - I like your style!


Finally, I would like to say my gratitude to the management of KPTM for giving me this valuable opportunity to undergo this workshop with Dr Quek.


In future, I hope to see more KPTM lecturers (or even the relevant Admin people) selected to attend this course. Like me, I am confident that the others can also gain a lot from it – at least the basic knowledge and skills in constructing test items with a lot better quality, validity and reliability level. And of course, God willing, at the end of the day our students will also benefit too... InsyaAllah.


6 comments:

Anonymous said...

Thanks for concise summary of the course content. I hope that I have an opportunity to attend beneficial and informatics course like what u had attend.

By the way, I like your math example 1 divide by 4 equals 0.25 and 1 divide by 5 equals 0.20. I agree and understood it well enough and I believe that no one can argue it.

But lately, I've problem. It trouble me so much. I need your help how can I prove that 1 divided by 2 equals to 1.0 plus 1.5. I'm very good at math, but this is the most complex formula that I cannot resolve till now. Can you help me to solve this mysterious formula?

Please, I really need your help. I'm in dizzy thinking about how to prove this formula. I feel so noob when I can't resolve this formula since I'm very excellent in math.

Thanks in advance.
From:
-Someone that begging for help and confuse-

SAS said...

Amboi amboi amboi Anonymous ... cuba nak tuang minyak kat api yg sdg membara ker? 1.5 & 1.0 tu nombor keramat tau? Sbb tu lah kita x leh nak wat simple math operation ke atas nyer. Heh3. Anyway, I appreciate ur comments. TQ.

Anonymous said...

Boleh sebenarnya, but it is in the term of analogy I dah dapat jawapannya.. For example, when 1 divided by 2, the result should be 0.5. When 3 divided by 2, the result should be 1.5.

Whatever the number divided by 2, it proved that when we add the result number with the result number again, the value should be the number that we divided by 2. For an example, 0.5 plus 0.5 equal to 1 (because 1 divided by 2 equal to 0.5), 1.5 plus 1.5 equal to 3 (because 3 divided by 2 equal to 1.5).

Therefore when there is a statement that 1 divided by 2 equal to 1.0 plus 1.5 it is considered irrelevant, unexplainable, unjustified and mysterious . The first argue about the statement is not because 1.0 plus 1.5 not equal to 2, but it is because 1.0 and 1.5 are not the same value. So the statement can’t be proved in mathematical theory.

Finally I found an answer to the problem; it is an analogy used in one institution located in near GMI but not formula in math. Yeah, I got my answer now, in my personal point of view, whatever the event occur in any organization, everything must be distributed evenly. Else, it is consider irrelevant, unexplainable, unjustified and mysterious.

SAS said...

Bijak lah kepala otak u Anonymous ... mmg auta to the power of infinity! Whatever it is, I've yet 2 c initiatives to make that 1.5 equals 1.0, or vice-versa.

An-Nimr said...

Yo bro!

Assalamualaikum. Aiyo, this piece heavy for me lah...Ummm... jgn terkejut bila bukak blog kita hari ni...(tak pandai nak photoshop)...

An-Nimr said...

Yo bro, it's me lah...tehsin! hahahaha