Testing in language programs (chapter 8)

Language Test ReliabilityLanguage Test Reliability
Teacher:Teacher: Dr. Golshan
Prepared by:Prepared by:Tahere Bakhshi
November 2015November 2015
In the name of God

A test should have:A test should have:
Reliability: (Same result under the same condition)Reliability: (Same result under the same condition)
Validity: (Scale to measure the size of head Not sthValidity: (Scale to measure the size of head Not sth
else)else)
Usability or Practicality: (Not too difficult, practicalUsability or Practicality: (Not too difficult, practical
to use)to use)
•The problem of measuring mental traits, languageThe problem of measuring mental traits, language
proficiency, motivation and … !proficiency, motivation and … !
• Tests should measure consistently !Tests should measure consistently !

VarianceVariance
VarianceVariance:: variance measureshow far aset of numbersisspread out.

Variance of Zero: Identical values
Small Variance: Expected value close to mean
High Variance: Spread out values, far from mean

Sources ofSources of
VarianceVariance
 Meaningful VarianceMeaningful Variance
Those sources that make variance related to the purpose
of the test.
To gain the goal: (Items be related to the purpose of designed test & students’
knowledge on topic. Test validity issue: (see Table 8.1, P. 170) )
Other Factors unrelated to the aim of the test :Other Factors unrelated to the aim of the test :
 Measurement error or Error VarianceMeasurement error or Error Variance
Those sources that make variance related to other
extraneous variables.

 Types of issues related to errorTypes of issues related to error
variancevariance1.Variance due to the environment: (1.Variance due to the environment: (Noise, classroom
temperature, outside noises, distractions, amount of space per person,
lighting, ventilation, or other environmental factors)).
2. Variance due to the administration procedure: (2. Variance due to the administration procedure: (Directions
of test, Quality of equipment and timing (Cassette or teachers ) )). Table
2.5, p.35
3. Variance due to examinees:3. Variance due to examinees: ((Condition of students: their
fatigue, health, hearing or vision)). ((Psychological factors:
motivation, memory, concentration, forgetfulness, impulsiveness,
carelessness and…). (). (Students’ testwiseness and Strategies))
4. Variance due to scoring procedure:4. Variance due to scoring procedure: Errors in doing scoring.
Subjective nature of scoring procedure.
5. Variance due to test and test items: (5. Variance due to test and test items: (Printing, knowing answer
sheet, number of items, item selection, quality of test items, test
security))
The mentioned sources of measurement error should beThe mentioned sources of measurement error should be
minimized so that there is no Variance in students’minimized so that there is no Variance in students’

((Dependable or trustworthy))
A test is considered reliable if it would give us the same result
over and over again.
How is reliability measured?How is reliability measured?
By comparing two sets of scores for a single assessment (such as twoBy comparing two sets of scores for a single assessment (such as two
rater scores for the same person).rater scores for the same person). After having two sets of scores for
a group of students, we can determine how similar they are by
computing a statistic known as the reliability coefficient.
Reliability Coefficient:Reliability Coefficient:
A numerical index of reliability, ranging from 0 to 1.
Number closer to 1 = high reliability. A low reliability
coefficient indicates more error in the assessment results.
Reliability is considered good or acceptable if the reliability
coefficient is .80 or above.
 Reliability of NRTsReliability of NRTs

Testing in language programs (chapter 8)

1. Test-Retest Reliability:1. Test-Retest Reliability:
SituationSituation: Same people taking two administrations of the same test.
ProcedureProcedure: Correlate scores on the two tests which yields the
coefficient of stability.
MeaningMeaning: the extent to which scores on a test can be generalized
over different occasions (temporal stability).
Appropriate use:Appropriate use: Information about the stability of the trait over time.
DisadvantagesDisadvantages: Requires two testing sessions, Learning, Test
effect.
 Three Basic Strategies to Estimate theThree Basic Strategies to Estimate the
reliability of a Test:reliability of a Test:

2. Parallel / Equivalent-Forms Reliability:2. Parallel / Equivalent-Forms Reliability:
 SituationSituation: Testing of same people on different but comparable
forms of the test. (Forms A & B)
 ProcedureProcedure: correlate the scores from the two tests which yields a
coefficient of equivalence.
 Meaning:Meaning: the consistency of response to different item samples (where
testing is immediate) and across occasions (where testing is delayed).
 Appropriate use:Appropriate use: to provide information about the equivalence of
forms.
Ali usually ………… late at night. A. study b. studiesAli usually ………… late at night. A. study b. studies
c. studyingc. studying
Reza often ………… the shopping in the afternoon. A. do b. doesReza often ………… the shopping in the afternoon. A. do b. does
c. doingc. doing

3. Internal Consistency3. Internal Consistency
Reliability:Reliability:
• Situation:Situation: a single administration of one test form. All items in
an internally consistent scale assess the same construct.
•Procedure:Procedure: Divide test into comparable halves and correlate
scores from both halves.
– Split Half with Spearman Brown adjustment
– Kuder Richardson #20 and #21
– Cronbach’s Alpha
•Meaning:Meaning: consistency across the parts of a measuring instrument
(“parts” = individual items or subgroups of items).
•Appropriate Use:Appropriate Use: Where focus is on the degree to which same
characteristic is being measured. A measure of test homogeneity.

Internal Consistency StrategiesInternal Consistency Strategies
All items in the test should be homogenous. And there should be a
relationship among them.
Split – HalfSplit – Half
ReliabilityReliability
Split – HalfSplit – Half
ReliabilityReliability
Cronbach AlphaCronbach AlphaCronbach AlphaCronbach Alpha
Kuder-RichardsonKuder-Richardson
FormulasFormulas
Kuder-RichardsonKuder-Richardson
FormulasFormulas

Split – Half Reliability)Split – Half Reliability)
In split-half reliability we randomly divide all items that purport to measure
the same construct into two sets. We administer the entire instrument to a
sample of people and calculate the total score for each randomly divided
half. the split-half reliability estimate, as shown in the figure, is simply the
correlation between these two total scores. In the example it is .87.
Odd/ even Items, easy and difficult item equally distributed.

Spearman Brown Prophecy FormulaSpearman Brown Prophecy Formula
k = the number of items I WANTk = the number of items I WANT
toto
estimate the reliability for dividedestimate the reliability for divided
byby
the number of items I HAVEthe number of items I HAVE
( )11
*
11
11
−+
=
kr
rk
rkk

Cronbach AlphaCronbach Alpha
Cronbach Coefficient Alpha used only if the item scores are other
than 0 & 1. (Such as Likert scale). )This is advisable for essay items,
problem solving and 5-scaled items. ; based on 2 or more parts of the
test, requires only one administration of the test.

Kuder – Richardson FormulasKuder – Richardson Formulas
Kuder and Richardson believed that all items in a test are designed to
measure a single trait.
KR21 is the most practical, frequently used and convenient method
of estimating reliability.
K – R20 = most advisable if the p values vary a lot
K – R21 = most advisable if the items do not vary much in difficulty,
i.e., the p values are more or less similar.
The KR21 formula is a simplified version of theThe KR21 formula is a simplified version of the
KR20.KR20.

Inter-rater ReliabilityInter-rater Reliability
Having a sample of test papers (essays) scored independently by
two examiners.
Inter-rater reliability is a measure of reliability used to assess the
degree to which different judges or raters agree in their assessment
decisions. Inter-rater reliability is useful because human observers will
not necessarily interpret answers the same way; raters may disagree as
to how well certain responses or material demonstrate knowledge of
the construct or skill being assessed.

Intra-rater ReliabilityIntra-rater Reliability
The degree of stability observed when a measurement is repeated
under identical conditions by the same rater.
•Note: Intra-rater reliability makes it possible to determine the
degree to which the results obtained by a measurement procedure can
be replicated.

Standard Error of MeasurementStandard Error of Measurement
 All tests scores contain some error
 For any test, the higher the reliability estimate, the lower the error
 The standard error or measurement is the average standard
deviation of the error variance over the number of people in the
sample.
 Can be used to estimate a range within which a true score would
likely fall.
 We never know the true score
 By knowing the S.E.M. and by understanding the normal curve, we
can assess the likelihood of the true score being within certain
limits.
 The higher the reliability the lower the standard error of
measurement, hence more confidence we can place in the
accuracy of a person’s test score.

Factors That Affect The ReliabilityFactors That Affect The Reliability
CoefficientCoefficient
• Test lengthTest length
• Range of scoresRange of scores
• Item similarityItem similarity

Testing in language programs (chapter 8)

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Testing in language programs (chapter 8) (20)

Recently uploaded (20)

Testing in language programs (chapter 8)