Unit 6: Measurement and Scaling Technique - Subjective Questions

Measurement is the systematic process of assigning numbers, symbols, or labels to the characteristics of objects, individuals, events, or concepts according to predetermined rules.

Main purposes of measurement:

• Quantification: It converts abstract characteristics, such as intelligence or satisfaction, into measurable data.
• Comparison: It enables researchers to compare individuals, groups, or situations.
• Statistical analysis: Numerical measurement permits the use of statistical techniques.
• Hypothesis testing: It provides empirical evidence for testing research hypotheses.
• Objectivity: Standardized rules reduce personal bias in observation and interpretation.
• Decision-making: Reliable measurements support valid conclusions and informed decisions.

Thus, measurement connects theoretical concepts with observable empirical evidence.

A sound measurement tool should possess the following characteristics:

• Validity: The instrument should measure what it is intended to measure.
• Reliability: It should produce consistent results when used repeatedly under similar conditions.
• Objectivity: The result should not depend excessively on the personal judgment of the researcher or scorer.
• Sensitivity: It should detect small but meaningful differences among respondents.
• Practicality: It should be economical, easy to administer, simple to score, and suitable for the research setting.
• Standardization: Administration, scoring, and interpretation procedures should be uniform.

A tool cannot be considered sound merely because it is reliable. It must also be valid, objective, sensitive, and practical for its intended purpose.

Reliability refers to the consistency, stability, and dependability of scores produced by a measurement tool. A reliable instrument gives similar results when the measured characteristic has not changed.

Major methods of estimating reliability:

• Test-retest reliability: The same instrument is administered to the same respondents at two different times, and the two sets of scores are correlated.
• Parallel-forms reliability: Two equivalent versions of an instrument are administered, and their scores are compared.
• Split-half reliability: Items are divided into two comparable halves, and the scores of the halves are correlated.
• Internal consistency: The consistency among all items is estimated using measures such as Cronbach's alpha.
• Inter-rater reliability: The degree of agreement between two or more observers or scorers is assessed.

A reliability coefficient generally ranges from to , with a higher value indicating greater consistency.

Validity is the extent to which a measurement tool accurately measures the concept it is intended to measure.

• Content validity: Examines whether the items adequately represent the entire domain of the concept. It is commonly established through expert judgment and a table of specifications.
• Criterion-related validity: Examines how well the instrument corresponds with an external criterion. Concurrent validity uses a criterion measured at the same time, while predictive validity uses future performance as the criterion.
• Construct validity: Determines whether the instrument behaves according to the theoretical meaning of the underlying construct. It may be evaluated through convergent validity, discriminant validity, and factor analysis.

Content validity focuses on coverage, criterion validity on external correspondence, and construct validity on theoretical consistency.

Reliability concerns the consistency of measurement, whereas validity concerns the accuracy and appropriateness of the interpretation made from measurement scores.

Major differences:

• Reliability asks, "Does the tool produce stable and consistent results?"
• Validity asks, "Does the tool measure what it is supposed to measure?"
• Reliability is commonly assessed using repeated measurements, equivalent forms, or internal consistency.
• Validity is assessed through content evidence, external criteria, and theoretical relationships.
• Reliability is necessary for validity, but it is not sufficient by itself.

A tool can be reliable without being valid. For example, a weighing machine that consistently adds kg gives stable results and is therefore reliable, but its readings are inaccurate and hence invalid. A valid measurement generally requires an acceptable degree of reliability.

The development of a measurement tool generally involves the following steps:

1. Define the construct: Clearly specify the concept and its theoretical meaning.
2. Identify dimensions: Divide the construct into measurable components or indicators.
3. Review existing tools: Examine relevant literature and previously validated instruments.
4. Prepare an item pool: Develop more items than will be required in the final tool.
5. Select the response format: Choose an appropriate scale, such as a Likert or rating scale.
6. Obtain expert review: Evaluate content relevance, clarity, and representativeness.
7. Conduct a pilot test: Administer the draft instrument to a sample resembling the target population.
8. Perform item analysis: Examine item difficulty, discrimination, correlations, and response patterns.
9. Assess reliability and validity: Establish the psychometric quality of the instrument.
10. Revise and standardize: Remove weak items and prepare uniform instructions, scoring rules, and interpretation guidelines.

These steps make tool construction systematic, evidence-based, and suitable for the intended population.

Pilot testing is the trial administration of a draft instrument to a small sample similar to the intended respondents. It helps identify unclear wording, unsuitable response options, excessive completion time, and administrative difficulties.

Item analysis evaluates the performance of individual items after pilot data have been collected. It may examine:

• Item difficulty: The proportion of respondents answering an item correctly in an achievement test.
• Item discrimination: The ability of an item to distinguish between high- and low-scoring respondents.
• Item-total correlation: The relationship between an item's score and the total scale score.
• Distractor efficiency: The effectiveness of incorrect alternatives in multiple-choice items.
• Response distribution: The presence of extreme skewness, missing responses, or floor and ceiling effects.

Together, pilot testing and item analysis support the revision or removal of weak items and improve the reliability, validity, clarity, and practicality of the final instrument.

Scaling is the process of creating a continuum on which objects, persons, attitudes, perceptions, or other characteristics are located according to the amount or intensity of an attribute they possess.

Importance of scaling:

• It converts qualitative judgments into analyzable data.
• It makes abstract constructs such as attitude, motivation, and satisfaction measurable.
• It indicates relative position, intensity, or preference.
• It facilitates comparisons among respondents or groups.
• It supports statistical analysis and hypothesis testing.
• It improves consistency by providing standardized response categories.

Measurement assigns values according to rules, while scaling specifically concerns the construction of a continuum and the placement of responses on it. A well-designed scale should have clearly defined categories and should match the nature of the construct.

Comparative scales require respondents to evaluate one object in direct comparison with another object.

• Results usually provide relative or ordinal information.
• Common techniques include paired comparison, rank order, constant sum, and Q-sort.
• Example: Asking a respondent whether Brand A or Brand B is preferred.

Non-comparative scales require each object to be evaluated independently rather than against another specified object.

• They may produce ordinal or approximately interval data, depending on the design and assumptions.
• Common techniques include continuous rating scales, Likert scales, semantic differential scales, and Stapel scales.
• Example: Rating Brand A's quality from to .

Comparative scales are useful for establishing preferences among alternatives, whereas non-comparative scales are better suited for independently measuring the intensity of an attitude or perception.

A Likert scale measures the degree of agreement or disagreement with a series of statements. A common five-point format ranges from Strongly Disagree to Strongly Agree.

Construction:

1. Define the attitude or construct.
2. Prepare a large set of favorable and unfavorable statements.
3. Obtain expert review and pilot responses.
4. Conduct item analysis and retain discriminating items.
5. Arrange the items with uniform response categories.

Scoring: Favorable items may be scored from to . Scores for unfavorable items are reversed. The total score is usually calculated as:

where is the score on item and is the number of items.

Advantages: It is simple, economical, reliable, and suitable for measuring attitude intensity.

Limitations: It may be affected by acquiescence, social desirability, central-tendency bias, and the assumption that response categories are equally spaced.

The Thurstone equal-appearing interval scale is an attitude scale in which statements are assigned scale values by a panel of judges according to how favorable or unfavorable they are toward an issue.

Construction procedure:

1. Prepare a large number of statements covering the full attitude range.
2. Ask judges to classify the statements into ordered categories, often from extremely unfavorable to extremely favorable.
3. Calculate the median category assigned to each statement as its scale value.
4. Calculate the interquartile range to assess disagreement among judges.
5. Select statements distributed across the continuum and having low judge disagreement.
6. Ask respondents to indicate the statements with which they agree.
7. Determine a respondent's score from the median or mean scale value of the endorsed statements.

The technique attempts to create approximately equal intervals between scale points. Its major strengths are systematic item weighting and broad attitude coverage, while its limitations include complex construction and dependence on judges' assessments.

A Guttman scale is a cumulative and unidimensional scale in which items are arranged from the weakest to the strongest expression of a characteristic. Agreement with a stronger item implies agreement with all weaker items.

For example, if the ordered items represent increasing levels of political participation, a respondent who agrees with the fourth item should theoretically agree with the first three items.

The fit of responses to the cumulative pattern is assessed using the coefficient of reproducibility:

where:

• is the total number of response errors,
• is the number of respondents, and
• is the number of items.

A coefficient of about or above is commonly considered acceptable, although additional evidence of scalability should also be examined. The scale is easy to interpret but difficult to construct because real attitudes do not always form a perfect cumulative pattern.

The semantic differential scale, developed by Osgood and associates, measures the connotative meaning of an object or concept using bipolar adjective pairs.

A respondent rates a concept on a series of scales such as:

• Good Bad
• Strong Weak
• Active Passive

Its dimensions are often grouped into:

• Evaluation: Good-bad or pleasant-unpleasant
• Potency: Strong-weak or powerful-powerless
• Activity: Active-passive or fast-slow

Applications:

• Measuring brand or institutional image
• Comparing perceptions of products or services
• Studying attitudes toward social concepts
• Developing respondent profiles
• Evaluating changes before and after an intervention

The scale is flexible and easy to analyze, but adjective pairs must be relevant, genuinely bipolar, and understandable to the target population.

Likert, Thurstone, and Guttman scales all measure attitudes, but their construction and interpretation differ.

• Likert scale: Respondents indicate their degree of agreement with each statement. Items usually receive equal weight, and scores are summed. It is comparatively easy to construct and widely used.
• Thurstone scale: Judges first assign scale values to statements. Respondents then indicate which statements they endorse. It aims to approximate equal intervals but requires considerable time and expert judgment.
• Guttman scale: Items form a cumulative hierarchy. Endorsing a stronger item should imply endorsement of all weaker items. It provides a clear cumulative interpretation but is difficult to achieve with complex attitudes.

Comparison:

• Likert emphasizes intensity of agreement.
• Thurstone emphasizes judge-determined item positions.
• Guttman emphasizes cumulative ordering.

The choice depends on the construct, required level of precision, available resources, and whether a cumulative structure is theoretically appropriate.

These are important comparative scaling techniques:

• Paired comparison: Respondents choose one preferred object from every possible pair. For objects, the number of pairs is:

It is simple for a small number of objects but becomes burdensome as increases.

• Rank-order scaling: Respondents arrange several objects from most preferred to least preferred. It is quick and easy, but it reveals order rather than the size of differences between ranks.

• Constant-sum scaling: Respondents distribute a fixed number of points, commonly , among alternatives according to their relative importance or preference. It provides ratio-like information about relative weights but may be cognitively demanding.

All three techniques compare alternatives directly. Paired comparison is suited to detailed two-object choices, rank order to overall ordering, and constant sum to the allocation of relative importance.

A nominal scale classifies observations into mutually exclusive and collectively exhaustive categories. Numbers assigned to categories act only as labels and do not indicate order or magnitude.

Examples include gender categories, blood groups, religion, department, and place of residence.

Statistical properties:

• It possesses the property of identity or classification.
• Categories must be distinct and non-overlapping.
• Equality and inequality comparisons are meaningful.
• Any one-to-one relabeling of categories is permissible.

Permissible statistics:

• Frequency and percentage
• Mode
• Proportions and contingency tables
• Chi-square tests
• Measures of association designed for categorical data

The mean, median, standard deviation, and arithmetic operations are generally not meaningful for nominal data because the category codes do not represent quantity or order.

An ordinal scale classifies observations into categories that have a meaningful order or rank. However, the distance between successive ranks is not necessarily equal.

Examples include class rank, socioeconomic status categories, satisfaction levels, and preference rankings.

Statistical properties:

• It possesses identity and order.
• Statements such as are meaningful.
• Differences such as are not necessarily meaningful.
• Any strictly increasing transformation that preserves order is permissible.

Appropriate statistics:

• Frequency and percentage
• Mode and median
• Percentiles and quartiles
• Range and interquartile range
• Spearman's rank correlation
• Non-parametric tests such as the Mann-Whitney and Kruskal-Wallis tests

Its major limitation is that rank differences cannot be interpreted as equal quantitative differences. Therefore, arithmetic means and standard deviations require caution and additional assumptions.

An interval scale has identity, order, and equal intervals between adjacent values, but it does not possess a true or absolute zero.

Examples include temperature measured in Celsius or Fahrenheit, calendar years, and many standardized test scores.

Statistical properties:

• Equal numerical differences represent equal differences in the measured characteristic.
• Addition and subtraction are meaningful.
• Ratios are not meaningful because zero is arbitrary. For example, is not twice as hot as .
• A permissible linear transformation has the form:

Appropriate statistics:

• Mean, median, and mode
• Range, variance, and standard deviation
• Pearson's correlation
• Regression, -tests, and analysis of variance when relevant assumptions are satisfied

The absence of a true zero distinguishes an interval scale from a ratio scale.

A ratio scale possesses identity, order, equal intervals, and a true zero representing the complete absence of the measured quantity.

Examples include height, weight, age, income, distance, and time duration.

Properties of a ratio scale:

• Differences between values are meaningful.
• Ratios are meaningful; for example, kg is twice kg.
• All common arithmetic operations may be performed.
• A permissible transformation has the form:

Comparison with the interval scale:

• Both have ordered values and equal intervals.
• An interval scale has an arbitrary zero, while a ratio scale has a true zero.
• Ratio statements are invalid for interval data but valid for ratio data.
• Interval transformations allow changes in origin and unit, , whereas ratio transformations allow only changes in unit, .

Ratio measurement supports the widest range of descriptive and inferential statistical procedures.

The four levels of measurement differ in the information they preserve and the statistical operations they support.

• Nominal scale: Possesses identity only. Examples include blood group and department. Categories may be relabeled through any one-to-one transformation. Frequencies, percentages, mode, and chi-square tests are appropriate.
• Ordinal scale: Possesses identity and order. Examples include ranks and satisfaction categories. Any strictly increasing transformation is permissible. Median, percentiles, rank correlation, and non-parametric tests are appropriate.
• Interval scale: Possesses identity, order, and equal intervals, but no true zero. Examples include Celsius temperature and standardized scores. Transformations of the form , where , are permissible. Mean, standard deviation, correlation, regression, and many parametric tests may be used.
• Ratio scale: Possesses identity, order, equal intervals, and a true zero. Examples include weight, age, and income. Transformations of the form , where , are permissible. All arithmetic and most statistical procedures are meaningful.

The properties are cumulative:

A researcher should choose statistical methods according to the actual measurement properties of the data rather than merely according to the numerical codes assigned to responses.