Unit 7: Data Collection Methods - Subjective Questions

Observation is a systematic method of collecting primary data by watching and recording the behavior, actions, events, or conditions of individuals or groups as they occur.

Major characteristics:

• It involves direct or indirect watching of the subject or situation.
• Data are recorded systematically rather than casually.
• It may be conducted in natural or controlled settings.
• The researcher may or may not interact with the subjects.
• It can provide information about actual behavior rather than only reported behavior.

Situations where observation is suitable:

• When respondents cannot accurately describe their behavior, such as studying customer movement in a store.
• When respondents may be unwilling to provide accurate answers about sensitive or socially undesirable behavior.
• When the researcher wants to study non-verbal communication, workplace activities, or natural interactions.

Observation is particularly useful when actual conduct is more important than opinions or recollections.

Structured observation:

• Uses a predetermined observation schedule or checklist.
• Specifies what behaviors, events, or characteristics must be recorded.
• Produces data that are easier to classify, compare, and analyze.
• Is appropriate when the research objectives and variables are clearly defined.

Unstructured observation:

• Does not use a rigid observation schedule.
• Allows the researcher to record a wide range of relevant events and behaviors.
• Is flexible and useful in exploratory research.
• May produce rich qualitative information.

Advantages of structured observation:

• Improves consistency among observers.
• Reduces irrelevant recording.
• Simplifies coding and statistical analysis.

Limitations of structured observation:

• May overlook unexpected but important behavior.
• Can restrict the researcher to predefined categories.

Advantages of unstructured observation:

• Captures unexpected findings.
• Provides detailed contextual information.
• Is useful when little is known about the subject.

Limitations of unstructured observation:

• Data may be difficult to organize and analyze.
• Observer judgment can introduce subjectivity.
• Different observers may record different aspects of the same event.

Participant observation occurs when the researcher becomes involved in the activities of the group being studied. The researcher may participate openly or, in some cases, covertly.

Non-participant observation occurs when the researcher observes the subjects without taking part in their activities.

Comparison:

• Participant observation provides deeper understanding of the group's experiences and meanings.
• Non-participant observation generally provides greater distance and reduces the researcher's influence on the setting.
• Participant observation is useful for studying cultures, communities, and organizational practices.
• Non-participant observation is suitable for studying behavior in classrooms, public places, stores, or laboratories.

Strengths of participant observation:

• Produces detailed contextual information.
• Helps the researcher understand behavior from the participants' perspective.

Strengths of non-participant observation:

• Allows more objective recording.
• Makes it easier to maintain a consistent observation procedure.

Ethical concerns:

• Participants should normally provide informed consent.
• Privacy and confidentiality must be protected.
• Covert observation requires strong ethical justification because participants may not know they are being studied.
• The researcher must avoid causing harm or interfering unnecessarily.

Common sources of error and bias:

• Observer bias: The observer records information according to personal expectations.
• Observer effect: Subjects change their behavior because they know they are being observed.
• Selective attention: The observer notices only certain behaviors.
• Recording error: Events are forgotten, misunderstood, or recorded inaccurately.
• Sampling error: The observation period or location does not adequately represent the population.
• Interpretation error: The observer confuses an interpretation with an actual observation.

Ways to improve reliability and validity:

• Define each behavior operationally and clearly.
• Train observers using common instructions and examples.
• Use more than one observer and compare their records.
• Conduct pilot observations before the main study.
• Use standardized checklists, recording forms, or digital devices.
• Observe subjects on multiple occasions and in different settings.
• Keep descriptive observations separate from personal interpretations.
• Minimize the observer's interference with the setting.

These procedures help produce observations that are consistent, accurate, and relevant to the research objectives.

Experimentation is a data collection method in which the researcher deliberately changes one or more conditions and observes the effect on another variable while controlling other relevant factors.

Essential elements of an experiment:

• Independent variable: The factor deliberately manipulated by the researcher.
• Dependent variable: The outcome measured to determine the effect of the manipulation.
• Experimental group: The group exposed to the treatment or changed condition.
• Control group: The group that does not receive the treatment or receives a standard condition.
• Control of extraneous variables: Other factors are held constant or statistically controlled.
• Randomization: Subjects are assigned to groups by chance to reduce selection bias.
• Replication: The experiment is repeated with sufficient subjects or trials to improve reliability.
• Measurement procedure: Outcomes are measured using valid and reliable instruments.

A well-designed experiment permits the researcher to examine whether changes in the independent variable are responsible for changes in the dependent variable.

Laboratory experiment:

• Conducted in an artificial or highly controlled environment.
• Allows the researcher to control many extraneous variables.
• Provides precise measurement and easier replication.
• May have limited realism because participants know they are in a research setting.

Field experiment:

• Conducted in a natural or real-life setting.
• Participants are usually exposed to conditions that resemble their normal environment.
• Provides greater realism and practical relevance.
• Offers less control over outside influences.

Advantages of laboratory experiments:

• Strong control over variables.
• Easier identification of cause-and-effect relationships.
• Efficient data collection and replication.

Limitations of laboratory experiments:

• Artificial behavior may reduce external validity.
• Participants may respond differently because they know they are being studied.

Advantages of field experiments:

• More natural behavior.
• Findings may generalize better to real-world conditions.

Limitations of field experiments:

• Difficult to control all external variables.
• Greater logistical and ethical challenges.
• Replication may be more difficult.

Internal validity refers to the extent to which an observed change in the dependent variable can confidently be attributed to the independent variable rather than to other factors.

External validity refers to the extent to which the findings of an experiment can be generalized to other people, settings, times, or situations.

Improving internal validity:

• Use random assignment to experimental conditions.
• Include a suitable control group.
• Standardize the treatment and measurement procedures.
• Control or balance extraneous variables.
• Use reliable and valid instruments.
• Prevent experimenter and participant expectations from influencing results through blinding where appropriate.

Improving external validity:

• Select a sample that represents the target population.
• Conduct the study in realistic settings when possible.
• Replicate the experiment with different groups and contexts.
• Use realistic treatments and outcomes.
• Report the research setting and sample characteristics clearly.

There is often a trade-off: highly controlled laboratory conditions may increase internal validity while reducing the naturalness required for external validity.

The experimental design process generally includes the following stages:

• Identify the research problem: State clearly what relationship or effect will be investigated.
• Formulate objectives and hypotheses: Develop a testable prediction about the expected relationship between variables.
• Define variables: Specify the independent, dependent, and extraneous variables.
• Operationalize variables: Decide how each variable will be manipulated or measured.
• Select the sample: Identify the population and choose appropriate participants.
• Assign participants to groups: Use random assignment where possible.
• Select the design: Choose a suitable design, such as a post-test-only, pretest-post-test, or factorial design.
• Conduct a pilot study: Test the procedures and instruments on a small scale.
• Implement the experiment: Apply the treatment consistently while monitoring the conditions.
• Collect and analyze data: Use appropriate descriptive or inferential techniques.
• Interpret findings: Determine whether the evidence supports the hypothesis.
• Report limitations and conclusions: Explain the implications, possible sources of error, and the extent to which the results can be generalized.

A survey is a systematic method of collecting information from individuals by asking them questions about their characteristics, attitudes, opinions, knowledge, intentions, or behavior.

Surveys are preferred when:

• Information is needed from a large number of respondents.
• The research concerns opinions, attitudes, preferences, or self-reported behavior.
• The researcher needs standardized information that can be compared across respondents.
• The population is geographically dispersed.
• Quantitative analysis is required.
• Time and financial resources are limited.
• A description of the population is needed rather than direct manipulation of variables.

Common survey modes:

• Face-to-face interviews.
• Telephone surveys.
• Mail surveys.
• Online surveys.
• Self-administered questionnaires.

Surveys can provide broad coverage efficiently, but their quality depends on appropriate sampling, clear questions, adequate response rates, and accurate reporting by respondents.

Personal interviews:

• Allow clarification of questions and probing for detailed responses.
• Usually produce richer data and higher completion rates.
• Are expensive and may be affected by interviewer bias.

Telephone surveys:

• Are faster and less costly than personal interviews.
• Permit some clarification and interviewer interaction.
• May exclude people without telephone access and are often limited in length.

Mail surveys:

• Are relatively inexpensive and allow respondents time to consider answers.
• Provide greater privacy for sensitive topics.
• Often have low response rates and do not allow immediate clarification.

Online surveys:

• Are fast, economical, and suitable for automatic data capture.
• Can use skip patterns and multimedia elements.
• May exclude people with limited internet access and can suffer from duplicate, careless, or nonrepresentative responses.

The appropriate mode depends on the target population, topic sensitivity, required depth, budget, time, and expected response rate.

Major sources of survey error:

• Coverage error: Some members of the target population are not included in the sampling frame.
• Sampling error: The selected sample differs from the population by chance.
• Nonresponse error: People who do not respond differ systematically from those who respond.
• Measurement error: Questions or response scales fail to measure the intended concept accurately.
• Response bias: Respondents give inaccurate answers because of memory problems, social desirability, or lack of knowledge.
• Processing error: Mistakes occur during coding, data entry, cleaning, or analysis.

Methods for reducing errors:

• Use a complete and current sampling frame.
• Select an appropriate probability sampling method.
• Calculate an adequate sample size.
• Use simple, neutral, and unambiguous questions.
• Pilot-test the questionnaire.
• Protect confidentiality and explain the purpose of the study.
• Send reminders and offer convenient response options.
• Train interviewers and standardize administration.
• Check data for missing, inconsistent, and impossible values.

Reducing survey error improves the accuracy and generalizability of the findings.

A questionnaire is a structured research instrument containing a set of written or electronically presented questions used to collect information from respondents in a standardized manner.

Functions of a questionnaire:

• Translates research objectives into measurable questions.
• Collects comparable information from many respondents.
• Records demographic, behavioral, attitudinal, and factual data.
• Provides a consistent basis for coding and analysis.
• Reduces variation in the way questions are presented.

Open-ended item:

• Allows respondents to answer in their own words.
• Produces detailed and unexpected information.
• Is useful for exploration and complex opinions.
• Requires more time to answer and is more difficult to code.

Closed-ended item:

• Provides a fixed set of response options.
• Is quick to answer and easy to compare and analyze.
• Requires response options to be complete and mutually exclusive.
• May restrict respondents or fail to capture an answer that is not listed.

The choice depends on the research objective, respondent characteristics, and intended method of analysis.

Good questionnaire questions should follow these principles:

• Use simple language: Replace technical terms with words familiar to respondents.
• Ask one thing at a time: Avoid double-barreled questions.
• Be specific: Include a clear time period, place, or behavior.
• Avoid leading questions: Do not suggest the preferred answer.
• Avoid loaded or emotional wording: Use neutral language.
• Avoid assumptions: Do not presume that every respondent has the same experience.
• Avoid vague terms: Words such as "often" or "regularly" should be defined when necessary.
• Avoid double negatives: They can confuse respondents.
• Provide balanced response options: Include positive, negative, and neutral alternatives where appropriate.
• Ensure relevance: Ask only questions related to the research objectives.

Examples:

• Poor: "Do you agree that the excellent new service is useful?"
• Better: "How useful do you find the service?"
• Poor: "How satisfied are you with the price and quality of the product?"
• Better: Ask separate questions about price and quality.

Clear wording reduces measurement error and improves the quality of responses.

The questionnaire design process involves the following stages:

• Define the research objectives: Identify exactly what information is required.
• Specify the information needed: Determine the concepts, variables, and indicators to be measured.
• Identify the target respondents: Consider their language, knowledge, literacy, and access to the survey mode.
• Choose the method of administration: Select face-to-face, telephone, mail, online, or self-administered delivery.
• Determine the question content: Decide whether each objective requires a factual, behavioral, attitudinal, or demographic question.
• Select question types: Choose open-ended, closed-ended, ranking, rating, or multiple-choice formats.
• Write the questions: Use clear, neutral, concise, and respondent-appropriate wording.
• Design the response categories: Make categories mutually exclusive and collectively exhaustive where possible.
• Arrange the questions: Begin with easy and relevant questions, group similar topics, and place sensitive questions later.
• Design instructions and layout: Use clear directions, readable formatting, and appropriate skip patterns.
• Pretest the questionnaire: Identify ambiguity, missing options, timing problems, and respondent discomfort.
• Revise and finalize: Incorporate findings from the pretest and prepare the final instrument for administration.

Sequencing:

• Begins with simple, interesting, and non-threatening questions.
• Moves from general questions to specific questions.
• Groups questions on the same topic together.
• Places sensitive or demographic questions near the end when appropriate.

Layout:

• Should be clean, readable, and visually consistent.
• Response options must be aligned and clearly separated.
• Numbering should be logical and continuous.
• Online layouts should work on different screen sizes.

Instructions:

• Explain how each question should be answered.
• Clarify whether one or multiple responses are allowed.
• Provide definitions for unusual terms.
• Tell respondents when to skip a question or section.

Skip patterns:

• Direct respondents only to questions relevant to their experiences.
• Reduce respondent burden and avoid irrelevant answers.
• Prevent logical contradictions in the data.
• Must be clearly written and tested to avoid routing errors.

A well-designed structure improves completion rates, reduces confusion, and produces more accurate data.

A rating scale is a measurement tool that asks respondents to indicate the degree, intensity, or position of their attitude, opinion, or evaluation.

Likert scale:

• Measures agreement or disagreement with a statement.
• Common response categories range from strong agreement to strong disagreement.
• Is useful for measuring attitudes and perceptions.
• Several items may be combined to measure one construct.

Semantic differential scale:

• Measures a concept between two opposite adjectives, such as useful and useless.
• Respondents select a position along a numbered or verbal continuum.
• Is useful for measuring the image or meaning associated with an object, brand, or service.

Ranking scale:

• Requires respondents to place alternatives in order of preference or importance.
• Shows relative priority among options.
• Does not necessarily show the distance between ranked alternatives.

Likert scales measure agreement, semantic differential scales measure position between bipolar descriptions, and ranking scales measure relative order.

A pilot test is a small-scale trial of a questionnaire conducted before the main survey. It helps identify and correct problems in the instrument and administration procedure.

Aspects to evaluate:

• Whether respondents understand the wording of each question.
• Whether any question is ambiguous, leading, sensitive, or difficult to answer.
• Whether response options are complete, mutually exclusive, and balanced.
• Whether the order of questions is logical.
• Whether skip instructions work correctly.
• How long the questionnaire takes to complete.
• Whether respondents lose interest or stop before finishing.
• Whether sensitive questions cause discomfort or refusal.
• Whether the instrument produces sufficient variation in responses.
• Whether data can be coded and analyzed as intended.
• Whether interviewers or administrators need additional training.

After the pilot test, the researcher should revise, remove, add, or reorder questions as necessary. Pilot testing improves validity, reliability, usability, and response quality.

Reliability is the consistency or stability of a measurement. A reliable questionnaire produces similar results when the underlying condition has not changed.

Validity is the extent to which a questionnaire measures the concept it is intended to measure.

Assessing reliability:

• Test-retest reliability: Administer the same instrument at two different times and compare the results.
• Internal consistency: Examine whether items intended to measure the same concept produce consistent responses.
• Inter-rater reliability: Compare ratings made by different observers or coders when applicable.

Assessing validity:

• Content validity: Determine whether the instrument covers all relevant aspects of the concept.
• Construct validity: Check whether the items behave as expected in relation to the theoretical concept.
• Criterion-related validity: Compare results with an appropriate external standard or measure.

Improving both:

• Define concepts clearly.
• Use multiple well-designed items for complex constructs.
• Avoid ambiguous and biased wording.
• Use established scales where suitable.
• Obtain expert review.
• Conduct pilot testing and revise weak items.

A questionnaire may be reliable without being valid, because consistent measurement does not guarantee that the correct concept is being measured.

Coding is the process of assigning numerical or symbolic labels to respondents' answers so that the data can be organized, entered, summarized, and analyzed.

Why coding is necessary:

• Converts responses into a form suitable for data processing.
• Makes similar answers easy to group and compare.
• Reduces ambiguity during analysis.
• Supports tabulation and statistical calculations.
• Helps identify missing, invalid, or inconsistent responses.

Main stages:

• Develop a coding scheme: List each question, response option, and assigned code.
• Code closed-ended questions: Assign values to each predefined response category.
• Code open-ended questions: Review responses, identify themes, create categories, and assign codes.
• Assign special codes: Use clearly defined codes for missing, not applicable, refused, or do-not-know responses.
• Prepare a codebook: Document variable names, labels, value codes, and measurement levels.
• Enter the data: Transfer codes accurately into the database or statistical software.
• Check the coded data: Detect out-of-range values, inconsistent entries, and coding errors.

A well-documented coding system improves the accuracy, transparency, and reproducibility of data analysis.

Open-ended responses require a systematic procedure because respondents can use different words to express similar ideas.

Procedure:

• Read a representative sample of responses.
• Identify recurring ideas, concepts, or themes.
• Develop categories that reflect the research objectives and the actual responses.
• Define each category clearly.
• Assign a numerical code to each category.
• Include an "other" category only when necessary and record unusual responses separately.
• Allow multiple codes when a response contains more than one relevant idea, if the research design permits it.
• Apply the coding scheme consistently to all responses.
• Use independent coders for a subset of responses and compare their decisions.
• Revise category definitions when substantial disagreement occurs.

Potential problems:

• Categories may overlap.
• Important responses may be forced into unsuitable categories.
• Coder interpretation may introduce bias.
• Rare but meaningful answers may be ignored.
• The original meaning may be lost during summarization.

These problems can be controlled through clear definitions, coder training, pilot coding, intercoder checks, and preservation of the original responses.