Home Analysis of Professional Literature
Class 5:
Quantitative Research Design:
Sampling & Measurement
James Neill
Last updated:
21 Feb 2003

Introduction

Key Terms

Sampling

Probability Sampling

Non-probability Sampling

Measurement

Important Aspects of Measurement

Recommended reading

Quantitative Exam


Introduction

 

In the previous class, we examined:

  • experimental, quasi-experimental, and non-experimental quantitative research designs,

  • potential threats to internal and external validity

In the practical exercise (Quantitative Exam Task 1 - 5%) you developed three quantitative research designs (experimental, quasi-experimental, and non-experimental) for your research question, and considered the strengths and weaknesses, as well as practical issues, for each of these designs .

 

Now it is time to deal with two more important aspects of quantitative reserach design:

 


Key Terms

 

Sampling

  • sampling

  • sampling frame

Probability Sampling

  • random sampling

  • stratified random sampling

  • systematic random sampling

  • cluster (area) random sampling

  • multi-stage random sampling

Non-Probability Sampling

  • accidental, haphazard, or convenience sampling

  • purposive sampling

  • modal instance sampling

  • expert sampling

  • proportional and nonproportional quota sampling

  • heterogeneity sampling

  • snowball sampling

Measurement

  • measurement

  • instrumentation

  • psychometrics

  • test-retest reliability

  • internal reliability (internal consistency)

  • test manual

  • normative data


Sampling

 

There are a wide range of possible options to consider when sampling.  At all times, the purpose of the study needs to borne in mind and the various strengths of weaknesses, as as the practicality, of different sampling methods need to be weighed.

 

Sampling involves selecting individual units to measure from a larger population.  The population refers to the set of individual units which the research question seeks to find out about.  For example, my population of interest may be adolescent females in the United States.  In particular, I may be interested to know whether socio-economic status influences American adolescent females' mental health.  There are a variety of ways I could consider to obtain a representative sample.  A sample is representative when it allows the results of the sample to be generalized to the population.

 

The sampling frame is the group of individuals who had a real chance of being selected for the sample.  For example, if I use as my sampling frame the lists of students held by public and private schools in America from which to select a sample of adolescent females, then only students on those lists have a real chance of being selected.  This may differ from the population to which I wish to generalize the results of my study (all adolescent females in the United States).  In this case, my sample will almost certainly be bias because adolescent females with poor mental and with lower socio-economic status are probably less likely to be on school lists than other students.

 

As you can see from this example, the process of sampling, even when done systematically, can introduce potentially critical biases into a research study.  Due our bias sampling technique we may enhance the risk of incorrectly concluding that there is no relationship between socioeconomic status and the mental health of adolescent females in the United States because we did representatively sample from adolescent females who weren't on school registers.

 

There are two main types of sampling - the key is whether or not the selection involves randomizationRandomization means that each unit within a sampling frame has an equal chance of being selected.  By selecting randomly from a sampling frame, probability theory says that our sample, more often than not, should approximately represent the whole population.  You can read more about sampling terminology and probability theory in social science research on Bill Trochim's Sampling Terminology page.

 


Probability Sampling

 

Probability sampling involves the use of randomization.  These are the main types of probability sampling:

 

Probability Sampling Method Brief Description
random sampling every unit has an equal chance of selection
stratified random sampling (proportional or quota sampling) divide population into strata, then randomly select samples from each stratum
systematic random sampling systematically select every xth unit from the list of n units
cluster (area) random sampling divide population into clusters, randomly sample clusters, then sample all units within selected clusters
multi-stage random sampling hierarchically combines random sampling methods

 

Read Bill Trochim's Probability Sampling for more detailed explanations of the random (probability) sampling methods.

 


Non-probability Sampling

 

Non-probability sampling does not involve the use of randomization.  Therefore, to be considered representative, non-probability sampling methods cannot rely on the theory of probability (random theory).  We can also use purposive or even accidental, haphazard, or convenience sampling to get a representative sample relying on other techniques than randomization.

 

Non-probability Sampling Method Brief Description
accidental, haphazard, or convenience sampling units are sampled according to what is conveniently, accidentally, or haphazardly available
purposive sampling units from a prespecified group are purposively sought out and sampled
modal instance sampling mode is the "most common" occurence; in modal instance sampling, units are prototypical of a predefined group are sampled
expert sampling units which are identified as having particularly high quality of information are sampled
proportional and nonproportional quota sampling sample until exact proportions of certain types of units are obtained, or until sufficient units in several different categories are obtained
heterogeneity or diversity sampling opposite of modal sampling; intentionally samples units from throughout spectrum of  responses
snowball sampling initial unit(s) are sampled, and these units then identify more units to sampled, and so on


Read Bill Trochim's Non-Probability Sampling for detailed explanations of these  non-probably sampling methods.


Measurement

In quantitative research, you end up with "numbers" after carrying out your research.  These are analyzed, and then interpreted in light of the research question and other relevant theory and research findings.

In order to create the "numbers" for quantitative research (data), a measurement process needs to take place.  In other words, you need to convert some human phenomenon (in the human sciences) accurately into numerical data.

The process of converting phenomena into data is called "measurement".

In the human sciences, much of what we try to measure is subjective (e.g., even concepts like "physical fitness" have fuzzy definitions).  As a result, measurement becomes a difficult and complex issue, and noise is always created in the data due to inaccuracies in the process of measurement.  Thus, it is vital to minimize noise in by using reliable and valid methods of measurement.


Important Aspects of Measurement

Avoid falling prey to "availability bias", which means using whatever measurement tools are on hand, or simply those which others have used previously.  All instruments contain noise, and many contain a lot of noise.  So, conduct a systematic search to identify possible instruments, evaluating each according to relevant criteria, such as:

  • match between the measure and the variable of interest

  • psychometrics

    • structure

    • reliability (see links below)

    • validity (see links below)

  • cohort appropriateness

    • length

    • complexity of structure

    • reading level of instructions & items

    • scaling

  • time

  • cost

    • copyright

    • capital equipment

    • consumables

  • ethical issues

    • sensitive topics

    • invasive application

    • manipulative

    • stressful or imposing

  • other

    • compatible with other measures?

    • test-interaction with IV issues?

    Reading: Bill Trochim:


Recommended Reading

Gall, M. D., Borg, W. R., Gall, J. P. (2003).  Educational research: An introduction. (7th Edition). White Plains, New York: Longman. Recommended: Skim read:

  • Chapter 6: Selecting Sample (pp. 163-186)

  • Chapter 7: Collecting Research Data with Tests and Self-Report Measures (pp. 188-219)

  • Chapter 8 "Collecting Research Data with Questionnaires and Interviews" (pp. 221-252) .