Richard L Epstein
Page 37
150? Why is it that opinion polls regularly extrapolate to the preferences of all
voters in the U.S. from sampling 1,500 or less?
Roughly, the idea is to measure how much more likely it is that your
generalization is going to be accurate as you increase the number in your sample.
If you want to find out how many people in your class of 3 0 0 sociology students are
spending 15 hours a week on the homework, you might ask 15 or 2 0 . If you
interview 30 you might get a better picture, but there's a limit. After you've asked
100, you probably won't get a much different result if you ask 150. And if you've
asked 2 0 0 , do you really think your generalization will be different if you ask 2 5 0 ?
It hardly seems worth the effort.
CHAPTER 14 Generalizing
Often you can rely on common sense when small numbers are involved. But
when we generalize to a very large population, say 2,000, or 20,000, or 200,000,000,
how big the sample should be cannot be explained without at least a mini-course on
statistics. In evaluating statistical generalizations, you have to expect that the people
doing the sampling have looked at enough examples, which is reasonable if it's a
respected organization, a well-known polling company, physicians, or a drug
company that has to answer to the Food and Drug Administration. Surprisingly,
1,500 is typically adequate for the sample size when surveying all adults in the U.S.
2. Is the sample studied well?
Choosing a large enough representative sample is important, but it's not enough.
The sample has to be investigated well.
The doctor taking your blood to see if you have diabetes won't get a reliable
result if her test tube is contaminated or if she forgets to tell you to fast the night
before. You won't find out the attitudes of students about sex before marriage if you
ask a biased question. Picking a random sample of bolts won't help you determine if
the bolts are O.K. if all you do is inspect them visually, not with a microscope or a
stress test.
Questionnaires and surveys are particularly problematic. Questions need to be
formulated without bias.
• At the bottom of the barrel: Issues entrepreneurs cared about least in the election
1. Electric utility deregulation, 2%
2. Superfund reform, 3%
3. Pension simplification and reform, 9%
4. Estate tax reform, 12% List of the Week from
5. Product liability and tort reform, 24% Arthur Andersen
I What questions did they ask? Wouldn't windshield wiper standardization laws
have ranked lower?
Even then, we have to rely on respondents answering truthfully. Surveys on sexual
habits are notorious for inaccurate self-reporting. Invariably, the number of times
that women in the U.S. report they engaged in sexual intercourse with a man in the
last week, or month, or year is much lower than the reports that men give of sexual
intercourse with a woman during that time. The figures are so different that it would
be impossible for both groups to be answering accurately.
3. Three premises needed for a good generalization
A generalization is an argument. You need to examine it as you would any
argument: Does the argument rely on slanted or vague language? What unstated
premises are missing? Do you have good reason to believe the premises? Does the
conclusion follow from the premises? For some generalizations you will have to rely
SECTION C When is a Generalization Good! 289
on "the experts" for whether to believe the premises, which include "The sample is
representative," "The sample is big enough," "The sample was studied well,"
whether stated or not. Even if you have a degree in statistics, you will rarely have
access to the information necessary to evaluate those premises.
Premises needed for a good generalization
• The sample is representative.
• The sample is big enough.
• The sample is studied well.
But you could choose a big enough representative sample, study it well, get a
trustworthy generalization, and still have a lousy argument.
Dick: A study I read said that people with large hands are better at math.
Suzy: I guess that explains why 1 can't divide!
You don't need a study to know that people with large hands do better at math: I
Babies have small hands, and they can't even add. The collection of all people |
is the wrong population to study.
4. The margin of error and confidence level
It's never reasonable to believe exact statistical generalizations: 37% of the people in
your town who were surveyed wear glasses, so 37% of all people in your town wear
glasses. No matter how many people in your town are surveyed, short of virtually all
of them, you can't be confident that exactly 37% of all of them wear glasses. Rather,
"37%, more or less, wear glasses" would be the right conclusion.
That "more or less" can be made fairly precise according to a theory of
statistics. The margin of error tells us the range within which the actual number for
the population is likely to fall. How likely is it that they're right? The confidence
level measures that. For example,
The opinion poll says that when voters were asked their preference, the incum-
bent was favored by 53% and the challenger by 47%, with a margin of error of
2%, and a confidence level of 95%. So the incumbent will win tomorrow.
From this survey they are concluding that the percentage of all voters who
favor the incumbent is between 51% and 55%, while the challenger is favored by
between 45% and 49%. The confidence level is 95%. That means there's a 95%
chance it's true that the actual percentage of voters who prefer the incumbent is
between 51% and 55%. If the confidence level were 60%, then the survey wouldn't
be very reliable: There would be a 4-out-of-10 chance that the conclusion is false,
given those premises.
290 CHAPTER 14 Generalizing
The confidence level, then, measures the strength of the generalization as an
argument. Typically, if the confidence level is below 95%, the results won't even
be announced. To summarize for the example above:
Margin of error ± 2% gives the range around 53% in which it is likely
that the population value lies. It is part of the conclusion: Between 51%
and 55% of all voters favor the incumbent.
Confidence level of 95% says exactly how likely it is that the population
value lies in that range. It tells you how strong the generalization is.
The bigger the sample, the higher the confidence level and the lower the
margin of error. The problem is to decide how much it's worth in extra time and
expense to increase the sample size in order to get a better argument.
So will the incumbent win? The generalization that a majority of voters at the
time of polling favor the incumbent is strong. But to conclude the incumbent will
win depends on what happens from the time of the polling to the voting. It depends
on how fixed people are in their opinions and on a lot of other unstated premises.
5. Variation in the population
Dick: It takes me forever to download anything on the Internet.
Tom: But you've got a PowerMac G5 just like mine.
Dick: Yeah, but I only have a dial-up connection.
Tom: Get on to that high-speed service that Clurbach Internet is
providing. I've got it and it's super fast. I can download a song
from the Apple site in less than a minute.
Tom is generalizing. His conclusion is that any other computer like his on the same
kind of Internet connection will be as fast. But isn't that a hasty generalization?
No. Tom's generalization is good, because any other computer like his (that's
in running order) with the same Internet connection should perform exactly as his.
They're all supposed to be the same.
How big the sample has to be depends on how much variation there is in the
population. If there is very little variation, then a small sample chosen haphazardly
will do. Lots of variation demands a very large sample, and random sampling is the
best way to get a representative sample.
6. Risk
With a shipment of 30 bolts, inspecting 15 of them and finding all of them O.K.
would allow you to conclude that all the bolts are O.K. But if they're for the space
shuttle, where a bad bolt could doom the spacecraft, you'd want to inspect each and
every one of them.
On the other hand, suppose that for the first time you try eating a kumquat.
SECTION C When is a Generalization Goodl 291
Two hours later you get a stomachache, and that night and the next morning you
have diarrhea. I'll bet you wouldn't eat a kumquat again. But the argument from
this one experience that kumquats will always do this to you is pretty weak—
it could have been something else you ate, or a twenty-four-hour flu, or . .. .
Risk doesn't change how strong an argument you have, only how
strong an argument you want before you'll accept the conclusion.
7. Analogies and generalizations
Analogies are not generalizations, but they often require a generalization as premise.
The analysis of analogies usually ends in our trying to come up with a general
claim that will make a valid or strong argument. Analogies lead to generalizations.
This car is like that one. They both had bad suspension. And here's another from
the same manufacturer, which the owner says has bad suspension, too. So if you buy
one of these cars, it will have bad suspension, too. From two or three or seventeen
examples, you figure that the next one will be the same. That's an analogy, all right,
but the process is more one of generalization, for it's the unspoken general claim that
needs to be proved: (Almost) all cars from this manufacturer have bad suspension.
Summary We generalize all the time: From a few instances (the sample) we conclude
something about a bigger group (the population). Generalizations are arguments.
They need three premises to be good: the sample is representative; the sample is big
enough; the sample is studied well.
Often we can figure out whether these premises are true. But it's harder for
large populations with a lot of variation. The best way to ensure that a sample is
representative is to choose it randomly. Haphazardly chosen samples are often used,
but we have no reason to believe a sample chosen haphazardly is representative.
With polls and scientific surveys we usually have to decide whether to believe
the experts. They should tell us the margin of error and confidence level. We can
develop some sense of when a generalization is good or bad. Our best guide is to
remember that a generalization is an argument, and all we've learned about
analyzing arguments applies.
Key Words generalization random sampling
population law of large numbers
sample gambler's fallacy
inductive evidence hasty generalization
statistical generalization anecdotal evidence
representative sample margin of error
biased sample confidence level
haphazard sampling variation in a population
292 CHAPTER 14 Generalizing
Exercises for Chapter 14
1. Your candidate is favored by 56% to 44%, with a margin of error of 5% and a
confidence level of 94%. What does that mean?
2. You read a poll that says the confidence level is 71%. Is the generalization reliable?
3. a. What do we call a weak generalization from a sample that is obviously too small?
b. Can a sample of one ever be enough for a strong generalization?
4. The larger the in the population, the larger the sample size must be.
5. What premises do we need for a good generalization?
6. a. You're at the supermarket trying to decide which package of strawberries to buy.
Describe and evaluate your procedure as a sampling and generalizing process
(of course you can't actually taste one).
b. Now do the same supposing the package is covered everywhere but on top.
7. Suppose you're on the city council and have to decide whether to put a bond issue for
a new school on the next ballot. You don't want to do it if there's a good chance it will
fail. You decide to do a survey, but haven't time to get a polling agency to do it.
There are 7,200 people in your town. How would you go about picking a sample?
8. The president of your college would like to know how many students approve of the
way she is handling her job. Explain why no survey is going to give her any useful
ideas about how to improve her work.
9. The mayor of a town of 8,000 has to decide whether to spend town funds on renovating
the park or hiring a part-time animal control officer. She gets a reputable polling
organization to do a survey.
a. The results of the survey are 52% in favor of hiring an animal control officer and
47% in favor of renovating the park, with 1% undecided, and a margin of error of
3%. The confidence level is 98%. Which choice will make the most people happy?
Should she bet on that?
b. The results are 6 1 % in favor of hiring an animal control officer and 31% in favor
of renovating the park, with 8% undecided, and a margin of error of 9%. The
confidence level is 94%. Which choice will make the most people happy?
Should she bet on that?
EXERCISES for Chapter 14 293
10. A "Quality of Education Survey" was sent out to all parents of students at Socorro High School (Socorro, NM) for the school year 2000-2001. Of 598 forms sent out, 166 were
returned. For one of the issues the results were:
My child is safe at school 6% (10 forms) strongly agreed, 42.8% (71) agreed,
28.9% (48) disagreed, 13.9% (23) strongly disagreed, 7.8% (13) did not know,
and 0.6% (1) left the question blank.
What can you conclude?
11. Flo to Dick: I talked to all the people who live on this street, and everyone who has a
dog is really happy. So if I get my mom a dog, she'll be happy, too.
How should Dick explain to Flo that she's not reasoning well?
Here are some of Tom's attempts to use the ideas from this chapter.
Maria: Every time I've seen a stranger come to Dick's gate, Spot has barked. So Spot
will always bark at strangers at Dick's gate.
Generalization (state it; if none, say so): Spot will bark at every stranger who
comes to the gate.
Sample: All the times Maria has seen a stranger come to the gate.
Sample is representative? (yes or no) Who knows?
Sample is big enough? (yes or no) No
.
Sample is studied well! (yes or no) Yes—Maria knows if Spot barked when
she was there.
Additional premises needed:
Good generalization! No. The sample isn't good.
You almost got it. Ihe generalization shouldn't convince, you— that's right. But the
problem isn't that the sample isn't "good," but that Maria hasn't given any reason to
believe that it's big enough and representative. Is "every time" once? 'Twice? I5O times?
And are those times representative? It's enough that you have no reason to believe that
the sample is representative to make this a bad generalization, that is, a bad argument.
In a study of 5,000 people who owned pets in Anchorage, Alaska, dog owners expressed
higher satisfaction with their pets and their lives. So dog owners are more satisfied with
their pets and their own lives.
Generalization (state it; if none, say so): Dog owners are more satisfied with
their pets and their own lives.
Sample: The people surveyed.
Sample is representative! No.
Sample is big enough! Don't know.
Sample is studied well! Not sure—I don't know what questions were asked.
Additional premises needed:
Good generalization! No. The sample isn't good.
Rjght. Once you note that the sample isn't representative, you know immediately that
the argument isn't good.
294 CHAPTER 14 Generalizing
Every time the minimum wage is raised, there's squawking that it will cause inflation
and decrease employment. And every time it doesn't. So watch for the same worthless
arguments again this time.
Generalization (state it; if none, say so): Raising the minimum wage won't
cause inflation and decrease employment.
Sample: Every time in the past that the minimum wage was raised.
Sample is representative? Yes.
Sample is big enough? Yes—it was all the times before.
Sample is studied well? Yes—assuming the speaker knows what she's
talking about.
Additional premises needed: None.
Good generalization? Yes.
The sample is big enough, since it can't get any bigger. But is it representative? Is there
any reason to thinkthat the situation now is like the situations in the past when the
minimum wage was raised? It's like. an analogy: This time is like. the past times. Until
the speaker fids that in, we shouldn't accept the conclusion.