BOWLING ALONE
Page 53
The second worrisome limitation is that the DDB Needham Life Style data come not from random samples of the population, but from a form of quota sampling called “mail panels.” Participants in such surveys—which are frequently used by commercial polling firms—are initially self-selected. Given that the few people who choose to participate might differ significantly from the many who do not, this sampling procedure requires that we consider seriously the possibility of response bias in these data. I have assessed this potential problem in more detail elsewhere, but a brief overview is appropriate here.18
The sampling begins when Market Facts acquires from commercial list brokers the names, addresses, and sometimes demographic characteristics of very large numbers of Americans—from driver’s license bureaus, telephone directories, and many other sources. Large samples from these lists are then invited by mail to express willingness in advance to respond periodically to mail and phone inquiries about commercial products and services, as well as other current issues.19 According to Market Facts officials, the rate of favorable response to such invitations varies across different sectors of the population—from perhaps less than 1 percent among racial minorities and inner-city residents to perhaps 5–10 percent among middle-aged, middle-class “middle Americans.” From this prerecruited “mail panel” (numbering perhaps five hundred thousand at any one time) are then drawn random, demographically balanced samples for the annual DDB Needham Life Style surveys (as well as hundreds of other commercial and other surveys throughout the year).20 Each Life Style respondent is mailed a long written questionnaire that he or she is asked to complete and return within several weeks. At this stage the response rate (roughly 70–80 percent) is typically higher than for conventional random samples. As far as I have been able to ascertain, there has been no substantial change in these procedures over the last two decades, although less careful procedural records have been kept than would be characteristic of comparable academic archives, and in particular, systematic data on the rate of favorable responses to the initial mail invitations are lacking.
Compared with conventional random samples, the mail panel approach has several potential drawbacks.
1. Because the initial recruitment is by mail, literacy in English is an essential requirement, and thus the bottom of the educational ladder is underrepresented, as are non–English speakers.
2. Effective response rates are much lower among racial minorities.21
3. Adults under twenty-five are slightly underrepresented, probably because their mobility makes them harder to track.
Social traits that are especially common in those sections of the population are thus underrepresented in the DDB Needham Life Style sample. In round numbers, the sample contains 10 percent too few high school dropouts, 10 percent too few single respondents, 10 percent too many parents, and half as many racial minorities. Moreover, the sample may also underrepresent the highest and lowest categories of family income. These data reasonably represent the middle 80–90 percent of American society, but they do not well represent ethnic minorities, the very poor, the very rich, and the very transient.22 They may also slightly overrepresent the portion of the public that is most engaged with the mass media. Thus a crucial question about the DDB Needham Life Style survey archive is the degree to which these known sample biases inhibit our ability to estimate social trends from these data.
How accurately do the DDB Needham Life Style data represent trends in American society? In the absence of a full census of social behavior—something that not even the U.S. Census Bureau believes in anymore—the two key questions here are as follows:
1. Do people who join a mail panel differ in substantively relevant ways from people who are willing to respond to conventional surveys?
2. Has the degree of difference between the Life Style panel and conventional surveys changed over time, thus rendering judgments about trends suspect?
If the answer to question 1 is “Yes,” then in some respects the DDB Needham Life Style data may be inaccurate descriptively. Only if the answer to question 2 is also “Yes,” however, will the trends in the Life Style data misrepresent trends as they would appear in a conventional random survey. A constant bias would be disconcerting, to be sure, but only a changing bias would affect our judgments about trends.
With respect to the quality of mail panel respondents, reassuring information is available from several studies that have directly compared results from mail panels and conventional samples. First, apart from the demographic disparity just described (fewer young, poor, and racial minorities in the mail panel), there are surprisingly few differences between the two approaches, even on variables that might be thought to be especially sensitive to the difference in technique. The two different samples do not differ in religious affiliation and religiosity; in public policy views (on tax policy, abortion, gun control); in their views about their own and the nation’s economic circumstances; in their altruism (volunteering, philanthropy) or general “positivity”; in their basic consumer orientations, purchasing habits, ownership or use of common products; in their health or fitness; or in their leisure time. The only significant differences are 1) partisanship (mail panels are slightly less Democratic, probably because of the underrepresentation of racial minorities); and 2) media usage (mail panelists watch slightly more television and read slightly more newspapers).23 That low response rates may not bias substantive results can also be inferred from a recent study that compared results from “easy to reach” and “hard to reach” samples. Aside from clear differences on racial issues, the upshot is that there are no significant differences on other issue stances, on media use, on engagement in daily activities, and on feelings about other people.24
Additional reassurance comes from a comparison between the two most widely reported national surveys of consumer confidence in the United States, one (from the University of Michigan) that relies on conventional random sampling and another (from the Conference Board) that relies on a mail panel. The long-run changes charted by the two methods have been very similar. (The semiannual correlation between the two indexes over more than three decades is R2 = .55.) For fine-grained, month-to-month changes, one or the other of these two surveys might be preferred, but the broad-gauge impressions of annual trends that one would glean from the two are quite similar.
To explore more fully the reliability of DDB Needham Life Style data, I took advantage of the fact that this data set includes more than a dozen diverse questions that are comparable to questions posed on a regular basis over roughly the same time span in the General Social Survey. These measures include attitudes toward feminism, the legalization of marijuana and abortion, views of the Soviet Union, financial worries, military service, basic social values, smoking, video usage, hunting and gun ownership, and (especially relevant to our interests) social trust, church attendance, and leisure activities.25 For each of these items, I posed three tests:
1. Do the levels of response on these variables differ between the two samples, taking into account obvious differences in question wording?
2. Do the trends that one would infer about the underlying trait differ between the two samples?
3. Do the underlying patterns of demographic correlates of these variables differ between the two samples?
As Steven Yonish and I report in detail elsewhere, in every case the answer is “No.”26 For purposes of describing and explaining this wide range of attitudes and behavior, the two surveys are virtually indistinguishable, despite marked differences in sampling (random vs. quota), questioning procedure (personal interviews vs. mail questionnaires), and in some cases question wording. Not only are the trends on all comparable items that I have found virtually identical in the two archives, but the deeper structure of relations between these items and demographic categories is also very similar.
According to the General Social Survey, for example, the probability that in 1990 a thirty-five-year-old single white mother with two years of college educ
ation and a part-time job who rented an apartment in a middle-size New England city favored marijuana legalization is 35 percent, whereas the comparable probability according to the DDB Needham Life Style data was 38 percent, a difference well within sampling error. Similarly, controlling simultaneously for year of survey, year of birth, marital status, employment status, parental status, education, income, race, region of the country, and type of residence, the GSS data suggest that women attend church exactly 5.3 more times per year than men, whereas the DDB Needham Life Style data imply that the difference between the sexes in churchgoing is 4.8 times per year—once again a difference well within sampling error. That the DDB Needham Life Style data pass this very stringent test of comparability with the GSS data—the most scientifically reputable data available on these topics—increases our confidence in the DDB Needham Life Style archive.
Finally, the two archives contain directly comparable questions about a range of leisure activities. Appendix table 1 shows the responses to a series of questions regarding “leisure or recreational activities… done in the past twelve months.” The incidence of these activities in the two surveys was astonishingly similar, well within the limits of sampling error. How
Appendix Table 1: Leisure Activities As Measured in Two National Survey Archives
Leisure Activities During Preceding Twelve Months (1993)
General Social Survey wording DDB Needham Life Style survey wording
GSS Life Style
Went out to see a movie in a theater Went to the movies
72% 70%
Recorded a TV program so you could watch it later Videotaped a TV program on a VCRa
63% 70%
Grew vegetables, flowers, or shrubs in garden Worked in garden
62% 68%
Participated in any sports activity, such as softball, basketball, swimming, golf, bowling, skiing, or tennis
59% 69%
Played softball and/or went swimming and/or played golf and/or went bowling and/or went skiing and/or played tennisb
Attended an amateur or professional sports event Attended a sporting event
56% 56%
Went camping, hiking, or canoeing Went camping and/or went hikingc
44% 44%
Visited art museum or gallery Visited an art gallery or museum
41% 47%
Made art or craft objects, such as pottery, woodworking, quilts, or paintings Worked on a crafts project (needlework, etc.)d
41% 48%
Went hunting or fishing Went hunting and/or went fishing
37% 37%
Played a musical instrument like a piano, guitar, or violin Played a musical instrument
24% 23%
Went to classical music or opera performance Went to classical concert
16% 17%
Attended auto, stock car, or motorcycle race Went to auto racee
16% 9%
a Life Style data are available for 1988–91 only. Figure here is for 1991.
b Since the Life Style questionnaire asked about each of these sports separately, in effect six separate probes were employed. This difference almost certainly inflated the Life Style results, relative to the single GSS question.
c Hiking was included in Life Style surveys in 1975–84 and 1996–97; figure for 1993 was interpolated. Canoeing was never included in Life Style surveys.
d Life Style data are available for 1994–97 only. Figure here is projected for 1993.
e Attendance at auto races was included in Life Style surveys only in 1997, and the figure is used here.
Appendix Table 2: Algorithm for “Annualizing” Estimated Frequencies
GSS Response Alternatives
Imputed Score
DDB Needham Life Style Response Alternatives
Imputed Score
Never
0
None in the past year
0
Less than once a year
0.5
1–4 times
2
Once a year
1
5–8 times
6
Several times a year
6
9–11 times
10
Once a month
12
12–24 times
18
2–3 times a month
30
25–51 times
38
Nearly every week
40
52+ times
54
Every week
52
More than once a week
60
many Americans went to the movies in 1993? GSS says 70 percent, DDB says 72 percent. How about hunting and fishing? GSS says 37 percent, DDB says 37 percent. How about classical concert–going? GSS says 16 percent, DDB says 17 percent. In other words, the profiles of leisure activities represented in the mail panel of the DDB Needham Life Style survey and in the random sample of the General Social Survey were essentially identical.27
In short, just as ice cores, though not infallible, are an invaluable source of information about climatic change, particularly when cross-checked against other measures, for the purposes of estimating basic trends in social participation over the last quarter of the twentieth century the DDB Needham Life Style archive is a valuable source of information, particularly if (as throughout the analyses reported in this book) the results from this archive are consistent with results from other modes of measurement.
Both the General Social Survey and the DDB Needham Life Style surveys typically ask respondents to estimate the frequency of various activities, such as church attendance, but the two surveys use slightly different categories for this purpose. In order to facilitate comparison between these two archives—and, more generally, to simplify presentation of estimated frequencies for various activities—I converted the raw data in each case into estimated annual frequencies, using the algorithm in appendix table 2. Reasonable observers might differ over exactly what “several times a year,” for example, means in quantitative terms, but my basic results are not sensitive to exactly what integers are assigned to the various ranks.28
Another valuable data archive used frequently in this book derives from the Americans’ Use of Time project, managed in recent decades by Professor John Robinson of the University of Maryland, based on careful time diaries kept by national samples of Americans in 1965, 1975, 1985, and 1995. Abundant details about these data are available in Robinson’s book with Geoffrey Godbey, Time for Life: The Surprising Ways Americans Use Their Time.29 One special feature of these data, however, deserves brief mention. One major advantage of this data archive is that it begins in 1965, just about the same time that (other data suggest) various forms of social capital began to decline. However, the 1965 data differed somewhat from the subsequent years, in that the 1965 sample excluded respondents who lived in areas with no city greater than fifty thousand in population, as well as households in which no member aged eighteen to sixty-five was part of the nonagricultural labor force. Since the 1965 sample excluded rural and retired families, the raw figures for that year slightly misrep-resent what would have been found in a national sample that year. To estimate 1965 figures that are more nearly comparable to the later, nationwide data, we adjusted the raw data for 1965, using the observed differences in the 1975 and 1985 surveys between the full national sample and the subset of respondents who would have been included within the 1965 sampling frame. In addition, we weighted the raw data to ensure that each day of the week was equally represented in the final sample. These adjustments account for minor discrepancies between the results presented by Robinson and Godbey and the results reported here.
APPENDIX II
Sources for Figures and Tables
FIGURE NUMBER
TITLE
SOURCE OF DATA
1
Trends in presidential voting (1820–1996), by region
Walter Dean Burnham, unpublished estimates of elector
al turnout. For earlier estimates, see Walter Dean Burnham, “The Turnout Problem,” in Elections American Style, ed. A. James Reichley, (Washington, D.C.: Brookings, 1987), 113—114.
2
Political organizations with regular paid staff, 1977–1996
U.S. Bureau of the Census, County Business Patterns, 1977–1996 (Washington, D.C., various years). U.S. residential population in this and subsequent figures from Statistical Abstract of the United States (Washington, D.C.: U.S. Bureau of the Census, various years).
3
Citizen participation in campaign activities, 1952–1996
National Elections Studies survey archive, 1952–96.
4
Trends in civic engagement I
Roper Social and Political Trends survey archive, 1973–94.
5
Trends in civic engagement II
Roper Social and Political Trends survey archive, 1973–94.
6
Trends in civic engagement III
Roper Social and Political Trends survey archive, 1973–94.
7
The growth of national nonprofit associations, 1968–1997
National nonprofit organizations from Encyclopedia of Associations (Detroit, Mich.: Gale Research, various years), as associations, 1968–1997 reported in Statistical Abstract of the United States (various years).
8
Membership rate in 32 national chapter-based associations, 1900–1997
See appendix III for list of associations and relevant “constituency” for each. Membership data obtained from national headquarters of various associations and annual reports of those organizations, consulted at the Library of Congress, supplemented and confirmed by data from World Almanac (New York: Press Pub. Co. [New York World], various years), Encyclopedia of Associations (Detroit, Mich.: Gale Research, various years), histories of particular organizations (such as Gordon S. “Bish” Thompson, Of Dreams and Deeds [St. Louis: Optimist International, 1989], and Edward E. Grusd, B’nai B’rith: The Story of a Covenant [New York: Appleton-Century, 1966]), and the project on civic engagement directed by Professor Theda Skocpol at Harvard University. I am grateful to Professor Skocpol for exchanging membership data; she bears no responsibility for my interpretation of the data. Membership data for missing years were estimated by linear interpolation. Some organizations typically report membership figures including non-U.S. members, and those non-U.S. members typically constitute a growing fraction of total membership; wherever possible, we excluded such non-U.S. members from the data, in order to focus on trends within the United States. Data on population of underlying constituencies (such as wartime veterans, rural youth, and so on) from published and unpublished data from the U.S. Bureau of the Census, especially the Statistical Abstract of the United States (Washington, D.C.: U.S. Bureau of the Census, various years), and Historical Statistics of the United States: Colonial Times to 1970 (Washington, D.C.: U.S. Bureau of the Census, 1975). Annual market share figures across the 1900–97 period were standardized, and those annual Z-scores were then averaged across all thirty-two organizations to generate figure 8.