by Alan
The essence of designing research to permit causal inference is that the investigator can arrange things so that a single element (independent variable) can be isolated that distinguishes the treatment of two groups. Then, when the groups are later compared on an outcome (dependent) variable, a causal inference becomes inescapable because the groups differed in only one way (their respective form of specific treatment) leading up to the outcome.
In experimental studies, techniques including random assignment to conditions of the IV, alternative activities to occupy the control group's time and efforts, and double-blindness, help ensure as best as possible that the treatment of the experimental and control groups differs only on the one essential element that is of scientific interest.
In non-experimental research, isolating the key differentiating factor between the experience of two groups is much more difficult. We may observe groups in society that appear to differ on Factor A (of interest to us), but they may also differ on Factors B, C, D, etc. For example, we may be interested in how our Factor A, type of school attended (public or private), affects the outcome of standardized test scores. Children who differ on Factor A may also, however, tend to differ on family income (Factor B), neighborhood environment (Factor C), etc. Ultimately, therefore, any difference seen in the final outcome measure between children who attended public and private schools cannot be pinpointed definitively to have been caused by school type (Factor A), because Factors B and C also distinguished the groups' experiences.
Writing on the New York Times "Freakonomics" blog, Justin Wolfers reviews a study by Scott Carrell and Mark Hoekstra on whether the presence of a disruptive child in a classroom can adversely affect the learning and behavior of the other children. To obtain an objective measure of children's likely disruptiveness, Carrell and Hoekstra examined official records looking for children who came from a home in which there was an allegation of domestic violence. Here are some key excerpts from Wolfers's entry (bold emphasis added by me):
Around 70 percent of the classes in their sample have at least one kid exposed to domestic violence. The authors compare the outcomes of that kid’s classmates with their counterparts in the same school and the same grade in a previous or subsequent year — when there were no kids exposed to family violence — finding large negative effects.
Adding even more credibility to their estimates, they show that when a kid shares a classroom with a victim of family violence, she or he will tend to under-perform relative to a sibling who attended the same school but whose classroom had fewer kids exposed to violence. These comparisons underline the fact that the authors are isolating the causal effects of being in a classroom with a potentially disruptive kid, and not some broader socio-economic pattern linking test scores and the amount of family violence in the community.
In a research area such as this, where random-assignment experiments are impossible to conduct, Carrell and Hoekstra have thus done their best to hold everything constant -- the school, the grade level, and, via the sibling component, children's home environment -- so that observed differences in children's school performance can more confidently be attributed to the putative effect of having a disruptive classmate.
Sibling designs are used fairly often in social science research. We hope to provide more in-depth discussion of this approach in our future postings.
Saturday, August 2, 2008
Monday, June 30, 2008
Gun-Law Opinion at the Supreme Court
by Alan
Correlation and causality reached the U.S. Supreme Court last week -- or at least the written dissent of one justice -- as a 5-4 majority interpreted the U.S. Constitution's Second Amendment to confer an individual or personal right to gun ownership, as opposed to only a collective right (i.e., belonging to "a well-regulated militia...").
Cases such as this are supposed to be decided on constitutional issues, in terms of the history and meaning of the document. However, as sometimes happens, policy issues such as whether gun-control laws are good or bad for society find their way into the discourse.
Shown below is a passage from a New York Times article, which quotes Justice Stephen Breyer's attempt to make sense of empirical studies of gun and crime (Breyer's full dissenting opinion is available here).
According to the study, published last year in The Harvard Journal of Law and Public Policy, European nations with more guns had lower murder rates. As summarized in a brief filed by several criminologists and other scholars supporting the challenge to the Washington law, the seven nations with the most guns per capita had 1.2 murders annually for every 100,000 people. The rate in the nine nations with the fewest guns was 4.4.
Justice Breyer was skeptical about what these comparisons proved. “Which is the cause and which the effect?” he asked. “The proposition that strict gun laws cause crime is harder to accept than the proposition that strict gun laws in part grow out of the fact that a nation already has a higher crime rate.”
Whatever positions individuals might take on gun-control legislation, I hope most would agree that careful examination of the direction of causality from inherently correlational studies -- like that exhibited by Breyer -- is a good thing.
Correlation and causality reached the U.S. Supreme Court last week -- or at least the written dissent of one justice -- as a 5-4 majority interpreted the U.S. Constitution's Second Amendment to confer an individual or personal right to gun ownership, as opposed to only a collective right (i.e., belonging to "a well-regulated militia...").
Cases such as this are supposed to be decided on constitutional issues, in terms of the history and meaning of the document. However, as sometimes happens, policy issues such as whether gun-control laws are good or bad for society find their way into the discourse.
Shown below is a passage from a New York Times article, which quotes Justice Stephen Breyer's attempt to make sense of empirical studies of gun and crime (Breyer's full dissenting opinion is available here).
According to the study, published last year in The Harvard Journal of Law and Public Policy, European nations with more guns had lower murder rates. As summarized in a brief filed by several criminologists and other scholars supporting the challenge to the Washington law, the seven nations with the most guns per capita had 1.2 murders annually for every 100,000 people. The rate in the nine nations with the fewest guns was 4.4.
Justice Breyer was skeptical about what these comparisons proved. “Which is the cause and which the effect?” he asked. “The proposition that strict gun laws cause crime is harder to accept than the proposition that strict gun laws in part grow out of the fact that a nation already has a higher crime rate.”
Whatever positions individuals might take on gun-control legislation, I hope most would agree that careful examination of the direction of causality from inherently correlational studies -- like that exhibited by Breyer -- is a good thing.
Tuesday, June 24, 2008
Announcement: Causality Study Fortnight, in the UK
The following message was sent to the SEMNET listserve discussion group:
CAUSALITY STUDY FORTNIGHT
http://www.kent.ac.uk/reasoning/Csf/
8-19 September 2008
CENTRE FOR REASONING
University of Kent, UK
8-9 September: 2 days of tutorials on causality, probability and their use in science.
10-12 September: CAPITS 2008 a 3-day conference on causality and probability in the sciences.
15-19 September: a week of advanced research seminars on causality and probability.
The final programme and the book of abstracts are now available on the CSF
website.
For further information email f.russo@kent.ac.uk
CAUSALITY STUDY FORTNIGHT
http://www.kent.ac.uk/reasoning/Csf/
8-19 September 2008
CENTRE FOR REASONING
University of Kent, UK
8-9 September: 2 days of tutorials on causality, probability and their use in science.
10-12 September: CAPITS 2008 a 3-day conference on causality and probability in the sciences.
15-19 September: a week of advanced research seminars on causality and probability.
The final programme and the book of abstracts are now available on the CSF
website.
For further information email f.russo@kent.ac.uk
Tuesday, June 3, 2008
Summer Workshop at Claremont Graduate University
At the Claremont Graduate University's summer workshops coming up this August, Michael Scriven will be teaching one entitled, "The Logic of Evaluation and Causality: What Makes What Works Work?"
Thursday, May 22, 2008
Announcement: Symposium on Causality, in Germany
The following message was sent to the SEMNET listserve discussion group:
Dear colleagues,
We would like to kindly invite you to the Symposium on Causality 2008, scheduled for July 17th to 19th in Dornburg (near Jena), Germany. The symposium brings together different traditions of analysis of causal effects (regression-based analyses, analyses based on propensity scores, analyses with instrumental variables) to discuss the state-of-the-art in the analysis of causal effects, with a special focus on non-standard designs and problems (missing data, non-compliance, multilevel designs, regression discontinuity designs).
The symposium will be structured along seven focus presentations by leading proponents in different fields of causality research. Each focus presentation will be discussed and supplemented by two invited discussants, followed by an open discussion among all participants. Focus presentations will be given by Donald B. Rubin, Thomas D. Cook, William R. Shadish, Rolf Steyer, Steven G. West, Christopher Winship and Michael E. Sobel.
There will also be ample room for participants to present and discuss their research during the symposium. Participants who want to present their research findings are asked to register for the symposium no later than June 15 and submit a title and an abstract for their presentation together with their registration. The mode of presentation (oral presentation or poster) will be determined by the organization committee depending on the total number and quality of the submissions. Other participants should register no later than June 29.
The registration fee for the symposium is 80 Euros including a daily bus transfer from Jena to Dornburg and refreshments during the conference. You can also register for the conference dinner for additional 30 Euros. To register, please visit our webpage [English, German], where you can also find additional information about the contents and structure of the symposium. If you have any questions do not hesitate to contact us.
We hope to see you soon in Jena!
Rolf Steyer and Benjamin Nagengast
Dear colleagues,
We would like to kindly invite you to the Symposium on Causality 2008, scheduled for July 17th to 19th in Dornburg (near Jena), Germany. The symposium brings together different traditions of analysis of causal effects (regression-based analyses, analyses based on propensity scores, analyses with instrumental variables) to discuss the state-of-the-art in the analysis of causal effects, with a special focus on non-standard designs and problems (missing data, non-compliance, multilevel designs, regression discontinuity designs).
The symposium will be structured along seven focus presentations by leading proponents in different fields of causality research. Each focus presentation will be discussed and supplemented by two invited discussants, followed by an open discussion among all participants. Focus presentations will be given by Donald B. Rubin, Thomas D. Cook, William R. Shadish, Rolf Steyer, Steven G. West, Christopher Winship and Michael E. Sobel.
There will also be ample room for participants to present and discuss their research during the symposium. Participants who want to present their research findings are asked to register for the symposium no later than June 15 and submit a title and an abstract for their presentation together with their registration. The mode of presentation (oral presentation or poster) will be determined by the organization committee depending on the total number and quality of the submissions. Other participants should register no later than June 29.
The registration fee for the symposium is 80 Euros including a daily bus transfer from Jena to Dornburg and refreshments during the conference. You can also register for the conference dinner for additional 30 Euros. To register, please visit our webpage [English, German], where you can also find additional information about the contents and structure of the symposium. If you have any questions do not hesitate to contact us.
We hope to see you soon in Jena!
Rolf Steyer and Benjamin Nagengast
Tuesday, May 6, 2008
Special Series of Articles in Developmental Psychology
by Alan
The March 2008 issue of Developmental Psychology contains a special series of around 15 methodolocially and statistically oriented articles (Table of Contents). Three of the articles explicitly refer in their titles to causal inference, and others of the articles may have relevant ideas, as well. The three titles mentioning causation are as follows:
From statistical associations to causation: What developmentalists can learn from instrumental variables techniques coupled with experimental data (Gennetian, Magnuson, & Morris)
Using full matching to estimate causal effects in nonexperimental studies: Examining the relationship between adolescent marijuana use and adult outcomes (Stuart & Green)
Combining group-based trajectory modeling and propensity score matching for causal inferences in nonexperimental longitudinal data (Haviland, Nagin, Rosenbaum, & Tremblay)
At this stage, I have only skimmed through these (and other) articles in the issue. The techniques of "instrumental variables" and "matching" have, of course, been around for many years. I will be interested to see in greater depth what new contributions these articles make with such established techniques. Only within the past six months did I first hear the term "propensity score;" in skimming the many articles in the issue that use propensity scores, however, I've learned that this approach, too, has been around for decades!
Causal inference from nonexperimental data clearly is a complex, tricky endeavor. Perhaps it is for this reason that the kinds of techniques discussed in the special series have needed a quarter-century or longer to be absorbed, tested in different contexts, and diffused across disciplines.
The March 2008 issue of Developmental Psychology contains a special series of around 15 methodolocially and statistically oriented articles (Table of Contents). Three of the articles explicitly refer in their titles to causal inference, and others of the articles may have relevant ideas, as well. The three titles mentioning causation are as follows:
From statistical associations to causation: What developmentalists can learn from instrumental variables techniques coupled with experimental data (Gennetian, Magnuson, & Morris)
Using full matching to estimate causal effects in nonexperimental studies: Examining the relationship between adolescent marijuana use and adult outcomes (Stuart & Green)
Combining group-based trajectory modeling and propensity score matching for causal inferences in nonexperimental longitudinal data (Haviland, Nagin, Rosenbaum, & Tremblay)
At this stage, I have only skimmed through these (and other) articles in the issue. The techniques of "instrumental variables" and "matching" have, of course, been around for many years. I will be interested to see in greater depth what new contributions these articles make with such established techniques. Only within the past six months did I first hear the term "propensity score;" in skimming the many articles in the issue that use propensity scores, however, I've learned that this approach, too, has been around for decades!
Causal inference from nonexperimental data clearly is a complex, tricky endeavor. Perhaps it is for this reason that the kinds of techniques discussed in the special series have needed a quarter-century or longer to be absorbed, tested in different contexts, and diffused across disciplines.
Monday, April 7, 2008
Note from Les Hayduk
Commentary continues to come in on the criteria for causality. This latest note is from Les Hayduk. He has agreed to my reprinting of these lightly edited comments, which he originally posted in full to the SEMNET discussion forum on Saturday, April 5, 2008, with the subject heading: “correlation-causality blog – improvements.” Dr. Hayduk requests that continuing discussion of his comments take place on the SEMNET forum (click here for an introduction to SEMNET). – Alan
I had a look at the blog Alan provided (see below) and found this easily readable, traditional, and in some ways extremely UN-helpful. I will pick up on two of the things that seem standard, but that slant people's thinking in ways that are unhelpful, and hence where I see improvements are possible. The two matters I will take on are: experiments as the supposed benchmark/gold-standard against which SEM is to be evaluated (I doubt this), and the conditions for causality (2 of the 3 traditional conditions are wrong, the third is imprecise).
There is some substantial artificiality of comparing single experiments and single SEM studies, but I skip this for the moment, though I suspect it may eventually become an important matter.
Some failings of experiments: 1) random assignment of cases (say people) to groups should result in the groups being SIGNIFICANTLY different on 5 out of every 100 characteristics, in the long run. (SEM analysis of the experimental data can include potentially problematic variables, to see if they happen to be among the 5%, if the experimenters are not too proud to combine experiments with SEM.)
2) Experiments minimize, but do NOT statistically control for any remaining measurement error. Such control can and should be done by SEM statistics. This is NOT a feature of the experiment, but involves the statistics that could be connected to the experiment. Notice that comparing experiments and SEM is implicitly comparing two different things: the methods, and the statistics that usually go along with the methods.
SEM should be used IN CONJUNCTION WITH experimentation, so I see Alan as (possibly unknowingly) working against the helpful combining of SEM with experimentation.
Within a single experiment the mechanisms of action WITHIN the study are usually NOT well-investigated with experiments, but can be much better done in a single SEM (via inclusion of indicators of the appropriate/anticipated intervening causal structures).
Model testing is LESS well done in experiments than in SEM. SEM has an overall model test, and experiments do not usually have a comparable test (if ANOVA, or regression, or mean-differences are used as the statistical procedures). These procedures provide parameter tests parallel to those in SEM, but they have no parallel to the OVERALL MODEL TEST in SEM. Often experimenters are not even aware that they do not have an overall model test comparable to SEM's overall model test.
Enough on this for now, so I will move to the criterion for causality. Here is a quote from Alan's blog:
…contemporary SEM practitioners would probably be more comfortable with suggestions of causation if the data were collected longitudinally (more specifically, with a panel design, in which the same respondents are tracked over time). Of the three major criteria for demonstrating causality, longitudinal studies are clearly capable of demonstrating correlation and time-ordering; provided that the most plausible “third variable” candidates are measured and controlled for, the approximation to causality should be good…
Time sequence is NOT required for causation. Causes can go “both ways simultaneously” – there are such things as reciprocal causes (e.g. Rigdon, 1995, Multivariate Behavioral Research 30(3): 359-383) and variables can even cause themselves (for example, see my 1996 book chapter 3, or Hayduk, 1994, Journal of Nonverbal Behavior 18:245-260).
Correlation is NOT required. Suppressor effects can result in a variable causing another variable, and yet other parts of the causal system can counteract the causal covariance contribution, so the covariance between the variables is zero. (See Duncan, 1975, Introduction to Structural Equation Models, page 29 [equation for Greek-ro-subcript23] and realize that one term in the equation can be positive and the other of equal-magnitude yet negative.)
[Moderator’s note: See also Dean Keith Simonton’s discussion of suppressor variables and causal inference, in the posting immediately below.]
“Third variable” control should refer to MANY variable control – there can be many common causes, and many correlated causes that influence the two variables, and even reciprocal effects where the jargon of “third variables” is not quite correct. The full causal structure should be attended to, including misplacement of causally downstream variables to upstream locations. The issue here is the full proper causal specification of the model, not something connected to just third variables.
I notice you mentioned Judea Pearl's work. [Interested readers are encouraged to] have a look at the SEMNET archive for the comments Judea Pearl provided to SEMNET some years back [and] discussion of some of Pearl's work in SEM 2003, 10(2):289-311, which was designed to help SEM people understand one part of Pearl's book that directly connects to SEM.
I had a look at the blog Alan provided (see below) and found this easily readable, traditional, and in some ways extremely UN-helpful. I will pick up on two of the things that seem standard, but that slant people's thinking in ways that are unhelpful, and hence where I see improvements are possible. The two matters I will take on are: experiments as the supposed benchmark/gold-standard against which SEM is to be evaluated (I doubt this), and the conditions for causality (2 of the 3 traditional conditions are wrong, the third is imprecise).
There is some substantial artificiality of comparing single experiments and single SEM studies, but I skip this for the moment, though I suspect it may eventually become an important matter.
Some failings of experiments: 1) random assignment of cases (say people) to groups should result in the groups being SIGNIFICANTLY different on 5 out of every 100 characteristics, in the long run. (SEM analysis of the experimental data can include potentially problematic variables, to see if they happen to be among the 5%, if the experimenters are not too proud to combine experiments with SEM.)
2) Experiments minimize, but do NOT statistically control for any remaining measurement error. Such control can and should be done by SEM statistics. This is NOT a feature of the experiment, but involves the statistics that could be connected to the experiment. Notice that comparing experiments and SEM is implicitly comparing two different things: the methods, and the statistics that usually go along with the methods.
SEM should be used IN CONJUNCTION WITH experimentation, so I see Alan as (possibly unknowingly) working against the helpful combining of SEM with experimentation.
Within a single experiment the mechanisms of action WITHIN the study are usually NOT well-investigated with experiments, but can be much better done in a single SEM (via inclusion of indicators of the appropriate/anticipated intervening causal structures).
Model testing is LESS well done in experiments than in SEM. SEM has an overall model test, and experiments do not usually have a comparable test (if ANOVA, or regression, or mean-differences are used as the statistical procedures). These procedures provide parameter tests parallel to those in SEM, but they have no parallel to the OVERALL MODEL TEST in SEM. Often experimenters are not even aware that they do not have an overall model test comparable to SEM's overall model test.
Enough on this for now, so I will move to the criterion for causality. Here is a quote from Alan's blog:
…contemporary SEM practitioners would probably be more comfortable with suggestions of causation if the data were collected longitudinally (more specifically, with a panel design, in which the same respondents are tracked over time). Of the three major criteria for demonstrating causality, longitudinal studies are clearly capable of demonstrating correlation and time-ordering; provided that the most plausible “third variable” candidates are measured and controlled for, the approximation to causality should be good…
Time sequence is NOT required for causation. Causes can go “both ways simultaneously” – there are such things as reciprocal causes (e.g. Rigdon, 1995, Multivariate Behavioral Research 30(3): 359-383) and variables can even cause themselves (for example, see my 1996 book chapter 3, or Hayduk, 1994, Journal of Nonverbal Behavior 18:245-260).
Correlation is NOT required. Suppressor effects can result in a variable causing another variable, and yet other parts of the causal system can counteract the causal covariance contribution, so the covariance between the variables is zero. (See Duncan, 1975, Introduction to Structural Equation Models, page 29 [equation for Greek-ro-subcript23] and realize that one term in the equation can be positive and the other of equal-magnitude yet negative.)
[Moderator’s note: See also Dean Keith Simonton’s discussion of suppressor variables and causal inference, in the posting immediately below.]
“Third variable” control should refer to MANY variable control – there can be many common causes, and many correlated causes that influence the two variables, and even reciprocal effects where the jargon of “third variables” is not quite correct. The full causal structure should be attended to, including misplacement of causally downstream variables to upstream locations. The issue here is the full proper causal specification of the model, not something connected to just third variables.
I notice you mentioned Judea Pearl's work. [Interested readers are encouraged to] have a look at the SEMNET archive for the comments Judea Pearl provided to SEMNET some years back [and] discussion of some of Pearl's work in SEM 2003, 10(2):289-311, which was designed to help SEM people understand one part of Pearl's book that directly connects to SEM.
Subscribe to:
Posts (Atom)