Two studies published in the Journal of Clinical Oncology in January 2018 probed the same question: Which factors are the main contributors to disparities in cancer survival?
Though the studies shared a common aim, they reached different conclusions. One attributed survival differences to the stage at which patients were diagnosed. The other concluded insurance status is the primary factor involved.
Alongside these studies, the Journal ran an editorial that unpacked these divergent findings. The piece, written by University of Michigan oncologists Dr. Lauren Wallner and Dr. Jennifer Griggs, explains the methodological differences between the two papers, which highlight broader challenges in the field.
We spoke with Dr. Wallner to get her advice on how journalists might reconcile different research findings and how best to conceptualize the growing field of disparities research.
This interview has been edited and condensed.
What should we make of studies on similar topics with disparate findings?
You’re going to expect in any research study where you’ve got different populations, you’ve got different information that you have available or that you’re collecting, there’s going to be variation in the results that you find. Because across study settings, across methodologies, you’re not doing the exact same thing over and over again … It’s something to be addressed, absolutely, and I think it’s something we could do a better job of.
What about the studies you addressed in your editorial? Are both findings valid?
I think what you would take away from it is it’s a multifactorial issue. These are all things that are contributing to the difference in outcomes that we’re seeing in these populations, and I think which one sort of floats to the top or ends up being the strongest influencer of those outcomes in either of these studies depends in large part on the population that they sampled, the way they defined their variables, the analytic approach that they took.
I think if you step back and take a 30,000-foot view, what you see is that the factors that we tend to think of as important in influencing cancer outcomes are what they’ve found in these studies. It’s not anything that’s completely novel or groundbreaking or completely different from what we would expect, based on prior literature. Both of these studies used rigorous methods, they used great databases that we often use, but I think the differences speak to the methodologies that they used … there’s a variety of reasons why these differences in results can be explained, but stepping back and looking at disparities research as a whole, I think to strengthen disparities research methods, that, hopefully, will ultimately reduce the disparate research findings that we have.
How should journalists report on conflicting findings?
To the extent that you can report on what you think is driving the differences in those findings, and typically if there’s an editorial written, or oftentimes in the discussions of papers, authors will compare their research to other literature that exists in the area and try to describe it and extrapolate on that — to any extent that you can highlight why they might be different, I think that’s important. But again, taking that 30,000-foot view, look at it and the message largely remains the same — there’s all these important factors that are driving disparities, and some of which we can absolutely try and address, with intervention in those populations, for example, and some driven by clinical characteristics and things like that.
[…]
Journalists can also think about, there are quite a few frameworks out there now that are supposed to be guiding how we report our research results … these really call for granular information to be reported in the methods, and I think having those guidelines handy and taking a look at those to make sure the studies that journalists are looking at with a critical eye are up to those standards, is important. … It helps you to critically appraise a paper… particularly in a situation where you’ve got two studies that have slightly different results, this helps you go through the nuances of the methods and try to think about where those differences may lie or may be explained apart.
And I think just conservatively interpreting results of one single research study is always something that I caution people to do. I think there’s always a tendency to overextrapolate the meaning of your research results in one study population in one setting with one set of methodology, and so part of it is looking at the body of literature as a whole, in that topic area, and thinking about the level of evidence. … This is a huge issue in the dissemination of research findings across settings, across audiences. … And as investigators, we’re guilty of this as well, where we will report relative measures of association, without reporting the absolute differences, and so relative measures of association may look like there’s a huge difference, but then when you actually look at the absolute difference, it’s maybe not clinically meaningful. So I think reporting both is important. And I think that’s absolutely the case in any sort of press … particularly, I think, for a lay audience, I think it’s really important, and it’s something I think we can all do a much better job of.
Is the issue of reproducibility unique to disparities research?
No. I think it’s an aspect in most medical research … There are definitely strides being made to increase the reproducibility of results, and that involves more detailed information about how you’re specifying variables, and your analyses, and things so that others can replicate them in different settings, different populations, etc.
Are there particular challenges to producing reproducible results in disparities research?
In disparities research, I think the issue starts with the word ‘disparity.’ What do we consider to be a disparity and how are we defining a disparity? I think there’s a lot of variety around that. So that contributes to the issue. And I think that in disparities research, oftentimes you’re assessing differences across multiple levels, so both the individual level — looking at a person’s socioeconomic status or position, for example — but then also at the neighborhood level or contextual level — looking at the areas in which they live and factors that you can collect related to that.
So I think it gets complicated quickly, and oftentimes what happens is we, as disparities investigators, only have limited information available to us. These studies that we’re using, or these registries, for example, we focus on looking at differences across race or ethnicity or some combination of variables too. We look at individual-level socioeconomic status or something like that, [and] we don’t have enough information on all these other factors, so we are sort of in a sense ignoring those. Right? And so I think that contributes to it as well.
I think we, as disparities researchers, definitely can do a better job of trying to pick out databases that have more comprehensive information, merging databases together that maybe have pieces of it, that together can provide a more comprehensive picture. And we, when we ourselves are doing primary data collection, making sure that we’re collecting information on multiple levels on a variety of measures related to socioeconomic position, for example.
And then also just not narrowing our definition of what we think a disparity is, to just be something based on race or ethnicity, or socioeconomic position … I think ‘disparities’ is a very broad term that is used in a variety of different ways, and I think that sometimes that can contribute to the issue of having different results across different studies.
Why is it important to broaden the definition of disparities?
We need to think about disparities in a broader sense, that, spanning characteristics, may at least in this health context, put populations or individuals at risk of poorer outcomes. So that could be anything from, you know, the more traditional things we just talked about, which are typically talked about — race, ethnicity, socioeconomic position, their geographic location, their neighborhood, those contextual factors — but it could also be things like different health characteristics they have, whether they have other chronic health conditions, or they’re obese, or gender and sexual minorities. So in my mind, I think it’s a much broader definition, and I think it’s something that, going forward in disparities research, particularly as it relates to health outcomes, we need to do a better job of assessing, and not taking this narrow focus on just the typical race and SES [socioeconomic status] variables that we might be looking at.
… I do think it gets a little bit difficult when you’re publishing medical studies, because you’re limited in the word count that you can use. But, oftentimes, studies are lacking a conceptual framework to guide how they’re going to define the variables. So I think that is an easy step … I think most researchers think about that and consider that. But oftentimes it’s not reported, so it’s hard to tell when you’re reading studies, to understand the nuance of how they are actually defining their variables and what conceptual framework went into shaping that.
What do you think of the prospects of standardizing variables across studies?
I think it largely depends in part on the data. So oftentimes, we’re limited by the data that we have, and so one of the upstream solutions to this issue is trying to do research in more diverse populations. And not just diversity across race, ethnicity, but also socioeconomic status, urban versus rural, gender and sexual orientation. So I think the more diversity that you have, with that comes the ability to look at the granular level at some of these variables that I think often are ignored. …
So I think the solution is really to try to think critically about how to sample more diverse populations, and how to make sure that we’re doing research in more diverse populations, across all of these constructs, because at the end of the day, you want to have generalizable results, you want to make them such that they are rigorous, reproducible. And doing it in a population that’s largely the same across any of these constructs is really going to limit that.
How do biological factors like genetics play into all of this?
I think there’s room for it all and I would say that increasingly we are seeing studies on the genetic underpinnings in these populations and how those influence or help. And I think that’s absolutely important. I think for most illnesses, including cancer, it’s a confluence of gene and environmental interaction. All these factors come together to influence a person’s risk of disease or outcomes after having disease, etc. And so I think to my earlier point about trying to draft as comprehensive a picture as you can, in this context, genetics research is another aspect of that.
Are there best practices for data collection?
So, I think there’s quite a bit of literature to support defining race and ethnicity by self-report. If available. But I can tell you that in dealing with medical claims research, you’ll oftentimes have these variables in there that are not necessarily the patient’s self-report. It could be the physician’s interpretation of their race or ethnicity, and that could be very different from how the patient self-reports their race and ethnicity, so it gets complicated.
[…]
Again, it goes back to this idea of comprehensive data collection, and the availability of the data, so that if we have not only information on race and ethnicity, but we’ve also got socioeconomic information, income, education, for example, we’ve got geographic location — these are all things that you can think about taking a look at, and not necessarily just ignoring the other ones, because you’re only interested in differences by white versus black, for example. Which I do think tends to be an issue too, because we do tend to look at the extremes, in terms of how we categorize our variables, and what we determine to be the reference group. And so, oftentimes, those other variables are not highlighted as strongly in the results. And so, the papers that went along with the editorial, they’re primarily interested in black and white differences in cancer outcomes, because we know that there are disparities that exist there from prior research. But when you take an approach like that, you are at risk of ignoring other issues that might be going on, other disparities that might be going on in these other populations as well.