“Bias in research has real-world pains that disproportionately affect people that look like me. Dark-skinned people. Women… Having the equity lens to know that people respond to different research methods differently — it’s important. It’s expensive. It’s time-consuming. But it’s necessary… to ensure that we receive accurate results, that we treat all groups the same, we don’t lump folks together, and that we’re better able to interpret the data.” -Sian Lewis
At the center of Harmony Labs is research. Our research utilizes the Narrative Observatory, data infrastructure about U.S. media consumption, to provide strategists and changemakers with insights necessary to create effective, constructive stories. Since late 2021, we’ve been hard at work building an audience-narrative architecture for the Robert Wood Johnson Foundation’s (RWJF) efforts to advance health equity in the United States.
This work requires an unflinching commitment to data equity in research design and modeling, and to implement standards of practice that clearly and thoroughly engage with what data equity should be. But barriers like a lack of incentive, a lack of resources, and a lack of concrete steps have led to data equity being historically omitted from most research best practices, and rectifying harmful narratives is often rooted in rectifying harmful research design. Janay Cody, PhD and Sian Lewis, MS, both data scientists and equity consultants as well as members of the Harmony Labs Advisory Board, are currently working as equity advisors for our project with RWJF. In a recent conversation, Science Director Riki Conrey, PhD spoke with both Dr. Cody and Lewis to discuss:
What should data equity actually look like? How can we, and other researchers, enact practices that hold us accountable?
We hope the insights from their conversation can act as a resource and conversation starter for all researchers, changemakers, advocates, and storytellers.
“Data equity sits right in the middle … of strategy and science.” -Janay Cody, PhD
“I actually [talk] to the folks who are affected by the models that I’m building many times during the process. It is a part of my work. I literally build it into all my plans. It’s a part of the budget. It’s an essential part of it. I don’t think that folks have that lens at all. And I wish that they did. I think that a lot of the equity issues we went into would be alleviated if people took the time and spent the resources to do this.” -Sian Lewis
“Data equity is not diversity. Diversity is making sure there is representation of people from different backgrounds. That’s great. That is not equity. Equity is ensuring a measurable improvement in outcome, such that typical characteristics that we care about, like race and gender and age, don’t statistically determine particular outcomes.” -Janay Cody, PhD
Algorithms can be biased by design. They are simplifications of the world and use historical data to predict the future — essentially, mathematical stereotypes. Delivering models with an equity lens is hard, but possible, and often rooted in clarity and documentation.
“[Data equity] is … empowering because it finally gives a voice to people. I like to describe myself as the wind. You don’t see me, you just feel my effects… My goal is to give the evidence to the people who need it to actually say the thing that they need to say, make the claim, change the conversation… My goal is not to be seen. It is to be felt through the insights that I provide that are useful for people who need to accomplish a task. That’s very different than the way we think about research and academia and the way we were trained.” -Janay Cody, PhD
Listen to Dr. Cody walk through a recent example in which the structure of an experiment was questioned and restructured after applying an equity lens to the research design decisions. The transcript is available below.
“Oftentimes groups have to balance wanting to demonstrate their impact with actually wanting to make one. In digital campaigns, one of the ways in which they target is through zip codes, and for Black folks, and I’m thinking of Black folks in Pennsylvania, 3% of the zip codes encompassed the majority of that population. When it comes to designing a test, an experiment, to be able to demonstrate your impact in an election or at the end of a campaign, that’s very difficult to do with your zip code, the possible universe of zip codes being [shrunk] so much. So a lot of the things that I’ve had to do is talk about how we’re going to design an experiment. You can create your target universe based upon having a Black population, for example, that is, [one] that makes up at least 30% of the total population of the zip code, where there is a minimum of at least 5,000 respondents, and then you would find treatment and control that way so that you are not taking out most of your sample for an experiment. … There’s this other possibility of making the decision to not to do an RCT and instead choosing an alternative method, like a quasi experiment where you construct your treatment group strategically instead of randomly, and then you construct a comparison group through either propensity score matching, regression discontinuity… if time is on your side, you do … different estimations. But it’s the very real design question of, ‘Do I do an experiment at all, as I’m trying to balance: I want to make an impact in a Black community, but I also need to be able to demonstrate it to my funders and to people who want to support the scaling up of this particular initiative.’”
Though far from comprehensive, these foundational principles of data equity across research design and modeling, as well as an understanding of what data equity is not, have informed and are informing our current and ongoing work with RWJF. Follow this space for more updates concerning both this project and others.
Dr. Janay Cody is a behavioral data scientist and data equity advisor. Dr. Cody empowers results-driven leaders with equitable data policies, inclusive research, and measurement strategies to manage their impact.
Dr. Cody is highly regarded for the analytic solutions she created for philanthropies including the Kettering Foundation; government research teams including the Administrative Office of the Courts on behalf of the Judicial Council of California and the National Assessment of Educational Progress; and data and tech companies including Catalist, Movement Labs, Harmony Labs, and Analyst Institute.
Dr. Cody received her PhD in political science from the University of Notre Dame, du lac.
Sian Lewis, MS is a data scientist and researcher who serves as the Head of Research at A — B (www.abpartners.co). Also, she serves as a political data scientist by employing predictive analytics on Democratic political campaigns. Her research focuses on narratives, inoculation, and persuasion. She formerly served as the data science and analytics team manager for the Department of Defense Healthcare Management System Modernization project (DHMSM) at Booz Allen Hamilton. Her recent accolades include the 2021 Society of Women Engineers Spark award, the 2021 Black Engineer of the Year Modern Day Technology Leader Award, the 2020 Women of Color in STEM award, and the 2019 DC FemTech award.