Pro-environmental behavior is exaggerated on self-report questionnaires, particularly among those with stronger environmentalist identity

New research published in the journal Current Research in Ecological and Social Psychology suggests that the way in which pro-environmental behavior is measured impacts the magnitude of results, revealing that standard questionnaires elicited more exaggerated results compared to anonymous responding.

“The topic of the article combined two topics of great interest to me: psychological perspectives on climate change and the environment on the one hand, and how to improve research methods in the social sciences on the other hand,” said study author Katharina Koller, a researcher at the Centre for Social Innovation based in Vienna, Austria.

“Climate change and environmental protection are increasingly urgent topics, and it’s important to conduct rigorous research to understand factors such as people’s attitudes and intentions to develop effective and acceptable strategies for combating climate change. In turn, rigorous research requires good measures, which is especially difficult when it comes to behavior.”

“Moreover, behaviors around the topic of climate change & environment sometimes seem to be somewhat polarizing, connected with lots of different opinions and values. Polarising, norm-related behaviors are even more difficult to study, as people are aware that they might be judged depending on their acts. So I decided to investigate how good our measures are (i.e., their validity) when it comes to environmentally-friendly behaviors.”

In their article, Koller and colleagues note that prior research on pro-environmental behavior has suggested measurement error can arise due to factors such as social desirability, memory or recall bias, participants’ own interpretations of the responses options, or inaccurate perceptions of their own behaviors.

Free daily newsletter

Study 1 included 951 participants residing in the United States recruited through Prolific. They were randomly assigned to one of three treatment conditions (Social Identity, Intention, Social Norm) or the control condition. In the Social Identity condition, participants completed an environmental identity measure to increase their identity salience. Participants in the Intention condition responded to questions eliciting future intentions to engage in pro-environmental behavior. Those in the Social Norm condition were given instructions that emphasized “injunctive norms” – as in “what people should do.” Lastly, participants in the control condition completed a filler questionnaire.

Following the tasks of each condition, participants self-reported their pro-environmental behavior. Those in the Norm, Intention, and Control conditions also responded to the environmentalist identity measure used in the Identity condition. Lastly, participants provided sociodemographic information.

In Study 2, the researchers “used the Unmatched Count Technique (UCT) to investigate overreporting littering prevention behavior.” This method of indirect questioning involves endorsing the number of statements that apply to the participant without identifying the specific items. In this way, participants’ responses are more anonymous, potentially minimizing socially desirable responding.

A total of 388 United States residents participated in Study 2. Participants were randomly assigned to either the covert or standard condition. In the covert condition they received items on littering prevention behavior and decoy items about various domains of life (e.g., I have a dog). Those in the standard condition only saw the decoy items. Following the UCT task, participants in the standard condition also reported their behaviors by responding to each item individually. As in Study 1, participants completed the environmentalist identity measure and provided sociodemographic information.

While Study 1 did not reveal large effects, completing an identity questionnaire elicited slightly more endorsement of pro-environmental behavior, suggesting priming identity can impact self-reported behaviors. Study 2 showed that participants who completed the standard measure reported more pro-environmental behavior (i.e., littering prevention) compared to participants who responded to a covert measure utilizing the Unmatched Count Technique.

Interestingly, those who endorsed stronger environmentalist identity tended to endorse more pro-environmental behavior in the standard condition. However, when reporting anonymously, environmentalist identity was not predictive of past behavior.

I asked Koller what the average person should take away from this work. The researcher responded, “Difficult question, as it is a very methodological study. I think the skill of evaluating information, including research, becomes increasingly important for everyone, also with the rise of AI-generated or fake content. I think that my study suggests the importance of critically evaluating the methods and measures a study used, to be able to assess its conclusions.

“At the same time, I don’t think it’s the responsibility of the individual to develop that skill, but that this should be a part of school education and media, whereby rigorous but accessibly written journalism can go a long way to make science understandable for the average person. Perhaps a better understanding could also help to promote trust in science.”

As for questions that still need to be addressed, Koller said, “A lot of work still needs to be done to develop sound behavioral measures that measure actual behavior as accurately as possible. Recently, some researchers have been developing new experimental tasks in which participants can make pro-environmental choices that have an actual environmental impact, for instance by using less energy (you can find something about this here).”

“I’d like to see more work in that direction for different behaviors, as well as new methods for using data sources other than surveys, for instance, digital trace data. There is still lots of room for developing new questions and new methodologies in the area of measuring behavior.”

The research, “Identifying bias in self-reported pro-environmental behavior”, was authored by Katharina Koller, Paulina K. Pankowska, and Cameron Brick.