Posted on December 3, 2013

Who researches the researchers?

by Anouk S. Rigterink

In a standard class on research methods you will learn about biases that the researcher can introduce into the research him/herself. Researchers, we learn, sometimes unconsciously influence respondents to give answers that make for nice results; or respondents may give different answers depending on the age, gender, ethnicity or nationality etc. of the researcher.

I took notice of the fact that these ‘research biases’ exist, but was never too bothered about them. I happily let loose my econometric methods on data gathered by others. Papers I read did not seem too bothered either; the closest they come to acknowledging and dealing with these biases is by assuring the reader that sensitive questions (e.g. about sexuality) were asked to women by women and to men by men. Some of my friends use different research methods and told me that they planned to study both “the environment that they are in and themselves as a researcher in that environment”. That seemed rather ‘out there’ to me.

Earlier this year, I went to South Sudan, as part of a team from the Justice and Security Research Programme. My main task was to lead a household survey in this area on a number of topics, including how much people trust various local leaders.

The dice-throw of our random sampling sent us to one village where we met with a ‘Boma Chief’, a leader at the village level considered to be a ‘traditional leader’. During our first visit, during which we did little apart from introducing ourselves and explaining our purpose, this Boma Chief was acting unlike other chiefs we had encountered.  He did not take the lead in the conversation and remained seated in the background. From later conversations, we got the sense that this particular Boma Chief was not very well-liked by ‘his’ community.

As the next day was Sunday, we spent it in a larger town some miles away. The Boma Chief turned up in town too, and, knowing that we were returning to ‘his’ village on Monday, came asking for a ride. As we had an interest in him being there too, and did not see a reason for him to ride his bike all the way while we were driving anyway, we took him along. We spent three more days in this village gathering our survey data and left with the job done.

Of course we were aware of the risks of appearing to be too closely aligned with any leader, especially if we were asking people questions about the degree of trust in local leaders afterwards. But it is impossible to avoid all such association; doing research in a village without the local leader’s permission would be a serious mistake. In my mind, I had already written a disclaimer for our report, stating that the degree of trust in local leaders was likely biased upwards because of this.

It turned out we had an impact on the situation in this village rather more directly.

Some time and two villages later, I met with the Boma Chief again, this time in the county capital. He sounded angry. Through a translation, I understood that rumours had surfaced in ‘his’ village that we had given him money intended for the community, but that he had kept it for himself. Already mistrustful of the Boma Chief, the community took the fact that he had spent a Sunday in the same town with us, and had arrived in our car, as evidence of this. In reality, we had never given him anything of any kind. Nevertheless the Boma Chief said he was afraid to go back to ‘his’ village, fearing community members would turn violent.

We tried to ‘fix’ the situation as best we could; we gave the Boma Chief an official-looking letter stating we had never given him anything, and sent a contact in the area back to the village to explain this in person. Some weeks later, our contact reported that things had calmed down.

Our initial concerns that our behaviour might bias our data seem quite trivial now, seeing how our behaviour changed (or at least catalysed changes in) reality so directly. The data that we gathered suggests that trust in this particular Boma Chief is significantly lower than on average (although this is not unique among the villages that we visited). However, I can imagine a carbon-copy of our research team revisiting this village a week after our original visit, and finding trust in the Boma Chief to be much lower.

This blogpost does not offer a more insightful conclusion or a better solution to ‘researcher bias’ than the standard methods class: it is important to be aware of these biases. But I cannot say things would have gone differently, had I been more ‘aware’ (whatever that is). I also do not mean to devalue all data gathering exercises, let alone our own. To me, this experience shows that biases are unavoidable and come at us from unexpected directions. We will have to live with and be open about them.

 

Anouk S. Rigterink is a PhD Candidate at the Department of International Development at LSE. She works on the Indicators part of the Security in Transition Programme and at the Justice and Security Research Programme.

(Reposted from the JSRP blog)