Delphi study Round 1: Order out of Chaos – Creating a coding schema

Delphi study Round 1: Order out of Chaos – Creating a coding schema

It’s Thursday 10th April, 14h. Since Tuesday I’m looking at the responses of the first ten respondents from my Delphi study. I think by now I have already spent 16 hours on it and the coding schema is not yet there. The expertise group “Team research” is complicated as some of the answers are very cryptic and the experts disagree with each other. Just have a look at the distribution of answers on one of the question about which mix of team and professional identity is best for team performance:

Experts on Team Research Level of Agreement


But these complications aside, analyzing qualitative data is rewarding as the depth of understanding is progressively deepening. Running a number of statistical tests has a different flavor. As part of my audit trail for quality assurance in qualitative research this post deals with the creation of the coding schema.

Creating the coding schema

I used a bottom-up approach when coding the arguments by experts. The reason for this is that I hadn’t a clue what arguments the experts will use. Given that this is a Delphi study which seeks input from experts I felt that imposing a coding structure on the answers will not give me the necessary flexibility to account for the different view points. Of course, I could have also used a mixture of top-down and button-up, and adapted my pre-fabricated coding schema based on the provided arguments. However, I feel that this would have biased me towards the research fields I am more familiar with and the answers I expected. As this research induced bias is one of the shortcomings of Delphi study I tried to avoid it, by coding the answers bottom-up.

I used an iterative approach when coding. First, I read a couple of the answers from the different fields. This gave me a first insights into the types of arguments. Second, I started to look for key words in the answers within one expert group by underlining them and writing notes in the margins. Through this I got an idea of what each specific group deemed important without ‘physically’ creating the coding schema. At this point the codes were still intertwined with the answers. Thirdly, I coded the answers by writing concepts and arguments on memo cards. I chose to use memo cards as it gave me the chance to visually sort through the codes and sort them. At the beginning I just wrote the keyword on the memo card and put it aside. Over time, I started to group them and write notes on the memo cards to help me define the code. At that point I still had only 50 % of responses, but I hoped that I had enough information to create a nearly complete coding schema. Once I coded all the answers I received so far, I had to put a structure with the notes. The goal of the structure was to easily code the rest of the data, to be able to make links between the codes and the quantitative data, and to report them nicely to the Delphi participants. Luckily, I don’t have to share the office with anyone so I had 2 tables I could use to lay out the 48 pieces of paper.

The Coding Schema

The Coding Schema
The Coding Schema

What stuck

I finished ‘seriously’ coding the first 8 questions about social identity configuration and team performance. Glancing at the piles of codes this is what I can remember:


After another hour my favorite words became “analogues to previous argument” and “same as above”. Nothing to code here!

What made me laugh

“I would be happier to not have to decide about the worst option”. Looking at the response options from all researchers in that group, there was no agreement.

Ah-ha moments

Too many to keep count. They ranged from “Oh, that’s what is meant” to “That’s a very interesting point”.

Preliminary Conclusion

The goal of identifying with a team or a profession is to be aligned with the team and/or profession. This means that at least some level of identification is necessary. A nice (and compelling) argument is that professional development is stimulated by team performance which in turn is leveraged by high team identification.

Leave a Comment