EES 2026 LILLE · STRAND S01 NEW METHODS · SESSION 281
Interrogating evaluation:
a live experiment
and conversation
with a little help from AI
Silva Ferretti, independent consultant · Steve Powell, Causal Map Ltd
# Where is evaluation going?
## What we are about to do
Ninety minutes, one question, and two ways of answering it.
- You talk to an AI interviewer about where evaluation is going - We look at its rapid synthesis together, as an interim draft - Some tables talk it through with an AI moderator, others without AI - Tables report back, and we compare - We ask what each method made easier, harder or invisible
## A declaration of interest
Steve develops the AI interviewer we are using today, Qualia, so he has an interest in it.
There are other AI interviewing apps, and we will say briefly how they differ. What we are testing today is the method.
## From the field: South Sudan
Silva to fill: the South Sudan interviews, and the workshop that followed straight after.
## Some other uses, more or less structured
- **International Evaluation Academy**: a consultation across its network on evaluation and transformation, in place of a text-box questionnaire (2026) - **VIB, the Flanders Institute for Biotechnology**: over 100 PhD students on their journey, the challenges and the support (2025) - **Duoc UC, Chile**: 32 students in STEM programmes, in Spanish, on the gender gaps they meet from study into work (2024)

# Your turn: the interview
## Where is evaluation going?

Scan the code, or go to `chat1.qualiainterviews.com/?interview=ees26-where-is-evaluation-going` Pick your interviewer: -
neutral
-
challenging
-
exploratory
-
academic
-
surprise me
Talk or type. About twelve minutes.
# What did the AI hear?
## The rapid synthesis
An interim artefact, not a conclusion.
Paste or show the live synthesis here.
It reads the interviews on three fronts: whether evaluation needs a transformation or improvement; the future people want, what in today's evaluation holds it back, and what is already moving (Three Horizons); and what being interviewed by an AI was like.
## What does it miss?
You can make a convincing synthesis with AI.
What does it leave out?
Whose stories are there, and whose are flattened?
What would you have asked that it did not?
You told the interviewer what it missed. Does the synthesis hear that?
# At your tables
## Two kinds of table
**With an AI moderator**  `chat1.qualiainterviews.com/?interview=ees26-tables`
**No AI** Talk it through as a table, with paper and pen.
The AI moderator has read a running summary of the room's interviews. One person types for the table; everyone talks. About twenty-five minutes; keep the last five for your report back.
## Questions for every table - What do you recognise in the room's synthesis, and what is missing or wrong? - Which one change in where evaluation is going matters most, and why that one? - What should evaluators start doing, or stop doing? - Where do you disagree? - Agree two or three points to report back, including any disagreement.
## Want to build your own?
If you would like to set up an interview for your own work, we can help you at your table.
Sign up and design your interview at `manage.qualiainterviews.com`

More about Qualia at `qualiainterviews.com`

# Reporting back
## Two questions for each table
**Where is evaluation going?** And what is to be done?
**What did the method do?** What did it make easier, harder, more visible or less visible?
# Wrap-up
## What just happened
- The interview we just did was a real-time evaluation - Interviewing can be the boring part, and the AI took it on - It all means nothing if it does not change our minds along the way
## Thank you
**Silva Ferretti** silva.ferretti@gmail.com
**Steve Powell** steve@causalmap.app · qualiainterviews.com

`causalmap.app/events/ees2026`
# In our back pockets
## The management side  Designing, testing and reading the transcripts, live.
## When to use an AI interviewer, and when not
**Use it** - When you do not know exactly what to ask - For less structured, exploratory conversations - When respondents speak different languages - To reach more people than interviewers could
**Think twice** - When a questionnaire would do - When you have many specific questions - With sensitive topics or vulnerable groups - When people need a human in the room
Face-to-face interviews give depth to a few people; questionnaires give reach without depth. Where does AI interviewing sit?
## Five interviewers, one instruction Each of you chose how to be interviewed. The difference is one line of the instruction.
- **Neutral.** Calm and unobtrusive. Clear, direct questions; probe only to clarify or get examples. No opinions or interpretations. - **Challenging.** A critical colleague. Politely sceptical; spot contradictions, question assumptions, challenge claims that sound good but lack substance. - **Exploratory.** A curious ethnographer. Invite stories and details, follow threads I mention in passing, look for what is surprising or overlooked. - **Academic.** A demanding professor. Probe definitions, assumptions and the logic of arguments; link to wider debates without lecturing or inventing references.
"Surprise me" picks one without saying which, and asks you to guess at the end. Did the style change what you said?
From the interview's instruction, slightly shortened.
## What the interviewer is told
**The researcher's instruction** - The aims and the questions, in plain language - Sections marked by importance: some are asked whatever happens, some can go when time is short - The style, the languages, the tone
**Rules no instruction can remove** - Raise a safeguarding alert when someone discloses a risk of harm - Put the respondent's time before any target - Ask before ending the interview - Use the markers the app reads for sections, progress and closed questions
Writing the instruction is the research design. Testing it is the piloting.
## What the AI moderator read - A summary of the room's individual interviews, about 400 words, written by a separate AI call - First written once five interviews had at least five messages each, then rewritten as they grew - Paraphrase only: no quotes, no names, nothing that could identify anyone - Test runs and archived transcripts are left out - The tables' instruction says what to do with it: share its main points, then ask what is missing or wrong
## How we tested today's interviews
-
**7**
simulated respondents at once, in five languages, one speaking and one on a phone: all reached the end in their own language -
**30**
conversations at once: none failed; half the replies came within 2.5 seconds -
**12
of 12** rehearsed table conversations opened with the room's summary; the first wording managed 2 of 6, so we rewrote it -
**6
of 6** "Surprise me" interviews asked for the guess and revealed the style
Simulated respondents test the instrument, not the findings.
## Are people more candid with a machine?
In health-screening interviews with a virtual human, people who believed it was automated, rather than operated by a person, reported less fear of disclosing and less impression management, and observers rated them more willing to disclose.
Lucas, Gratch, King and Morency (2014), *Computers in Human Behavior* 37, 94–100.
- Probably strongest on sensitive or embarrassing topics - It may reverse when people suspect they are being watched or recorded - Does it hold for evaluators talking about their own field?
## Languages and channels
- **97 languages** in the language menu. - **Spoken dialects.** For seven Arabic dialects the interviewer is told to write the spoken variety, not Modern Standard Arabic. - **Talk or type.** Speech is transcribed in the same region as the rest of the data. - **Where people already are.** A web link on any phone, or WhatsApp and Telegram for EU interviews.
## Data, privacy and safeguarding
- **EU mode.** A separate database, and every AI call made inside the EU. Today's interviews run there. - **Anonymous by default.** The interviewer asks people not to give names, and flags it when they do. - **Redaction, if you want it.** Names and other identifying details are replaced before anything is stored, and the respondent sees what was kept. - **Safeguarding alerts.** When someone discloses a risk of harm, the interview's owner and editors get an email straight away. - **Retention.** You choose how long transcripts are kept. They are then deleted automatically, and the owner is sent a receipt.
## A first round of Outcome Harvesting: KLAR!, Austria Climate change adaptation in Bad Ischl and Ebensee, two pilots in 2024 and 2025.
-
**19**
of 39 invited stakeholders took part, most for 20 to 30 minutes -
**38**
outcome statements, 6 of them meeting the SMART criteria as they stood -
**32**
leads for human follow-up and validation
"It was fun writing with you." · "Repetition of questions somewhat tedious." · "I prefer communicating with humans."
Birgit Müllegger, Konny Rassmann and the Causal Map team; presented at UKES 2026.
## From transcripts to causal maps: Duoc UC, Chile - 50 invited, 32 interviewed by Qualia in Spanish, on the gender gaps women meet from STEM study into work - AI coded every causal claim with no codebook, inventing its own labels in Spanish - 251 causal links, put together into one map in Causal Map
The interview is half the job. The transcripts still need analysing, by people, or by AI whose work people can check.