Workshop

Steve Powell, Director

Gabriele Caldas, Outreach Coordinator

Who are we

• This is work in progress!
• It’s amazing!
• We are flexible, responsive, get in touch! hello@causalmap.app

​

​

​

​

​

​

Plan

• Poll - 3 min
• Let’s get interviewed - 5 min
• Motivation - 4 min
• What’s QualiaInterviews? - 2 min
• Qualia - Overview - 8 min
• Hands-on Exercise - 20 min
• Crafting the instructions - 3 min
• Analysing your data - 2 min
• Discussion (Finally…) - 10 min
• Q&A - 15 min

​

​

​

​

🔸

Let’s get interviewed

Let’s get interviewed!

https://shorturl.at/7EET8

🔸

5 min

In the chat: how did you feel?

🔸

10 min

Motivation

Why AI-led interviews?

Quality

Quantity

Face to face �Interviews

Structured questionnaires

Conversational AI interviewing?

Multi-language: hard Multi-language: easy

Slow to administer Fast to administer

Expensive Cheap

Noisy (Random) Reliable/Reproducible

Rigid Adaptable / flexible

Noisy (Interviewer) Reliable/Reproducible

Slow to analyse Fast to analyse

Insensitive Read between the lines

Predictable data Emergent, unexpected data

🎤

🎤

🎤

🎤

🎤

🎤

🎤

🎤

☑️

☑️

☑️Questionnaires vs 🎤 Human-human interviews vs. 🤖 AI Interviews?

☑️

☑️

☑️

☑️

☑️

🤖

🤖

🤖

🤖

🤖

🤖

🤖

Slow to set up Fast to set up

🎤

🎤

☑️

🤖

🤖

Scaling: hard Scaling: easy

🎤

☑️

🤖

🎤

☑️

🤖

☑️

DO: When to use AI interviewing

• Situations where you don’t know exactly what to ask
• Less structured, explorative conversations
• When your respondents may use different languages
• When you can scale up from H2H to AI interviewing

​

DON’T: When not to use AI interviewing

​

• When you could have used a questionnaire with open questions
• When you have a lot of specific questions to ask
• When you are dealing with sensitive information

14 min

Qualia

What’s QualiaInterviews?

A platform for designing and delivering AI-led online interviews.

“SurveyMonkey, but for stories”

One interview costs £1-2. You get £50 in credits if you sign up. No subscription.

How it works:

• Design and test your interview
• Send the interview link to your respondents
• Analyse the transcripts

​

Alternatives:

• Ailyze.com
• Yasna.ai

​

🔸

16 min

Qualia

Overview

Walkthrough of Qualia’s interface

• Create, test and manage the interviews

​

​

​

• User guide: https://www.qualiainterviews.com/documentation/

Privacy and consent

• Privacy
• At Qualia, all interviews are, by default, anonymous and we don’t store personally identifying information at all.
• Qualia stores data securely, but researchers should still be cautious with sensitive information.
• Consent
• Ensure data anonymisation and compliance with privacy laws (e.g., GDPR).
• The introductory pop-up, which includes your Description text is designed to get informed consent.

​

Multi language!

• It’s cool!
• Look at the docs

What instructions to give the AI?

​

You have a lot of flexibility. 10 overlapping tools in one.

Match your design to your analysis plan

• Highly structured
• Loosely structured
• Conditional logic
• Iterative questioning
• Causal mapping
• Social network diagrams
• Digging deeper
• Iterate a hypothesis
• “Thinking models”

What if it “isn’t working”??

​

• Tweak the instructions. Test. Iterate.
• Don’t overload it

🔸

• Test interviews before full deployment
• Monitor for inconsistencies and adjust the interview design as needed
• Document the steps you took
• Use learnings from each round to refine the interview design: it’s an iterative process

Preparing to launch

🔸

24 min

Breakout

Exercise: �Designing an AI-led interview with Qualia

STEP 1

Creating a new interview

• Give your interview a name

🔸

Exercise: �

STEP 2

Writing the interview instructions:

You can edit the texts directly and/or use the Interview Advisor

• Identify your research objectives → Plus, are there any specific questions that need to be answered?
• Define your target audience → this helps the AI identify the correct tone to talk to the respondents

🔸

Exercise: �

STEP 3

Testing the interview:

• Manually
• With the Automation panel (with different personas) 

🔸

Your task in the breakout rooms

• Agree on a simple interview you can start to create together: Context, audience, What do you want to find out?
• One of you (“the editor”) with a good wifi connection and a google account: log in at https://edit.qualiainterviews.com/ .
• Decide a name for the interview
• Editor uses the Interview Advisor and/or edit the interview texts directly. (Don’t all try to edit the interview together.)
• Editor tests your interview inside the editor interface
• Editor sends the link to the others to test
• Editor continues to tweak the Interview Texts
• Editor views the Transcripts and Overview

edit.qualiainterviews.com/

�User guide:�qualiainterviews.com/documentation/

You won’t finish all of this! �Main thing is to explore!

Ask for help in Zoom�😃😃😃

Feedback from breakout sessions

In the chat:

• what worked
• what didn’t work?

44 min

Qualia

Crafting the instructions

What instructions to give the AI?

= Like any other interview!

Best Practices

• Have an aim: (research aims, not predetermined questions)
• Listen actively
• Build rapport
• Probe for deeper insights
• Embrace Flexibility: Explore emergent topics that arise.
• Summarize Periodically: To ensure mutual understanding.
• Guide the Discussion: To align with your research objectives, while remaining open to unexpected insights.

Common Pitfalls to Avoid

• Avoid Rushing: Respect the natural flow of the conversation that may lead to richer insights.
• Steer Clear of Leading Questions

🔸

Tips for writing the instructions / interview guide

The QualiaInterviews platform already has some basic instructions to guide the interviewer behaviour, you can build on those.

• Think of the AI interviewer as a human intern. They will need very clear instructions to be able to conduct the interviews in the right way.
• Look at your instructions as the AI would see them. The AI will seem to not always follow your instructions exactly. This doesn't mean the AI is "broken" or “stupid”, it usually means you need to tweak your script.
• Start simple: so you can get the hang of it.
• Be clear and specific: If it’s hard for a human to understand it, it will be hard for the AI.
• Experiment with different approaches until you find the one that best fits what you want.
• Try the “Precision” model. These can follow longer and more complicated instructions.
• Contextualize: Provide enough context for the AI to understand what you want, and describe your target audience.
• Keep it short: In principle, you can ask the AI to keep any number of things “in its head” while constructing a response but in practice it seems it can end up getting confused (like a human).
• Use structure: Use sections, starting with a clear and concise statement of what you want the model to do, then provide the required context, then the specific instructions.
• Give examples. But check the AI does not get too distracted by these and use them too often.

​

🔸

47 min

Consider using special interview strategies

= Definitely not a fixed question-by-question approach

The interviewer adopts a certain style and adapts to the flow of the conversation (based on the instructions you give the AI).

Causal backchaining: which you saw in the USA problems interview example. For when you would like to get information about your respondents’ mental models of what influences what in their world. This is the kind of questioning that we use with when we want to do causal mapping on the final transcripts.

Social network questions: who is linked with whom and how? And who is linked with them?

Laddering techniques (Rietz and Maedche, 2022): to uncover the underlying attributes, consequences, and values (ACV) associated with participants' experiences of the topic discussed.

​

Analysing your data

Analysing your data

Global analysis (Leaves a lot of responsibility to the AI. Overview of trends and patterns across all interviews. You can also ask specific questions, like: what did the respondents mention about climate change?)

• Using the Overview Tab
• or upload transcripts to, for example, ChatGPT.

​

Detailed analysis

Respondent-by-respondent and/or line-by-line: more verifiable/transparent, eg:

• causal mapping
• social network analysis.

49 min

Analysing your data: simple overview

Part of the output from the Overviews tab of an interview about problems in the USA

49 min

Analysing your data: a detailed analysis

Example of causal mapping of an interview about problems in the USA. �This analysis was conducted systematically by coding causal claims (with AI) line by line, rather than just asking the AI for an overall summary.

49 min

Orange lines are negative links. �Blue lines are positive links.

Finally…

Limitations (what do you think?)

• Stochastic nature of conversations (they can "wander off"…)
• Missing subtle cues, (non-verbal!)
• Lack of rapport
• Limited intervention for distress, whistleblowing, safeguarding…
• AI is not “neutral”: its responses are based on training data, which may contain inherent biases. Try to mitigate with explicit instructions to take a specific stance.
• Inclusivity: access to internet and literacy.

​

The researcher’s role

• AI interviews as a way to scale rather than replace human-led interviews.
• Design and oversight: The researcher's role is more about designing and overseeing the interview and less about direct interaction.
• Ethical considerations: Ensure informed consent, considering participants' understanding of AI involvement in the interview process.
• Data interpretation: Interpreting and analysing the data becomes even more critical, accounting for the AI's influence on the interview dynamics.
• Quality control: Continuous monitoring and adjustment of the AI's performance is essential to maintain data quality and integrity.
• Responsibility: Take full ownership of the ethical implications of using AI: the AI is only a tool, it’s the researcher's responsibility to make decisions.

​

Strategies for maintaining rigour/transparency

• Iterate: design → test → check → improve and document decisions you made.
• Providing clear explanations of AI's role in the evaluation.
• Avoiding over-reliance on AI-generated summaries without human oversight.

​

🔸

Remember the ‘tips for writing the instructions’ slide

• Get extra £100 on top of the £50 credits you get when you register
• to conduct up to 150 interviews for free!

Extra free credits to use Qualia!

• Submit your proposal to gabriele@causalmap.app until March 28th:
• a short, succinct proposal (less than 500 words) saying how you would like to include QualiaInterviews in your research.

​

• All applications are welcomed, but proposals regarding research on diversity equity and inclusion will be given preference.
• Accepted applicants are expected to write a blog post about their experience with Qualia within 2 months.
• When we approve your application we will send you details of how to access credits and how to get support when using QualiaInterviews.

​

60 min

Q&A

Thank you!

Steve Powell

Co-founder and Director - Causal Map

hello@causalmap.app

Gabriele Caldas

Outreach Coordinator - Causal Map

gabriele@causalmap.app

Visit QualiaInterviews.com

Annexes

A case study

A real-world case study using Qualia

Background

DuocUC, a higher education institution in Chile, hired our consultancy (Causal Map Ltd) to conduct QuIP-style interviews with Qualia and analyse them using the Causal Map app. The interviews were motivated by concerns about the gender gaps faced by women pursuing STEM careers at the university.

Step 1: Setting up the interview in Qualia

• The instruction for the AI interviewer was similar to the instructions you could give to a human interviewer. And both the interview instructions and the interviews itself were conducted in Spanish.
• The AI asked questions about changes in 3 domains: educational experiences, professional development and relationship dynamics.
• We used GPT-4o which was the best AI model to date.

🔸

Step 2: Collecting stories with Qualia

• We sent the interview link to 50 people and were able to collect 32 interviews.
• We created special individual links to be able to track the interviews
• We downloaded the interview results from Qualia and uploaded them into Causal Map.

Step 3: Analysing stories with Causal Map

• We used AI (GPT-4o) to identify each and every causal link in the interviews, and for each link, to label the cause and effect.

​

• We used a “radical zero-shot” approach in which the AI is given no codebook and is simply told to invent its own codes (in Spanish). We gave the AI context about the project.
• We found 251 causal links mentioned by the respondents

A real-world case study using Qualia

🔸

References

• Chopra F and Haaland I (2023) Conducting Qualitative Interviews with AI.: 72.
• Jang ME and Lukasiewicz T (2023) Consistency Analysis of ChatGPT. arXiv preprint arXiv:2303.06273. Epub ahead of print 2023. DOI: https://doi.org/10.48550/arXiv.2303.06273.
• Ray PP (2023) ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems 3.
• Yuen M (2022) Chatbot market in 2022: Stats, trends, and companies in the growing AI chatbot industry. Insider Intelligence. https://www.insiderintelligence.com/insights/chatbot-market-stats-trends/