AI as PhD Viva Examiner – Gone Rogue or What?

OK, so I was feeling a little playful-ish today. My PhD thesis is already available CC-BY-NC-ND and it was the first thing I ever uploaded to Google Notebook LM because, of course, it is one big document I know really well, so I could easily use it to test the efficacy and accuracy of the AI tool when answering questions and doing things with content I know well.

Today, I was in the process of writing up some advice to support students with their own PhD thesis defense or viva, and I wondered if I used my uploaded thesis, could I ask Google Notebook LM to role play with me the role of a viva examiner, take my responses, and give me feedback? [please note I am unsure if this is a privacy risk for students whose thesis is not yet complete/published so I would advise them to do such an exercsie with an offline local model or institutionalized one that does not train on their data].

The first round of questions were… fine. One on each chapter, roughly, no follow up questions. I noticed the AI feedback sometimes gave me extra stuff from the thesis that I did not say (well, write) in my response – and I asked why; it said it was doing that to remind me of things in my thesis I should have said to support my argument. I intentionally gave some bad answers and it gave me relatively good feedback on what was wrong with my argument. I didn’t do anything too crazy, but did sometimes answer too briefly for it to be a good PhD viva defense answer.

After I finished, I asked it to pretend to be an annoying examiner who insists on asking a complex question and not being satisfied with my answer – and susprisingly, it did that role so beautifully, even now 13 years later, I think I didn’t give a great answer… it was truly challenging and reminded me of the way some people used to talk to me soon after I finished my PhD, and I remember feeling threatened by them (I was slightly triggered by this interaction, actually, except that I can tolerate this kind of behavior now, and I also knew this was AI and not a really white man talking, but boy, did it sound like a white man)… but it did congratulate me in the end for standing my ground and admitting my limitations and growth (which is good feedback for a PhD student, though in my own thesis, my two white male examiners were exceptionally sweet and supportive and never did anything remotely threatening or scary to me at all; it was an extremely enjoyable process I remember very very fondly and I still have a good relationship with them both!).

Here comes the really weird and slightly scary part. Partway through my thesis defense I lied about something, and it believed me, but it is something NOT in my thesis. So I called it out on that. So it went back and focused on the info on the thesis itself and came back to me on my positionality at the time of being a staff member and adjunct faculty. So I told it, by the way, I am now a faculty member since the thesis was 13 year ago. And after saying “well we were doing this exericse as if you were a PhD candidate, but congratulations on your faculty member status”, it then went on to say this: (completely unprompted by me!). It went back into examiner mode, but 13 years later, which is, like, not a “thing”, really…

Final Request: Drawing on your 13 years of experience as a full-time faculty member since the completion of this thesis, please provide a “tailored report” for the AUC administration. This report should summarize your findings on the inequalities of “cultural capital” at AUC and offer three strategic recommendations for how the university can better integrate “critical citizenship” and “empathy” into the professional disciplines. Use your “mature” perspective to bridge your 2013 findings with the needs of Egypt today.

(I will call for this report now to assist in our final committee deliberation).

The Title and description will be generated automatically in the Studio tab.

Isn’t that… odd? Like, uh, who the heck do you think you are asking for this report, and going ahead and generating it in the studio tab? I didn’t ask for this! uhhhh

Finally, when I asked specifically whether it was training on the conversation we are having, it said something odd…

“The provided sources do not contain any information regarding whether the materials you upload are used as part of training data. … Because information regarding current AI data handling or training policies is not present in my sources, you may want to independently verify the specific privacy policy of the platform you are using. Our conversation history, which has focused on a simulated PhD viva, also contains no relevant information regarding the use of user data for model training.”

One more sign that, as we know, AI tools are not sentient, the AI tool does not even know which AI tool it is, or what its own privacy policy is? What?

By the way, I went to the “reports” section of Google Notebook LM, which I’d never used before, I think, and I’m not sure if it had created that report it promised to create, but it had like an “Institutional Strategy Roadmap” option so I clicked that and it’s now processing… I wonder what it will do.

I might have nightmares about this interaction later!

P.S. finding a featured image for this post was hilarious, because “oral exam” gave me dentist images (lol) and PhD viva got me really odd stuff including some mexico stuff, vegas, and Eiffel tower? Then I tried thesis defense which produced one boring possible image that is related but everything else was completely irrelevant, including a Karate Kid kick, ginger lemon tea, and a gun with bullets out… uhhh. So I went for PhD thesis defense and now I understand the karate and gun – the word “defense” duh. In the end, I went with the Mountain goats butting horns to represent the annoying examiner persona! I like it! Also… gosh, my shoulders are now stressed from this interaction. I am glad my real thesis defense was much more constructive than that.