Study links faculty AI use to beliefs about thinking and learning

In the Harry Potter stories, the Mirror of Erised shows the person standing before it the deepest desire of their heart. Timothy J. Atherton sees something familiar in the experience of talking with generative artificial intelligence, where a large language model, or LLM, is designed to respond in ways that reflect a user’s language and interests.

“I think of AI as kind of like the Mirror of Erised from Harry Potter,” he said. “It will show you your heart’s desire, and you can sit there and waste your life in front of it.”

For Atherton, a physicist at Tufts University, AI also holds a mirror to higher education. It exposes questions about expertise, labor, access and what it means to learn a STEM discipline when some forms of thinking can be delegated to a machine.

Those questions became the focus of a study of 90 faculty members in Research Corporation for Science Advancement’s Cottrell Scholars community. Atherton and his colleagues found that faculty decisions about using AI depended less on institutional policies, access, or even beliefs about AI in general and more on pedagogical orientation — the set of views each person holds about the role AI should play in thinking and learning.

Their paper, “Faculty Orientations Shape Adoption of AI in Research and Teaching,” was published today in Physical Review Physics Education Research.

The study grew out of conversations among members of the RCSA community about computation in science. In 2018, an RCSA-funded team called “ESCIP,” short for Enhancing Science Courses by Integrating Python, started an educational project to treat computation as a disciplinary skill in physics, chemistry, and astronomy rather than simply a technical add-on.

By 2022, tools such as Google Colab, which the group used to give students a browser-based computational environment, were integrating LLMs directly into the same notebooks where students learned to code.

“All of a sudden, you had generative AI tools that were directly linked to the environment that we were teaching computation in,” said co-author Jay Foley, a chemist at the University of North Carolina at Charlotte.

The development presented possible benefits and serious complications. Could AI help students move past routine work to explore harder scientific questions? Or would it allow them to avoid the slow effort through which understanding develops? The RCSA-funded “CottreLLM” Collaborative began asking what these tools meant for teaching and how faculty should adapt.

Atherton had heard plenty of discussion about what AI might do and how universities should police student use. He heard much less from the people designing courses, defining disciplinary standards, and deciding what students needed to learn.

“What I didn’t hear much of in the conversation was, has anyone actually talked to the faculty and asked them what they would like and what is going on?” he said.

The group decided to replace speculation with a mixed-methods study that paired numerical survey questions with open-ended responses. Members of RCSA’s Cottrell Scholars community were a perfect starting point because of their commitments to scientific research and undergraduate teaching. As faculty-scholars in the physical sciences, they also might be expected to experiment early with new technologies and pedagogical ideas.

About 78% of survey respondents reported using AI in at least one context. Chatbots and virtual assistants were the most common tools, and respondents described using AI for tasks including coding, data analysis, writing, and lesson planning.

Atherton expected views about teaching to be related to classroom use. He was surprised that the same orientation was also associated with AI use in research and other work, suggesting that it reflected something broader than a teaching preference. When the team examined the open-ended responses, the reason became clearer.

“The thing that really emerged was how people think about thinking,” he said.

Faculty with lower orientation scores often described the struggle involved in solving a problem as an essential part of learning. They worried that AI could produce a quick answer while encouraging shallow understanding. Faculty with higher scores were more likely to see AI as a way to delegate selected tasks in service of higher-level work, provided that people continued to evaluate, verify, and revise what the tools produced.

It was really interesting to see, at least in our population, that there is a lot of nuance and a lot of internal deliberation that people are having and acting on.

Jay Foley

Neither group fit neatly into a pro-AI or anti-AI camp. Faculty who used the tools extensively still voiced reservations about reliability, ethics, and the loss of independent thought. Those who limited their use could still recognize practical value. Rather than two opposing groups, the researchers found a continuum of positions.

Those subtleties surprised Foley. Public conversations about AI can make it appear that people have divided into enthusiastic adopters and Luddites, he said. The survey responses told a different story.

“It was really interesting to see, at least in our population, that there is a lot of nuance and a lot of internal deliberation that people are having and acting on,” Foley said.

The relative weakness of several expected influences was also surprising. Sixty-four percent of respondents reported institution-wide strategic initiatives around AI, compared with 8% who reported department-level initiatives. Yet institutional initiatives and information sources showed little consistent relationship with the AI pedagogical orientation.

For Foley, the gap between administrative action and academic discussion was telling. The result suggested that policies and centrally distributed guidance may have limited influence unless faculty also have opportunities to work through how AI fits the goals and values of their disciplines.

The authors recommend discipline-specific conversations and low-stakes opportunities for faculty to try AI, examine its outputs, and decide when it supports learning and when it short-circuits an important stage of developing expertise. They also suggest considering assignments that require students to validate, interpret, and revise AI-generated work rather than accept it without scrutiny.

Atherton plans to test whether the orientation holds in a larger faculty population and whether it is associated with decisions instructors make in the classroom.

In addition to Atherton and Foley, faculty co-authors are Tova R. Holmes of the University of Tennessee, Knoxville; Christina L. Vizcarra of Barnard College; and Ning Sui of North Carolina State University. Former doctoral student Ian Descamps, now a lecturer at California Polytechnic State University, and Tufts University doctoral candidate Max Webel also contributed to the research.

Research Corporation for Science Advancement funded this work through a Cottrell Scholar Collaborative award, “CottreLLM.”

---