Virtual vs. In-Person Conferences? Scialog Data Provide Insight
Even after the return to in-person conferences, RCSA continued to record Scialog discussions in an effort to understand how participants interact and form collaborations.
Conference organizers can optimize professional interactions in ways that can shape the direction of science for years to come, according to a new paper that used six years of Scialog meeting data from before, during, and after the COVID pandemic to study the benefits and drawbacks of virtual and in-person conferences on team formation, community building, and engagement.
The paper, “Face-to-Face or Face-to-Screen: A Quantitative Comparison of Conference Modalities,” is published in PNAS Nexus with Northwestern University’s Daniel Abrams and his former Ph.D. student Emma Zajdela, now a post-doctoral fellow at Princeton University, as corresponding authors. RCSA Senior Program Directors Andrew Feig and Richard Wiener, and former RCSA Data Analytics Specialist Kimberly Huynh are coauthors.
“We were surprised to find that virtual and face-to-face interaction in assigned discussion groups had a similar effect on team formation,” Zajdela said. “The more two people spend time together, and the smaller the group in which they interact, the more likely they are to collaborate. However, virtual conferences were about half as effective as in-person in terms of community building.”
The study analyzed data from nine Scialog conferences – five virtual, and four in-person – held between 2015 and 2021, involving a total of 573 participants and making up more than 12,000 pairs of participants. A second, very different dataset used in the study encompasses over 250,000 pairs of participants who spoke in sessions at American Physical Society March Meetings. With these datasets, Abrams’s team was able to refine a mathematical model for predicting which conference attendees will team up.
“The richness of the Scialog dataset allowed us to identify how teams formed more precisely compared to the APS dataset, identify the important role of conversations in groups of three to four people, and study the effects of conferences on the connectedness of communities beyond team formation,” Zajdela said. “The APS dataset, although less detailed, allowed us to test our findings on a much larger scale.”
Remarkably, the study found that similarly sized meetings at in-person and virtual conferences performed similarly for team formation at both ends of the scale, whether at a focused Scialog conference with 50 people or 10,000-person APS March Meetings. However, the small Scialog meetings, virtual and in-person, were much more effective at catalyzing new collaborations than the large APS meetings.
At Scialog meetings, participants are assigned to breakout sessions and mini-breakout sessions using an algorithm that considers their prior knowledge of each other, scientific interests, and backgrounds. It produces numerous possible conference schedules.
“No other data set in the world matches this,” Abrams said. “RCSA had incredible foresight to collect and retain more than 10 years of data on conference attendees at an unprecedented level of precision. We know who knew whom before and after each conference, who spent time in discussions with whom (and how much time), who formed teams to apply for seed funding, and all the scientific publications that have been written by those attendees and teams post-conference. In addition, we have counterfactual scenarios that allow us to test for causality.”
The new study follows a previous paper presenting results from analysis of the Scialog dataset by Abrams and Zajdela in collaboration with RCSA program directors Feig and Wiener, “Dynamics of Social Interaction: Modeling the Genesis of Scientific Collaboration.”
“We think there are many more questions that the Scialog data can help answer,” said Abrams. “Some big questions we're working on now include how facilitators impact who speaks during group discussions, and ultimately who forms a new collaboration. This relates to the larger question of what good leadership looks like.”
He also said that as they aim to understand the patterns of scientific discussion with mathematical models, they are also looking into whether certain team compositions are more likely or less likely based on gender or minority status, and how inclusive scientific discussions are (do women speak less than men, and are they interrupted more?)
“Our research focused on the context of scientific conferences but may have wide-ranging implications in business, education, and even romantic settings,” said Zajdela. “In today’s polarized world, designing encounters between people from different backgrounds and opinions where productive conversations can occur and new teams can form is increasingly important.”
RCSA Senior Program Director Richard Wiener, who led the creation of the Scialog program in 2010, said advances in network science and an understanding of the importance of networks in a variety of situations prompted RCSA to build into its program regular measurement of how well the conferences’ unique format works.
“From the first Scialog in 2010, we asked participants to fill out a survey indicating how well they were connected to everyone else at the meeting so we could look at year-to-year changes in network connectivity,” Wiener said.
"When we shifted to virtual meetings in 2020, we realized that since we were on Zoom, we could record the conversations, and such recordings would be a source of important data to understand how people interact, specifically in scientific discussions and more generally in many types of conversations,” he said. “We have continued to record the discussions after returning to in-person meetings, which will give us the opportunity to get actionable items to improve the way we design conferences.”
The pandemic presented a host of technical challenges for RCSA conference organizers. Early in 2020, RCSA hosted several online gatherings in response to the pandemic. These gave RCSA an opportunity to explore different technologies and iron out best practices before launching the first fully online Scialog, the inaugural meeting of Negative Emissions Science, in fall 2020.
“Despite the challenges of going to an online environment, it was important to know that we could still catalyze collaborations effectively,” said Wiener. “It was initially surprising to see in the study that placing people in groups in formal interactions played a bigger role in getting people to form teams, but when we thought about it, we realized it was because there was a reduced opportunity for participants to interact in informal settings, like at meals, which play a larger role at in-person meetings.”
One of the biggest challenges of online meetings was keeping people engaged.
“We made a really strong effort to keep the conferences highly interactive,” said Wiener, who added that beyond prescribed interactions in different-sized breakout groups, trying to recreate informal interaction in a “happy hour” setting through use of the Gather platform was a mixed success.
“There was less willingness to come to an informal interaction, just because people were tired from a whole day of Zoom, and they had other things going on,” Wiener said. “There was also a general malaise during the pandemic, a psychological challenge for everybody. As we know, it was a very difficult time. The fact that these virtual meetings worked as well as they did truly is incredible.”
Whereas teams at in-person Scialogs write proposals for seed funding in real time, organizers wanted to minimize online fatigue, giving virtual participants a week after the conference to write their proposals. They also reduced the length of the online meetings to two days instead of three. Even so, the average percentage of participants who submitted a proposal was 8% lower – 80% at virtual conferences versus 88% at in-person meetings.
Scialog returned to in-person in spring 2022 with the second meeting of Advancing BioImaging, but some conference tools and techniques from the Zoom era became an integral part of the conferences and have remained in use – most notably, interactive ways of collaborating in the cloud using shared documents on a Google drive, and all-digital conference materials.
“The online meetings showed that we really did not need to go back to using room-size Post-it Notes, printed conference booklets, and colored paper handouts,” said RCSA Senior Program Director Andrew Feig.
He said a significant difference from pre-COVID times is that the Fellows now know before the conference with whom they are going to be meeting in their small-group assignments.
“They look each other up, do their due diligence, and think about their possible partnerships,” Feig said. “That creates an even stronger opportunity for remembering each other and collaborating,” Feig said.
For now, Scialogs will remain fully in-person, augmented as needed by hybrid technologies.
“We've decided against doing fully hybrid meetings because we think there are real challenges in keeping people on an equal footing,” said Wiener. “We want as much as possible for people to have the full experience, but we have chosen to do a modest amount of hybrid when people already at a conference have needed to isolate due to COVID, for example.”
One keynote speaker gave her address from her hotel room, and two different Scialog Fellows who needed to isolate were still able to participate virtually, find collaborators, and write successful proposals.
Though virtual meetings have their advantages, “there's something important to be said about putting people in a room together and seeing what happens when everyone is separated from their day-to-day lives and just focused on the event,” Feig said.
“That focus is where some of the magic happens with an in-person meeting that’s well-designed,” he said. “That doesn't mean that all meetings are well-designed. But if you can make those meetings exciting and engaging in ways that lead to deep interaction, you can get the outcomes that you're looking for.”
In the debate between virtual and in-person, Zajdela said each format offers lessons for the other.
“Virtual conferences may be less prone to homophily effects – that is, people joining teams with those similar to themselves – so they can promote diversity in teams,” Zajdela said. “But how can we replicate this effect at in-person conferences, and how can we foster community in an online environment through better-designed informal interactions?”
No matter the format, Zajdela said the goal is the same: “Can we engineer encounters so that the right people and ideas meet to foster greater scientific progress and innovation?”
Feig agrees.
“The big takeaway is that these two media are different. You can design an event for in-person, or you can design an event for virtual, but you don’t design the same event with the same structures. You need to reflect the media.”
