By Katrina Pugh, Ph.D., Information & Knowledge Strategy (IKNS)
This fall, colleagues of mine at MIT and the University of Maine published an article in Neural Computing and Applications about research we did on conversation features. By using generative AI to map and quantify rhetorical intent in conversation, our research provided clues to better collaboration, particularly in sustainability contexts.
As “civil” conversation is increasingly rare these days, we wanted to see whether “good” and “bad” moves in conversation create measurable, statistically significant impacts on conversation outcomes. Our multidisciplinary research team of marine scientists, data scientists, game theoreticians, and behavioral scientists started with a hypothesis that rhetoric in sustainability-related town-hall gatherings contains the seeds of its outcomes, and that participants can change rhetoric for the better.
My colleague Nancy Dixon, Ph.D., and I proposed a dialogue model based on principles first introduced by William Isaacs at MIT. This model combines the practical conversational elements of introducing, developing, and summarizing concepts with the social elements of drawing each other out, acknowledging each other’s worth, and being generally civil. As such, the five discussion disciplines in our model describe rhetorical intent: Integrity (statements), Integrity-Q (questions), Courtesy (respect, positivity), Inclusion (acknowledgment, welcome), and Translation (synthesis).
To train the AI model, we chose a particularly controversial sustainability topic in the state of Maine: aquaculture. Aquaculture promises livelihoods and food security but can be challenged by individuals concerned with navigation, biodiversity, traditional lifeways, and riparian landowners. From seven aquaculture town-hall-like meetings, and similar policy or network conversations, we generated transcripts and hand-coded more than 1,000 utterances for the discussion disciplines. We also surfaced 300 phrases corresponding to conversation outcomes: Intent-to-Act, Options-Generation, and Relationship-Building.
Using these training data, we evaluated the accuracy of different machine learning models to automatically identify the occurrence of the five disciplines in a given conversation—in essence, we were testing large language models and using transfer learning even before the release of ChatGPT. The model that provided the best outcome combined Bi-directional Encoder Representations from Transformers (BERT, the ancestor of ChatGPT) and Residual Network (ResNet, an optical model).
Next, we used the trained model and information retrieval on nearly 600 open-source transcripts containing more than 21,000 utterances. With that, we could do statistics. A binary logistic regression analysis showed that two discussion disciplines, Inclusion and Courtesy, had positive, statistically significant impacts on Intent-to-Act: A 10 percentage-point increase in the share of the Inclusion or Courtesy yielded a 45% or 34% increase, respectively, in the likelihood of Intent-to-Act. Using conversation, we seemed to show what our team member Erez Yoeli had previously also found by looking at how “being visible” increased commitment-fulfillment in energy conservation.
So what does this all mean? First, organizations can use these types of models to profile conversations and provide options to speakers who deliberate policy, facilitate teams, lead analyst calls, or sit on boards. This is important not only for sustainability matters but for all matters in civil society.
Second, we can bring this type of insight more fully into our teaching. We researchers were all teaching at universities where there is a strong ethos of intelligent conversation and engagement in our classrooms.
Ultimately, gaining a better understanding of conversation features and their impacts may lead to innovation, solution design, and ongoing collaboration. We are seeding new AI tools to spread this.
About the Program
The Columbia University M.S. in Information & Knowledge Strategy (IKNS) program provides students with foundations in information science, organizational psychology, and change management as well as practical skills in project management and executive leadership.
