Multi-Perspective Narratives (MPNs) are ubiquitous and very useful for verifying information from different alternative narratives, and thus, MPNs facilitate more informed decisions by providing a concise overall picture of the current situation. Despite great progress in the area of natural language processing (NLP), computers are still far from being able to analyze multi-perspective narratives accurately; addressing this limitation is the focus of this project.
In this ongoing project, we are developing a novel human-AI collaborative framework called CAMPeN (``Collaborative Analytics of Multi-Perspective Narratives’’), where the AI, given multiple alternative narratives as input, first extracts a set of candidate clauses w.r.t. the Overlap-Unique-Conflict criteria, separately, in a zero-shot fashion. Next, the human actively verifies clauses that were labeled with low confidence by the AI. Finally, the machine braids the high-confidence/verified clauses to construct the ultimate Overlap-Unique-Conflict style summary, which will be presented to the user. The major benefits of the proposed framework are two-fold: 1) it enables domain experts in fields other than machine learning/NLP (e.g., a military general) to quickly dig out/verify interesting hypotheses from multiple alternative narratives/descriptions without worrying about the underlying computational techniques and thus, democratizes AI, and 2) it can quickly verify facts and claims about real-world events by analyzing alternative narratives and braid them into a single narrative with a higher degree of Information Assurance.
This project adopts both zero-shot and reinforcement learning approaches for extracting overlapping, unique, and conflicting information from alternative narratives that can be trained in a self-supervised fashion without requiring a large collection of training data; therefore, the proposed framework needs minimal human supervision in comparison to the existing Multi-Document Summarization techniques. Additionally, the project borrows intuitions and insights from the classical set theory and applies the properties of set operators to develop novel reward/loss functions to enable effective training of reinforcement learning-based extraction networks.
The proposed work includes the following objectives.
In open-domain dialog systems, it is often uncertain how the end user would expect a new conversation to be grounded and structured. Therefore, the ideal solution must engage in a pre-conversation with the user about their expectations and preferred knowledge base for grounding purposes before the actual conversation happens. In other words, a “Conversation about Conversation”, i.e., a “Meta-Conversation” should happen with the user beforehand.
Based on this idea, we are currently developing an Artificial Intelligence-based Conversational Framework to create dialog-based interactive laboratory experiences for middle school science students and teachers in the context of simulation-based science experiments. A key component of the framework is an intelligent conversational agent (SimPal) that actively learns from teachers through a “Meta-Conversation” to solicit their instructional goals associated with simulation experiments and store them using a computational representation. In other words, the school teacher actively teaches the machine/agent what the instructional goals are for a particular scientific experiment in plain natural language. The agent then uses this representation to facilitate and customize an interactive knowledge-grounded conversation (powered by state-of-the-art Large Language Models) with students as they run experiments to enhance their learning experience. Unlike existing intelligent tutoring systems and pedagogical conversational agents, SimPal can work with any off-the-shelf third-party simulations, a unique feature of this project enabled by our proposed Meta-Conversation technique.
]]>A big challenge in democratizing AI is that no single AI model can be pretrained to be skilled in all possible tasks a user may want to perform. Therefore, being able to learn new skills in an ad hoc fashion is essential. To address this challenge, we are developing a Conversational Data Science solution that is capable of acquiring new predictive skills on the fly through intuitive, natural conversations with the user. In our ACM Computing Surveys 2022 paper, we highlighted the core technical challenges that need to be addressed:
However, one big hurdle in materializing Conversational Data Science is to ensure a conversation that is grounded in a unique data set that the user may provide from an unseen domain on the fly. To address this challenge, we recently proposed a prompting taxonomy called TeLER to design effective prompts for Large Language Models (LLMs) in order to build a natural language interface between humans and the AutoML tools (e.g., Scikit-Learn), which, in turn, facilitates acquiring new predictive skills via self-directed learning. Our experimental results (available on Arxiv) demonstrate the effectiveness of the proposed TeLER-taxonomy-based prompting technique for knowledge grounding.
We designed the architecture of our “Conversational Data Science” framework with four dialogue states: Data Visualization, Task Formulation, Prediction Engineering, and Result Summary and Recommendation. Each state marks a unique conversation phase, impacting the overall user-system interaction. Multiple LLM instances, serving as “micro-agents’’, ensure a cohesive conversation flow, granting us granular control over the conversation’s progression. In summary, we designed and developed an end-to-end system that demonstrates the viability of “Conversational Data Science” by evaluating the potency of taxonomy-based prompting of LLMs in solving such an ill-defined complex task.
This project focused on developing a novel metric for quantifying author influence in the context of narrative braiding, which is largely an unexplored research area till now. By definition, a braided narrative is created from the contributions of multiple authors. For this thrust, we also assume that there is a separate entity called the editorial board that polishes and edits the raw contributions of individual authors and is in charge of creating the final braided narrative. Under these assumptions, we consider three scenarios for quantifying author influence: 1) Quantify Author Influence for Single Author Contribution - Single Braided Narrative Scenario, 2) Quantify Author Influence for Multiple Authors - Single Braided Narrative Scenario, and 3) Quantify Author Influence for Multiple Authors - Multiple Braided Narratives Scenario. Another basic idea here is that influential authors become more trustworthy over time and serve as reliable sources in the future.
Previous studies have shown that popular Natural Language Generation (NLG) and Information Retrieval (IR) and evaluation metrics, e.g., nDCG, ROUGE, MAP, are not robust and often do not correlate with the utility perceived by humans. In this project, our main goal is to investigate how to make NLP/IR evaluation metrics more utility-centric.