Publications
Check Google Scholar for latest
publications
MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian, Danaé Metaxa
An open-source evaluation infrastructure that connects AI conversation
trajectories with user-defined goals and outcome assessments.
AIES 2026
evaluationcommunity-centered AI
Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
Victor Ojewale, Suresh Venkatasubramanian
A framework for evaluating whether autonomous agents recognize, communicate,
and act appropriately when they should abstain.
AIES 2026
agentsabstentionsafety
More Is Not Better: Visual Uncertainty Cues and the Fragility of Trust Calibration in LLM-Assisted
Decision Making
Victor Ojewale, Julia Ryan, Suresh Venkatasubramanian, C. Malik Boykin
An examination of how visual uncertainty cues shape trust calibration in
LLM-assisted decision making.
Computers in Human Behavior: Artificial Humans, 2026
HCIuncertaintytrust
Audit Trails for Accountability in Large Language Models
Victor Ojewale, Harini Suresh, Suresh Venkatasubramanian
A proposal for traceable records that support accountability investigations of
large language model behavior.
arXiv preprint arXiv:2601.20727, 2026
AI auditingaccountabilityinfrastructure
Beyond Static Leaderboards: A Roadmap to Naturalistic, Functional Evaluation of LLMs
Victor Ojewale, Suresh Venkatasubramanian
A roadmap for moving LLM evaluation beyond static leaderboards toward
naturalistic, functional assessments.
Second Workshop on Language Models for Underserved Communities (LM4UC) at AAAI 2026
LLM evaluationbenchmarks
Multi-lingual Functional Evaluation for Large Language Models
Victor Ojewale, Inioluwa Deborah Raji, Suresh Venkatasubramanian
A functional evaluation approach for assessing large language model behavior
across languages.
Findings of ACL, 2026
multilingualLLM evaluation
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane & Inioluwa Deborah Raji
An analysis of the gaps and opportunities in the tools and infrastructure used
to audit AI systems.
CHI, 2025
AI auditingtoolingHCI
AI Auditing: The Broken Bus on the Road to AI Accountability
Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione & Inioluwa Deborah Raji
A survey of the AI auditing landscape and the structural limits that prevent it
from delivering accountability on its own.
SaTML, 2024 (Distinguished Paper)
AI auditingaccountability
Comment on NIST–2023–0309
Inioluwa Deborah Raji, Abeba Birhane, Briana Vecchione, Ryan Steed & Victor Ojewale
NIST Request for Information (RFI)
Comment on NTIA-2023-0005
Inioluwa Deborah Raji, Briana Vecchione, Abeba Birhane, Ryan Steed & Victor Ojewale
NTIA AI Accountability Request for Comment
Feedback on Data Access, Digital Services Act
Inioluwa Deborah Raji, Briana Vecchione, Abeba Birhane, Ryan Steed & Victor Ojewale
European Commission, Call for Evidence