Search RESI
Search stays in your browser.
92 entries
No results. Try a shorter term, a researcher’s surname, or a topic such as “control.” You can also explore Research, People, or Support RESI.
RESI-MIT Seminars
Research talks on the foundations of AI safety, hosted by RESI and MIT.
Making superintelligence safe by design
The Institute for Responsible Superintelligence (RESI) is an independent nonprofit research institute developing ways to make powerful AI systems trustworthy and controllable by design.
Safe by design
Safe by design means safety properties are specified in advance and achieved by mechanisms whose guarantees can be analyzed before deployment.
The modern cryptography precedent
Modern cryptography developed definitions of computational security and primitives that could provably satisfy them under explicit mathematical assumptions: public-key encryption, digital signatures, zero-knowledge proofs, multi-party computation, and obfuscation.
Research excellence and independence
RESI combines research excellence with independence of industry and profit incentives.
What success will look like
Progress can be demonstrated through rigorous definitions of safety objectives; mechanisms and architectures with useful guarantees; results establishing what is achievable, impossible, or dependent on particular assumptions; evidence about how guarantees hold up when systems are combined; and working prototypes that can be evaluated in practice.
A research community spanning technical, legal, and economic expertise
Superintelligence safety has many aspects, and no single discipline sees the whole picture. Participating researchers will span computer science and other fields relevant to AI safety, including mathematics, economics and law, with an active visitor program.
How research shapes AI systems and policy
Our long-term goal is for safe by design approaches to be adopted in advanced AI systems and for research to inform policy and regulation.
Research outputs
Models, definitions, and impossibility results; mechanisms and composition; implementations and proofs of concept. Develop models and define safety requirements, analyze mechanisms, establish limits, and evaluate prototypes.
AI-assisted research
RESI is developing research systems that use frontier AI to help investigate AI safety.
Researchers, visitors, and students
RESI hosts a small number of researchers and visitors and welcomes inquiries from researchers whose expertise could contribute to RESI’s research, whether or not they have previously worked on AI safety.
How to contribute
To discuss support and the research it would enable, contact funding@resi.org. We also welcome conversations about compute credits, infrastructure, and other forms of support.
Help us build the foundations for superintelligence that is safe by design
RESI brings together researchers with a record of turning foundational ideas into usable mechanisms in cryptography and other fields.
What support enables
Our greatest need is support for people. Alongside reliable compute and a sustained interdisciplinary research environment, this funding will help establish the core institute and give its research community a durable base.
Working groups
Focused research agendas within and across RESI’s broad research themes.
Privacy
This website has no analytics scripts, advertising trackers, or embedded data-collection forms. Its search runs in your browser.
Verification and oversight
What can a less capable overseer reliably verify about a more capable AI’s properties, answers, and actions without reproducing all of the system’s work? A useful verification method must state which claims it can check, put bounds on how often it can be wrong, and set reasonable bounds on the resources its verifier needs. In mathematics, there are exact definitions of correct and incorrect statements and what constitutes a proof. For the open-ended answers generated by modern AI systems in many other fields, we need to define anew what counts as sufficient evidence and how to evaluate it before letting the system take action.
Control and delegation
What authority can we delegate to AI while preserving our ability to intervene as AI becomes more intelligent than us? Explore protocols that make the scope of a delegated task explicit: which tools may be used, which actions require approval, and when execution should stop. Guarantees should account for the environment and the power of the user, not just the AI model.
Composition
Which safety properties can be better achieved when connections between models, tools, and agents are limited? A property established for one component may fail when it interacts with another. Study interfaces, shared resources, patterns of interaction, and assumptions under which local guarantees support a claim about an entire system, as well as constructions that show when composition is impossible.
Shaping multi-agent dynamics
How should we design agents and structure their interactions to build safe multi-agent systems with provable guarantees? Behavior of an AI system often emerges from complex interactions between many agents (swarms). These agents cooperate, compete, assume roles, and communicate to further their individual or shared goals. Though swarms of agents have emerged only recently, the problem of constraining and shaping the dynamics of interacting parties is well studied in other settings. We draw on ideas from game theory, distributed computing, social psychology, and law to govern these dynamics and achieve desirable outcomes.
Formal verification
How can formal verification enable powerful AI throughout software development without creating new risks of cyberattacks or otherwise dangerous behavior? Develop mathematical proofs that the software surrounding AI enforces safety constraints, including containment that holds regardless of a model’s internal decisions. The scope extends from securing conventional computing infrastructure to verifying the hardware and software that run deep learning, and ultimately to systems that repeatedly improve themselves. Alongside these technical problems, study how to organize and equip researchers to produce the necessary proofs before the risks materialize.
Impossibility as a safety result
What limits on technologically achievable safety guarantees can be proved, and how can those limits inform which laws, policies, and regulations must be adopted? Not all safety goals are achievable, and sometimes provably so. Understanding these limits can help policymakers identify requirements that cannot be satisfied together, justify imposing restrictions, and clarify what access auditors need to assess safety claims.
Leveraging AI for AI safety research
How can we use AI to develop new AI safety concepts, definitions, insights, and formal guarantees? As AI grows smarter, it gains a greater capacity to act in unanticipated ways and to cause harm despite human safeguards. For humans to keep pace, we must use AI wisely as a tool to improve and accelerate AI safety research. This includes exploring whether recent developments in AI-assisted mathematics can accelerate the creation of mathematical foundations for AI safety.
Incentives and institutions
How should incentives work alongside technical safeguards? Technical mechanisms operate within organizations, markets, and legal institutions. Model those interactions and assess which guarantees remain useful when the people and systems involved have different objectives, and under what assumptions.
Cryptography and security
How do we repurpose cryptographic tools for secrecy, authentication, multi-party secure computation and obfuscation to navigate the AI multi-agentic world? Cryptography studies how parties can compute and communicate securely even when some behave maliciously. For AI agents, we need to model agents capabilities and adversary agents completely differently to protect users. Each traditional cryptographic goal requires a new precise model of the adversary's computational power, information, and ability to collude as well as a proof of security.
Legal alignment
How can legal constructs be leveraged to develop technical methods for building safe by design superintelligence? Explore how the law’s concepts, principles, and institutions can serve as a blueprint to propose technical methods for robustly safe AI systems -- and develop theoretical and empirical case studies of such legal alignment.
Attribution
Can we attribute model decisions to training data? The map from training data or information in large contexts can be opaque. Tools for disentangling the effects of these kinds of information on the behavior of AIs could serve as essential tools for interpretability and safety, but at present, all such tools are heuristic and unverifiable. If we want to fully understand AI behavior, we need reliable and rigorous ways to trace model decisions and outputs back to data.
Chloé Bakalar
Moral and political philosopher working on applied and normative AI ethics, including the development, evaluation, and governance of advanced AI systems. Her experience includes leading AI ethics at OpenAI and serving as chief ethicist at Meta.
Emily Uyeda Kantrim
Emily Uyeda Kantrim’s work spans nonprofit, government, research, and social innovation, with a focus on turning ambitious ideas into durable institutions, scalable systems, and measurable public impact.
Shafi Goldwasser
Scientific co-founder. MIT cryptographer and theoretical computer scientist, co-inventor of zero-knowledge proofs, and Turing Award winner. Her work helped establish the foundations of modern cryptography, including how to define security precisely and prove that a system satisfies it.
Adam Tauman Kalai
Scientific co-founder. AI safety and ethics researcher whose work spans learning theory, game theory, and fairness. Previously a research scientist at OpenAI, he bridges frontier-AI practice with theory, with recent work on hallucinations, fairness in chatbots, and safer generation through model consensus.
Ran Canetti
Boston University professor working on the design, analysis, and composition of cryptographic protocols: how to build secure systems and preserve their security when components are combined. His research also includes program obfuscation and its applications.
Miranda Christ
MIT postdoctoral researcher with a PhD from Columbia. Works on practically motivated problems in cryptography, particularly its intersection with machine learning, including watermarking methods for AI-generated content.
Vinod Vaikuntanathan
Scientific co-founder. MIT cryptographer and theoretical computer scientist with foundational work in lattice-based cryptography and fully homomorphic encryption, enabling computation on encrypted data. He explores links among cryptography, statistics, and machine learning. Gödel Prize winner.
Greg Gluch
Theoretical computer scientist using tools from cryptography and computational complexity to study AI safety and alignment. A Simons-Berkeley postdoctoral researcher based at MIT, he earned his PhD at EPFL.
Yannai A. Gonczarowski
Harvard economist and computer scientist working at the intersection of economic theory and theoretical computer science, including mechanism design and market design. He leads RESI’s working group on Game-Theoretic Foundations of AI Safety. ACM SIGecom Doctoral Dissertation Award winner.
Sam Gunn
MIT mathematics postdoctoral researcher working on cryptography and the foundations of AI. His research includes watermarking generated content and understanding how training data influences model behavior, alongside work in quantum computing. He earned his PhD at UC Berkeley.
Dylan Hadfield-Menell
MIT professor leading the Algorithmic Alignment Group. His research examines how AI systems can learn from human feedback and act in accordance with human goals and values, alongside methods for evaluating and overseeing deployed systems.
Tal Herman
Simons-Berkeley postdoctoral researcher based at MIT, with a PhD from the Weizmann Institute. Works on complexity theory and cryptography, with a particular focus on interactive proofs for data science and methods for verifying properties of probability distributions.
Yael Tauman Kalai
MIT cryptographer and theoretical computer scientist working on verifiable delegation, interactive proofs, and cryptographic protocols. Her work develops ways to check computations carried out by other systems efficiently and securely. Winner of the ACM Prize in Computing.
Noam Kolt
Assistant professor in law and computer science at the Hebrew University, where he leads the Governance of AI Lab. His research examines the governance of AI agents, how to evaluate AI systems’ compliance with law, and how advanced AI may affect the rule of law.
Anat Perry
Associate professor of psychology at the Hebrew University and fellow at Harvard’s Berkman Klein Center. She studies empathy and social cognition, including how people experience and value AI-generated empathy and what human–AI interaction means for relationships and human flourishing.
Daniela Rus
MIT roboticist and director of the Computer Science and Artificial Intelligence Laboratory (CSAIL). Her research spans autonomous systems and embodied intelligence, including methods for safe control and planning in the physical world. MacArthur Fellow.
Jonathan Shafer
MIT postdoctoral researcher and incoming professor at the Weizmann Institute of Science, working on computational and statistical learning theory and its connections to cryptography. His research includes methods for verifying machine learning and defending models against backdoors. He earned his PhD at UC Berkeley.
Yaron Singer
CEO and co-founder of Frontier Security, working on AI security, adversarial robustness, and robust optimization. Previously a VP at Cisco, founder and CEO of Robust Intelligence (acquired by Cisco), and Harvard professor of computer science and applied mathematics. Sloan Fellow.
Andrew V. Sutherland
MIT mathematician working in computational number theory and arithmetic geometry. His work also spans AI-assisted mathematical discovery, formal verification, and large-scale mathematical databases, connecting mathematical research with tools for discovering and checking new results.
Mirac Suzgun
Stanford computer science PhD candidate studying the capabilities and limitations of language models, particularly reasoning, factuality, and hallucinations. His work also examines the use of AI in law and medicine, alongside legal scholarship on AI governance. He holds a JD from Stanford Law School.
Neekon Vafa
Junior Fellow of the Harvard Society of Fellows, with a PhD in mathematics from MIT. Works on cryptography and its connections to statistics and trustworthy AI, including the security of machine-learning models against hidden backdoors.
Rebecca Wexler
Alfred W. Bressler Professor of Law at Columbia. Studies evidence and criminal law, with a particular focus on how technological and legal choices shape privacy, secrecy, and accountability. Her work includes the authentication of AI-generated evidence and applications of zero-knowledge proofs to legal verification.
Or Zamir
Assistant professor of computer science at Tel Aviv University working on algorithms, data structures, combinatorics, graph theory, cryptography, and the foundations of provable AI safety. Previously a member at the Institute for Advanced Study and an associate research scholar at Princeton University, he earned his PhD at Tel Aviv University.
A Cryptographic Perspective on Mitigation vs. Detection in Machine Learning
G. Gluch, S. Goldwasser. Preprint, 2025. Shows that, for certain tasks involving generated outputs, a defense can limit harmful behavior even when the harmful inputs cannot be efficiently identified.
A Theory for Worst-Case vs. Average-Case Guarantees for LLMs
Noga Amit, Shafi Goldwasser, Orr Paradise, Guy N. Rothblum. NeurIPS 2025. Models that can prove their answers A framework for learning models that supply interactive proofs of their outputs. Demonstrated on greatest-common-divisor computation, not unrestricted natural-language answers. A framework for learning models that supply interactive proofs of their outputs. Demonstrated on greatest-common-divisor computation, not unrestricted natural-language answers.
Algorithmic Collusion by Large Language Models
S. Fish, Y. A. Gonczarowski, R. I. Shorrer. EC 2026. Experiments show that pricing agents based on large language models can reach prices above competitive levels, with outcomes sensitive to changes in their instructions.
BarrierNet: Differentiable Control Barrier Functions for Learning of Safe Robot Control
W. Xiao, T.-H. Wang, R. Hasani, M. Chahine, A. Amini, X. Li, D. Rus. IEEE Transactions on Robotics 2023. Introduces a trainable safety layer that constrains a robot’s actions to satisfy specified requirements, with guarantees that depend on the model of the robot and its environment.
Consensus Sampling for Safer Generative AI
A. Kalai, Y. Tauman Kalai, O. Zamir. Preprint, 2025 (revised 2026) · Presented at IASEAI 2026. Selects an output from several models, or declines to answer, with a provable limit on the risk of harmful output relative to the more reliable models. The method requires access to each model’s output probabilities. Producing useful answers also depends on sufficient overlap among the reliable models’ output distributions.
Evaluating large language models for accuracy incentivizes hallucinations
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang. Nature · Published April 22, 2026. Why language models hallucinate Why do models guess instead of admitting uncertainty? This work analyzes how training and evaluation reward guessing, and argues for scoring systems that make acknowledging uncertainty a better strategy. This work analyzes how training and evaluation reward guessing, and argues for scoring systems that make acknowledging uncertainty a better strategy. Changing the scoring rule is not a guarantee that hallucinations disappear.
Governing AI Agents
N. Kolt. Notre Dame Law Review, 2026. Examines the responsibilities and limits of authority that arise when AI agents act for people, and explains why existing approaches to monitoring and accountability may be insufficient.
How to Avoid Debate: Scalable AI Safety via Doubly-Efficient Interactive Proofs
L. Chen, Y. Tauman Kalai, Z. Xi. ICML 2026. Develops methods for checking certain computations that use external information, such as database records or human judgments. The guarantees require specific structure in that information or computations that tolerate a limited number of errors.
How to sketch a learning algorithm
Sam Gunn. Preprint, April 2026; revised June 2026. Data deletion and attribution How would a model behave if some training data had been left out? A sketch of the learning algorithm predicts these counterfactual outputs, supporting data deletion queries and attribution, with error and failure guarantees under a stability assumption. How would a model behave if some training data had been left out? A sketch of the learning algorithm predicts these counterfactual outputs, supporting data deletion queries and attribution, with error and failure guarantees under a stability assumption. The method predicts counterfactual measurements rather than outputting a retrained model. Evidence for the stability assumption comes from small-scale microgpt experiments.
Legal Alignment for Safe and Ethical AI
N. Kolt et al. TMLR 2026. Surveys how legal rules, principles and methods can be used to specify how AI systems should act and to inform their design, and sets out three research pathways for doing so.
Oblivious Defense in ML Models: Backdoor Removal without Detection
Shafi Goldwasser, Jonathan Shafer, Neekon Vafa, Vinod Vaikuntanathan. STOC 2025 · Preprint, November 2024. Defense without detection Can a defense work without identifying an attack? This work gives backdoor-mitigation guarantees under assumptions about the ground-truth function. Develops methods for neutralizing hidden triggers that alter a model’s behavior, without first detecting them. The guarantees apply to learning tasks with specified mathematical structure in the ground-truth function.
On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment
S. Ball, G. Gluch, S. Goldwasser, F. Kreuter, O. Reingold, G. Rothblum. Preprint, July 2025 · ICLR 2026. Understanding the limits of filtering Cryptographic constructions expose computational limits of certain input and output filters. The results depend on explicit hardness assumptions and formal settings. Establishes computational barriers to external prompt and output filtering under cryptographic hardness assumptions and specified relationships between the model and filter. These worst-case results do not say that filtering has no practical value.
Planting Undetectable Backdoors in Machine Learning Models
S. Goldwasser, M. Kim, V. Vaikuntanathan, O. Zamir. FOCS 2022. Shows how a model can appear to function normally while containing a hidden trigger that changes its behavior. Under cryptographic assumptions, such backdoors can evade inspection within feasible computing limits.
Pseudorandom Error-Correcting Codes
M. Christ, S. Gunn. CRYPTO 2024. Introduces a way to hide information in apparently random data while allowing a key holder to recover it after specified alterations. This underpins methods for watermarking AI-generated content and hiding communication.
Superintelligence and Law
N. Kolt. Preprint, 2026 · Harvard Journal of Law & Technology, forthcoming. Sets out how AI agents acting autonomously become subjects of law, users of legal instruments such as contracts and courts, and in time writers and enforcers of law, and which assumptions in legal theory and doctrine that calls into question.
Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange
Vinod Vaikuntanathan, Or Zamir. FOCS 2026, accepted · Preprint, April 2026. The limits of transcript auditing Can ordinary exchanges conceal a second conversation? Under stated cryptographic and entropy assumptions, this work constructs covert communication between agents with no pre-shared secret, revealing limits of passive transcript auditing. Shows how cooperating AI agents could hide secret exchanges within ordinary-looking messages, even without a shared secret at the outset. The construction assumes cryptographic hardness and sufficient entropy in messages, and concerns passive transcript auditing.
Cryptographic verification
Shafi Goldwasser. Develops methods that enable AI systems to provide convincing evidence that their answers meet appropriate standards of correctness. Mathematical claims may admit independently checked proofs; other domains require task-appropriate standards of correctness.
Modular architectures
Adam Tauman Kalai. Studies how restricted modules, used alone or in structured interactions, can make AI systems safer by limiting what each component can do. Could a constrained component control spending while a more capable model proposes actions without direct access to payment credentials?
Monitoring and control
Vinod Vaikuntanathan. Investigates when monitoring can detect subversive behavior in AI models and agents, where it falls short, and what additional controls may be needed.
AI harnesses
Jonathan Shafer. Investigates how frontier AI can assist with definitions, conjectures, proofs, counterexamples, and experiments.
Game-theoretic foundations of AI safety
Yannai A. Gonczarowski. Investigates multi-agent AI alignment through game-theoretic and economic tools, asking where those tools are appropriate and where new ones are needed.
Legal alignment
Noam Kolt. Explores how legal constructs such as separation of powers, corporate governance, and fiduciary duties can inform technical approaches to safe by design superintelligence. One proposed direction is a theoretical or empirical case study of a legal construct adapted to a technical alignment problem. The focus is on using legal ideas to improve alignment, rather than primarily on applying existing legal doctrine to AI.
Formal verification
Adam Chlipala. Adam Chlipala's announced agenda asks how formal verification could support powerful AI in software development while reducing security risks. It distinguishes general software infrastructure, AI-specific infrastructure, and recursively self-improving systems. The agenda addresses both technical questions and how to organize the work, including which directions are ready to pursue.
Cryptography and cybersecurity
Yaron Singer. Uses mathematical foundations and modeling to study how to secure AI applications, protect against AI-generated attacks, and use AI to improve cyber defenses.
Ethics and superintelligence
Chloé Bakalar. Examines fairness, human control, and the distribution of resources in a world with superintelligent AI.
Data attribution
Sam Gunn. Studies how to attribute model behavior to subsets of training data with provable guarantees under concrete, realistic assumptions. Proposed directions include inference-time attribution, lower bounds, experimental tests of attribution schemes, and adaptive attribution queries.
AI safety using quantum effects
Greg Gluch. Investigates whether quantum resources could support restrictions on access, reuse, or revocation of AI capabilities that classical software alone cannot provide.
What do you mean by superintelligence?
Superintelligence means systems that greatly exceed individual humans, and potentially expert teams, across a broad range of cognitive tasks.
Does a proof make an entire AI system safe?
No. A proof concerns a specified property in a specified model, under stated assumptions. Whether those assumptions describe a deployed system, and whether the property captures what people need, are separate questions.
Does this replace testing or alignment research?
No. Formal analysis, empirical evaluation, and work on alignment address different parts of the problem. A useful design needs to be checked both against its specification and against the conditions in which it will operate.
Why an independent institute?
RESI combines academic independence with sustained focus on the foundations of AI safety. It brings researchers together across fields, with an open, mission-first culture.
How can research shape AI systems and policy?
RESI designs with practical constraints in mind and tests promising constructions in real systems. We will share proofs of concept so that frontier AI developers can evaluate and adopt them. Success also means influencing policy and regulation in ways that support human flourishing.
Do I need an AI safety background to participate?
We welcome inquiries from scientists whose tools could bear on these problems, whether or not they have worked on AI safety before. Email hello@resi.org.
Fireside Chat with Chloé Bakalar: From OpenAI and Meta to Responsible Superintelligence
Fireside Chat with Chloé Bakalar: From OpenAI and Meta to Responsible Superintelligence Chloé Bakalar · RESI / University College London Wednesday, October 28, 2026 · 3–4 p.m. Eastern Kiva Seminar Room (32-G449), MIT Stata Center 32 Vassar Street, Cambridge, MA Join on Zoom · Submit a question Details About the conversation What does it take to put ethical principles into practice at the frontier of AI development? Should AI alignment go beyond preventing harm to actively promoting human flourishing? And what new ethical questions arise as AI systems approach superintelligence? Drawing on Chloé’s experience leading AI ethics at OpenAI and Meta, as well as her research in moral and political philosophy, we’ll explore the challenges of translating ethics into practice, the emerging idea of positive alignment, and the ethical implications of increasingly powerful AI systems. Unlike our usual research seminars, this will be an informal, wide-ranging conversation, with opportunities for audience questions. You can submit questions in advance using the link above. We’ll also welcome questions during the conversation. About Chloé Bakalar Chloé Bakalar leads RESI’s ASI Ethics and Fairness working group and is a Senior Research Associate at University College London. Previously, she served as AI Ethics Lead at OpenAI and Chief Ethicist at Meta, where she worked on integrating ethical considerations into AI development and deployment. She is also a coauthor of the recent paper Positive Alignment: Artificial Intelligence for Human Flourishing , which proposes moving beyond simply preventing harmful AI behavior toward actively supporting human well-being and flourishing.
The Artificial-Empathy Paradox: Why Human Empathy Still Matters
Wednesday, October 21, 2026 · 3–4 p.m. Anat Perry · Hebrew University of Jerusalem / Harvard Berkman Klein Center The Artificial-Empathy Paradox: Why Human Empathy Still Matters Kiva Seminar Room (32-G449), MIT Stata Center 32 Vassar Street, Cambridge, MA Abstract and biography Abstract AI-generated empathy is convincing and well received until recipients realize it is artificial; a phenomenon I term the ‘artificial-empathy paradox’. We test this phenomenon across nine experiments, in which identical empathic responses to participants’ emotional situations are labeled as either human- or AI-generated. Human attributed responses were rated as more empathic and supportive, and elicited more positive and fewer negative emotions, compared to AI-attributed ones. Moreover, participants own uninstructed belief that AI aided human-attributed responses, reduced perceived empathy and support. These effects replicated across varying response lengths, delays, iterations and LLMs, being primarily driven by responses emphasizing emotional sharing and care. Additionally, people consistently chose human interaction over AI when seeking emotional engagement. Building on these findings, I introduce a theoretical framework, empathy as a predictive social signal, which proposes that, from the recipient’s perspective, empathy functions not only to regulate emotion but also as a social signal of care and future relational commitment. When empathy is perceived as cheap, automated, or outsourced, its predictive value collapses. I will conclude by describing ongoing work in the lab and discussing the implications of increasingly AI-mediated empathy for relationships and society, including how these developments relate directly to questions of AI safety and alignment. Biography Anat Perry is an associate professor of psychology at the Hebrew University of Jerusalem, where she directs the Social Cognitive Neuroscience Lab. She is a 2026–2027 fellow at Harvard’s Berkman Klein Center for Internet & Society and was the 2025–2026 Helen Putnam Fellow at the Harvard Radcliffe Institute for Advanced Study. Her research examines empathy and social processes through the lenses of psychology and neuroscience, with a growing focus on human–AI interactions and what they reveal about human empathy and social relationships. Perry combines theoretical insight with behavioral experiments, eye-tracking, EEG, and field studies to explore how we understand, feel with, and relate to others.
Resettability of AI Agents
Wednesday, October 14, 2026 · 3–4 p.m. Natalie Collina · MIT Resettability of AI Agents Kiva Seminar Room (32-G449), MIT Stata Center 32 Vassar Street, Cambridge, MA Abstract and biography Abstract How can we reliably test the behavior of AI agents when they may strategically adapt to the tests themselves? One of the few distinctive levers we have is resettability : unlike humans, AI agents can be repeatedly queried while erasing their previous outputs and private randomness. This motivates a broader theoretical question: what can AI agents learn, and what can we test about them, when we control what they remember? We propose a framework for studying AI agents as statistical decision-makers through the lens of resettability, asking how control over an agent’s persistent information changes what it can learn and what an evaluator can reliably test. We use online calibration as a case study. Surprisingly, resetting does not prevent effective learning: we show that resettable forecasters can achieve O ( T 2/3 ) calibration error. On the other hand, we establish an Ω( T 3/5 ) lower bound (higher than the current calibration lower bound), suggesting that memory may provide a fundamental statistical advantage. We also show that combining calibration error with disagreement between independent predictions yields a test that an informed forecaster can pass but no fixed uninformed resettable strategy can uniformly defeat. Our results take a step toward understanding AI agents as statistical decision-makers, and how control over their memory can enable stronger guarantees about their behavior. Biography Natalie Collina is a MAPS Postdoctoral Fellow and SERC Postdoctoral Fellow at MIT studying AI safety and game theory, especially repeated strategic interactions between human and AI agents. She earned her PhD at the University of Pennsylvania, advised by Michael Kearns and Aaron Roth. Her work has been awarded with an IBM PhD Fellowship in Trustworthy AI and a joint Best Paper Award/Best Student Paper Award from the ACM Conference for Economics and Computation.
From (Mis)aligned Models to Aligned Systems
Wednesday, October 7, 2026 · 3–4 p.m. Aaron Roth · University of Pennsylvania From (Mis)aligned Models to Aligned Systems Kiva Seminar Room (32-G449), MIT Stata Center 32 Vassar Street, Cambridge, MA Abstract and biography Abstract The world is being reshaped around AI as we watch. In the near future, and already today, many increasingly important tasks will be delegated to AI agents. The field of AI alignment studies how we can make AI models trustworthy: faithful agents that advance the interests of their users. But alignment might be hard. In this talk we ask what we might be able to guarantee from systems (either naturally emerging from markets, or intentionally designed) that consist of multiple AI models that are each individually mis-aligned. We will see that sometimes we can still guarantee good outcomes by leveraging a diversity of misalignment: models that are misaligned with their users, but also with each other. This talk is based on a number of joint works with Natalie Collina, Surbhi Goel, Meena Jagadeesan, Michael Kearns, Sophia Pi, Emily Ryu, Sikata Sengupta, and Mirah Shi. Biography Aaron Roth is the Henry Salvatori Professor of Computer and Cognitive Science, in the Computer and Information Sciences department at the University of Pennsylvania, with a secondary appointment in the Wharton statistics department. He is affiliated with the Warren Center for Network and Data Science. He is also an Amazon Scholar at Amazon AWS. He is the recipient of the Hans Sigrist Prize, a Presidential Early Career Award for Scientists and Engineers (PECASE), an Alfred P. Sloan Research Fellowship, an NSF CAREER award, and research awards from Yahoo, Amazon, and Google. His research focuses on the algorithmic foundations of data privacy, algorithmic fairness, game theory, learning theory, and machine learning. Together with Cynthia Dwork, he is the author of the book “The Algorithmic Foundations of Differential Privacy.” Together with Michael Kearns, he is the author of “The Ethical Algorithm.”
Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange
Wednesday, September 23, 2026 · 3–4 p.m. Or Zamir · Tel Aviv University Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange Star Seminar Room (32-D463), MIT Stata Center 32 Vassar Street, Cambridge, MA Abstract and biography Abstract AI agents are increasingly deployed to interact with other agents on behalf of users and organizations. We ask whether two such agents, operated by different entities, can carry out a parallel secret conversation while still producing a transcript that is computationally indistinguishable from an honest interaction, even to a strong passive auditor that knows the full model descriptions, the protocol, and the agents’ private contexts. Building on recent work on watermarking and steganography for LLMs, we first show that if the parties possess an interaction-unique secret key, they can facilitate an optimal-rate covert conversation: the hidden conversation can exploit essentially all of the entropy present in the honest message distributions. Our main contributions concern extending this to the keyless setting, where the agents begin with no shared secret. We show that covert key exchange, and hence covert conversation, is possible even when each model has an arbitrary private context, and their messages are short and fully adaptive, assuming only that sufficiently many individual messages have at least constant min-entropy. This stands in contrast to previous covert communication works, which relied on the min-entropy in each individual message growing with the security parameter. To obtain this, we introduce a new cryptographic primitive, which we call pseudorandom noise-resilient key exchange: a key-exchange protocol whose public transcript is pseudorandom while still remaining correct under constant noise. We study this primitive, giving several constructions relevant to our application as well as strong limitations showing that more naive variants are impossible or vulnerable to efficient attacks. These results show that transcript auditing alone cannot rule out covert coordination between AI agents, and identify a new cryptographic theory that may be of independent interest. Based on joint work with Vinod Vaikuntanathan. Biography Or Zamir is an assistant professor at Tel Aviv University’s Blavatnik School of Computer Science. His research spans algorithms, cryptography and the mathematical foundations of AI safety.
Learning Verifiers for Chain-of-Thought Reasoning and Human-AI collaboration
Wednesday, September 16, 2026 · 3:30 p.m. Avrim Blum · Toyota Technological Institute at Chicago Learning Verifiers for Chain-of-Thought Reasoning and Human-AI collaboration Hewlett Room (32-G882), MIT Stata Center 32 Vassar Street, Cambridge, MA Abstract and biography Abstract LLM reasoning is becoming a powerful tool for mathematics, planning, decision support, and agents that act on your behalf in the world. However, LLMs can make mistakes, perform unsafe actions, be overconfident, and in general are not always that reliable. In this work, we consider the problem of interactively training verifiers that could interface with these generators to make them more reliable, safer, and better aligned with their user’s preferences. We consider this problem from a learning-theoretic perspective, analyzing the types of guarantees achievable for learning capable verifiers for these tasks, and highlighting the different roles of “soundness” versus “completeness” forms of errors in our analysis. This work is joint with Maria-Florina Balcan, Korinna Fragkia, Zhiyuan Li, and Dravyansh Sharma. Biography Avrim Blum is Professor, Chief Academic Officer, and Interim President at the Toyota Technological Institute at Chicago (TTIC). Before TTIC, he was a faculty member in the Computer Science Department at Carnegie Mellon University for 25 years. Blum’s main research interests are in Machine Learning Theory, Algorithmic Game Theory, Approximation Algorithms, and Societal Issues in Machine Learning. He has served as Program Chair for the Conference on Learning Theory (COLT), the IEEE Symposium on Foundations of Computer Science (FOCS), and the Innovations in Theoretical Computer Science Conference (ITCS). Blum is recipient of the AI Journal Classic Paper Award, the ICML/COLT 10-Year Best Paper Award, the ACM Paris Kanellakis Award, the Sloan Fellowship, the NSF National Young Investigator Award, and the Herbert Simon Teaching Award, and he is a Fellow of the ACM.
Can the Institute for Responsible Superintelligence save us from rogue AI?
The Institute for Responsible Superintelligence, or RESI for short, seeks to develop mathematical principles and protocols to ensure AI models behave as intended. Boston Globe
Launching RESI, the Institute for Responsible Superintelligence
Adam Tauman Kalai introduces RESI and highlights motivating questions about incentives as capabilities grow, cautious action by interacting AI systems, and preserving the ability to shut systems down.
Announcing Formal Verification at RESI (The Institute for Responsible Superintelligence)
Adam Chlipala introduces an agenda spanning general software infrastructure, AI-specific infrastructure, and systems that improve themselves. The announcement considers both technical problems and how to organize an effective formal-verification effort.