What Can('t) Transformers Do?

Overview

With most advances in large foundation models (LFMs) being empirical, our theoretical understanding of what transformers can compute, express, and learn still lags behind. This workshop will convene theorists and empiricists to chart a rigorous agenda for the next generation of LFMs, asking What can and can't transformer-based LLMs do?.

We welcome both formal analyses and empirically grounded studies that shed light on theoretical questions, aiming to close the gap between proofs and practice while fostering new, interdisciplinary collaborations.

Questions? You can reach us via transformers-this.workshop-2025is.@googlegroups.comdecoy..

Call for Papers

We invite contributions that either introduce new formal results—proofs, impossibility theorems, or constructive separations—and show why they matter in practice, or design rigorous, carefully controlled experiments whose primary aim is to test, falsify, or refine a theoretical claim about transformers and language models.

• Core themes:
  • Theoretical analyses of transformer capabilities, including expressivity, learnability, inference-time scaling, in-context learning, and effects of architectural components.
  • Empirical studies of transformer behavior that inform theoretical understanding, including architectural or training innovations, mechanistic studies of failures, and comparisons of theorized and observed capabilities.
• Presentation format:
  • All accepted papers will be presented as posters; a subset will be invited for spotlight talks based on reviewer recommendations.
  • Submissions are non-archival. However, all authors will have the option to have their accepted abstracts accessible from the workshop website.
• Submission format:
  • Extended abstract (2 content pages), with unlimited references and supplementary material.
  • Anonymized, in NeurIPS format using the official style files.
  • Completing the NeurIPS checklist is optional.
• Submission:
  • Link: https://openreview.net/group?id=NeurIPS.cc/2025/Workshop/WCTD.
  • Important: Each submission must select at least one author to act as a reviewer for three other submissions.

Deadlines

• Submission deadline: August 22, 2025, AoE August 29, 2025, AoE
• Reviews due: September 18, 2025, AoE
• Accept/Reject Notification Date: September 22, 2025, AoE
• Camera-ready deadline: November 22, 2025, AoE
• Workshop Date: December 6 or December 7, 2025

Keynote Speakers

Surbhi Goel

University Of Pennsylvania

Talk: TBD

Jon Kleinberg

Cornell University

Talk: TBD

William Merrill

NYU

Talk: TBD

Organizers

Tobias Schnabel

Microsoft Research

Kiran Tomlinson

Microsoft Research

Lena Strobl

Umeå University

Michael Hahn

Saarland University

Tentative Timetable