<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-PR" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Aptos",sans-serif">See below!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Aptos",sans-serif"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">From:
</span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Psych.others <psych.others-bounces@mailman.yale.edu> on behalf of Brooke-Wilson, Tyler <tyler.brooke.wilson@yale.edu><br>
<b>Date: </b>Saturday, March 29, 2025 at 7:59</span><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span lang="EN-US" style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">AM<br>
<b>To: </b>psych.all@mailman.yale.edu <psych.all@mailman.yale.edu><br>
<b>Subject: </b>Paul Smolensky CompCog Talk This Tuesday 10AM, RM 1167 <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Forwarding a talk on Language Model’s In-Context learning that may be of interest –
</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Hi All, </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Computational linguist
<a href="https://scholar.google.com/citations?user=PRtkZzYAAAAJ&hl=en" target="_blank">
<span style="color:#1155CC">Paul Smolensky</span></a> will be presenting current work on In-Context Learning in Transformers this Tuesday, 10-11:30AM. Title and abstract below, plus link to sign up for 1-1 meetings with Paul. </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Time: </span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Tuesday 4/1,
10-11:30AM </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Location:</span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> 100 College
St., RM 1167</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Title: </span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Mechanisms of
Symbol Processing for In-Context Learning in Transformer Networks</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Authors:</span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> Paul Smolensky,
Roland Fernandez, Herbert Zhou, Mattia Opper, Jianfeng Gao</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Abstract: </span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Large Language
Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of predictions that artificial neural networks cannot master abstract symbol manipulation. We seek to understand
the mechanisms that can enable robust symbol processing in transformer networks, illuminating both the unanticipated success, and the significant limitations, of transformers in symbol processing. Borrowing insights from symbolic AI on the power of Production
System architectures, we develop a high-level language, PSL, that allows us to write symbolic programs to do complex, abstract symbol processing, and create compilers that precisely implement PSL programs in transformer networks which are, by construction,
100% mechanistically interpretable. We demonstrate that PSL is Turing Universal, so the work can inform the understanding of transformer ICL in general. The type of transformer architecture that we compile from PSL programs suggests a number of paths for enhancing
transformers’ capabilities at symbol processing.</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Suggested paper to pre-read:</span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> Section
1 of <a href="https://arxiv.org/abs/2410.17498" target="_blank"><span style="color:#54ACDC">https://arxiv.org/abs/2410.17498</span></a></span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Note:</span></b><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> There is a sign
up sheet to meet with Paul <a href="https://docs.google.com/spreadsheets/d/1cUpdHD5bQNfuPlNKqmFP7Om15hR3r3Ke2hKja1wO8Ug/edit?usp=sharing" target="_blank"><span style="color:#1155CC">here</span></a>. Meetings will be at 100 College St. (Check the sign up sheet
for exact location day of). Please feel free to send this to any students with relevant interests. </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"> </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Best, </span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Tyler
</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>