Targeted Evaluation of Language Models with SyntaxGym | ACL

Join the channel membership: https://www.youtube.com/c/AIPursuit/join

Subscribe to the channel: https://www.youtube.com/c/AIPursuit?sub_confirmation=1

Support and Donation: Paypal ⇢ https://paypal.me/tayhengee Patreon ⇢ https://www.patreon.com/hengee BTC ⇢ bc1q2r7eymlf20576alvcmryn28tgrvxqw5r30cmpu ETH ⇢ 0x58c4bD4244686F3b4e636EfeBD159258A5513744 Doge ⇢ DSGNbzuS1s6x81ZSbSHHV5uGDxJXePeyKy

Earn up to $170 welcome bonus on Huobi now with my crypto affiliate link: Binance ⇢ https://accounts.binance.com/en/register?ref=27700065 Binance Futures ⇢ https://www.binance.com/en/futures/ref/89891356 Huobi ⇢ https://www.huobi.com/en-us/topic/welcome-bonus/?invite_code=xj9pc

Coin Market Manager automatically imports and tracks all your crypto investments through the use of read-only API keys so the hard work is done for you!

A 10% discount on all purchases on COIN MARKET MANAGER, use my referral code now: https://app.coinmarketman.com/exchanges/?rf=25611

The video is reposted for educational purposes and encourages involvement in the field of research.

Abstract Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community:

1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization;
2. Two command-line tools, syntaxgym and lm-zoo, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine. ... https://www.youtube.com/watch?v=TzHyc3Qkla4