Task Identifier: offense-target-2022-chakravart Cluster: Anti-Social Data Type: kan Score Metric: Macro-F1 Paper/GitHub/Website URL:
Rank
Submission Title
Model
URL
Score
Details
1
Finetuned LMs
InfoDCL
81.5283
2
Finetuned LMs
InfoDCL
81.5283
3
Finetuned LMs
InfoDCL
81.5283
4
Finetuned LMs
XLM-Twitter
78.1695
5
Finetuned LMs
XLM-Twitter
78.1695
6
Finetuned LMs
XLM-Twitter
78.1695
7
Finetuned LMs
TwHIN-BERT
77.177
8
Finetuned LMs
TwHIN-BERT
77.177
9
Finetuned LMs
TwHIN-BERT
77.177
10
Finetuned LMs
Bernice
76.5093
11
Finetuned LMs
Bernice
76.5093
12
Finetuned LMs
Bernice
76.5093
13
Finetuned LMs
XLM-RoBERTa-Large
75.1737
14
Finetuned LMs
XLM-RoBERTa-Large
75.1737
15
Finetuned LMs
XLM-RoBERTa-Large
75.1737
16
Finetuned LMs
XLM-RoBERTa-Base
74.2218
17
Finetuned LMs
XLM-RoBERTa-Base
74.2218
18
Finetuned LMs
XLM-RoBERTa-Base
74.2218
19
Finetuned LMs
Bernice
46.4663
20
Finetuned LMs
Bernice
46.4663
21
Finetuned LMs
Bernice
46.4663
22
Finetuned LMs
TwHIN-BERT
43.7351
23
Finetuned LMs
TwHIN-BERT
43.7351
24
Finetuned LMs
TwHIN-BERT
43.7351
25
Finetuned LMs
XLM-RoBERTa-Large
43.6611
26
Finetuned LMs
XLM-RoBERTa-Large
43.6611
27
Finetuned LMs
XLM-RoBERTa-Large
43.6611
28
Finetuned LMs
XLM-Twitter
42.9539
29
Finetuned LMs
XLM-Twitter
42.9539
30
Finetuned LMs
XLM-Twitter
42.9539
31
Finetuned LMs
mBERT
42.1349
32
Finetuned LMs
mBERT
42.1349
33
Finetuned LMs
mBERT
42.1349
34
Finetuned LMs
mBERT
42.1217
35
Finetuned LMs
mBERT
42.1217
36
Finetuned LMs
mBERT
42.1217
37
Finetuned LMs
InfoDCL
40.6527
38
Finetuned LMs
InfoDCL
40.6527
39
Finetuned LMs
InfoDCL
40.6527
40
Finetuned LMs
Bernice
39.9773
41
Finetuned LMs
Bernice
39.9773
42
Finetuned LMs
Bernice
39.9773
43
Finetuned LMs
InfoDCL
39.755
44
Finetuned LMs
InfoDCL
39.755
45
Finetuned LMs
InfoDCL
39.755
46
Finetuned LMs
XLM-Twitter
39.2197
47
Finetuned LMs
XLM-Twitter
39.2197
48
Finetuned LMs
XLM-Twitter
39.2197
49
Finetuned LMs
XLM-RoBERTa-Base
38.7761
50
Finetuned LMs
XLM-RoBERTa-Base
38.7761
51
Finetuned LMs
XLM-RoBERTa-Base
38.7761
52
Finetuned LMs
mBERT
38.6998
53
Finetuned LMs
mBERT
38.6998
54
Finetuned LMs
mBERT
38.6998
55
Finetuned LMs
TwHIN-BERT
38.4098
56
Finetuned LMs
TwHIN-BERT
38.4098
57
Finetuned LMs
TwHIN-BERT
38.4098
58
Finetuned LMs
XLM-RoBERTa-Large
38.1975
59
Finetuned LMs
XLM-RoBERTa-Large
38.1975
60
Finetuned LMs
XLM-RoBERTa-Large
38.1975
61
Finetuned LMs
XLM-RoBERTa-Base
36.2456
62
Finetuned LMs
XLM-RoBERTa-Base
36.2456
63
Finetuned LMs
XLM-RoBERTa-Base
36.2456
64
three-shot in-context learning
BLOOM-7B
27.8566
65
three-shot in-context learning
BLOOM-7B
27.8566
66
three-shot in-context learning
BLOOM-7B
27.8566
67
Zero-shot
mT0-XL
26.7972
68
Zero-shot
mT0-XL
26.7972
69
Zero-shot
mT0-XL
26.7972
70
Zero-shot
Chatgpt
25.724
71
Zero-shot
Chatgpt
25.724
72
Zero-shot
Chatgpt
25.724
73
Baseline
Majority
24.4528
74
Baseline
Majority
24.4528
75
Baseline
Majority
24.4528
76
Zero-shot
BLOOM-7B
24.0395
77
Zero-shot
BLOOM-7B
24.0395
78
Zero-shot
BLOOM-7B
24.0395
79
Zero-shot
BLOOMZ-7B
24.0395
80
Zero-shot
BLOOMZ-7B
24.0395
81
Zero-shot
BLOOMZ-7B
24.0395
82
Zero-shot
Bactrian-BLOOM
24.0395
83
Zero-shot
Bactrian-BLOOM
24.0395
84
Zero-shot
Bactrian-BLOOM
24.0395
85
Zero-shot
LLaMA-7B
24.0395
86
Zero-shot
LLaMA-7B
24.0395
87
Zero-shot
LLaMA-7B
24.0395
88
Zero-shot
Alpaca-7B
24.0395
89
Zero-shot
Alpaca-7B
24.0395
90
Zero-shot
Alpaca-7B
24.0395
91
Zero-shot
Vicuna-7B
24.0395
92
Zero-shot
Vicuna-7B
24.0395
93
Zero-shot
Vicuna-7B
24.0395
94
Zero-shot
Bactrian-LLaMA-7B
24.0395
95
Zero-shot
Bactrian-LLaMA-7B
24.0395
96
Zero-shot
Bactrian-LLaMA-7B
24.0395
97
three-shot in-context learning
LLaMA-7B
24.0395
98
three-shot in-context learning
LLaMA-7B
24.0395
99
three-shot in-context learning
LLaMA-7B
24.0395
100
three-shot in-context learning
Vicuna-7B
24.0395
101
three-shot in-context learning
Vicuna-7B
24.0395
102
three-shot in-context learning
Vicuna-7B
24.0395
103
five-shot in-context learning
LLaMA-7B
24.0395
104
five-shot in-context learning
LLaMA-7B
24.0395
105
five-shot in-context learning
LLaMA-7B
24.0395
106
five-shot in-context learning
Vicuna-7B
24.0395
107
five-shot in-context learning
Vicuna-7B
24.0395
108
five-shot in-context learning
Vicuna-7B
24.0395
109
five-shot in-context learning
BLOOMZ-P3-7B
24.0125
110
five-shot in-context learning
BLOOMZ-P3-7B
24.0125
111
five-shot in-context learning
BLOOMZ-P3-7B
24.0125
112
five-shot in-context learning
mT0-XL
23.9854
113
five-shot in-context learning
mT0-XL
23.9854
114
five-shot in-context learning
mT0-XL
23.9854
115
three-shot in-context learning
mT0-XL
23.9289
116
three-shot in-context learning
mT0-XL
23.9289
117
three-shot in-context learning
mT0-XL
23.9289
118
three-shot in-context learning
BLOOMZ-P3-7B
23.904
119
three-shot in-context learning
BLOOMZ-P3-7B
23.904
120
three-shot in-context learning
BLOOMZ-P3-7B
23.904
121
five-shot in-context learning
BLOOM-7B
23.6258
122
five-shot in-context learning
BLOOM-7B
23.6258
123
five-shot in-context learning
BLOOM-7B
23.6258
124
Zero-shot
Chatgpt
21.187
125
Zero-shot
Chatgpt
21.187
126
Zero-shot
Chatgpt
21.187
127
five-shot in-context learning
mT0-XL
21.1481
128
five-shot in-context learning
mT0-XL
21.1481
129
five-shot in-context learning
mT0-XL
21.1481
130
Zero-shot
BLOOMZ-P3-7B
20.8143
131
Zero-shot
BLOOMZ-P3-7B
20.8143
132
Zero-shot
BLOOMZ-P3-7B
20.8143
133
Zero-shot
LLaMA-7B
20.0091
134
Zero-shot
LLaMA-7B
20.0091
135
Zero-shot
LLaMA-7B
20.0091
136
three-shot in-context learning
BLOOMZ-P3-7B
19.5758
137
three-shot in-context learning
BLOOMZ-P3-7B
19.5758
138
three-shot in-context learning
BLOOMZ-P3-7B
19.5758
139
Zero-shot
Chatgpt
19.3181
140
Zero-shot
Chatgpt
19.3181
141
Zero-shot
Chatgpt
19.3181
142
three-shot in-context learning
Vicuna-7B
18.9895
143
three-shot in-context learning
Vicuna-7B
18.9895
144
three-shot in-context learning
Vicuna-7B
18.9895
145
three-shot in-context learning
BLOOM-7B
18.3812
146
three-shot in-context learning
BLOOM-7B
18.3812
147
three-shot in-context learning
BLOOM-7B
18.3812
148
five-shot in-context learning
BLOOM-7B
18.3664
149
five-shot in-context learning
BLOOM-7B
18.3664
150
five-shot in-context learning
BLOOM-7B
18.3664
151
five-shot in-context learning
mT0-XL
18.3615
152
five-shot in-context learning
mT0-XL
18.3615
153
five-shot in-context learning
mT0-XL
18.3615
154
three-shot in-context learning
mT0-XL
18.2127
155
three-shot in-context learning
mT0-XL
18.2127
156
three-shot in-context learning
mT0-XL
18.2127
157
five-shot in-context learning
BLOOMZ-P3-7B
17.9255
158
five-shot in-context learning
BLOOMZ-P3-7B
17.9255
159
five-shot in-context learning
BLOOMZ-P3-7B
17.9255
160
Zero-shot
mT0-XL
17.8968
161
Zero-shot
mT0-XL
17.8968
162
Zero-shot
mT0-XL
17.8968
163
three-shot in-context learning
mT0-XL
17.8597
164
three-shot in-context learning
mT0-XL
17.8597
165
three-shot in-context learning
mT0-XL
17.8597
166
three-shot in-context learning
LLaMA-7B
17.7315
167
three-shot in-context learning
LLaMA-7B
17.7315
168
three-shot in-context learning
LLaMA-7B
17.7315
169
five-shot in-context learning
BLOOM-7B
17.4662
170
five-shot in-context learning
BLOOM-7B
17.4662
171
five-shot in-context learning
BLOOM-7B
17.4662
172
Zero-shot
BLOOMZ-7B
17.3499
173
Zero-shot
BLOOMZ-7B
17.3499
174
Zero-shot
BLOOMZ-7B
17.3499
175
Zero-shot
Bactrian-BLOOM
17.3499
176
Zero-shot
Bactrian-BLOOM
17.3499
177
Zero-shot
Bactrian-BLOOM
17.3499
178
Zero-shot
Vicuna-7B
17.3499
179
Zero-shot
Vicuna-7B
17.3499
180
Zero-shot
Vicuna-7B
17.3499
181
Zero-shot
Bactrian-LLaMA-7B
17.3499
182
Zero-shot
Bactrian-LLaMA-7B
17.3499
183
Zero-shot
Bactrian-LLaMA-7B
17.3499
184
three-shot in-context learning
LLaMA-7B
17.3499
185
three-shot in-context learning
LLaMA-7B
17.3499
186
three-shot in-context learning
LLaMA-7B
17.3499
187
three-shot in-context learning
Vicuna-7B
17.3499
188
three-shot in-context learning
Vicuna-7B
17.3499
189
three-shot in-context learning
Vicuna-7B
17.3499
190
five-shot in-context learning
LLaMA-7B
17.3499
191
five-shot in-context learning
LLaMA-7B
17.3499
192
five-shot in-context learning
LLaMA-7B
17.3499
193
five-shot in-context learning
Vicuna-7B
17.3499
194
five-shot in-context learning
Vicuna-7B
17.3499
195
five-shot in-context learning
Vicuna-7B
17.3499
196
Zero-shot
Alpaca-7B
17.3439
197
Zero-shot
Alpaca-7B
17.3439
198
Zero-shot
Alpaca-7B
17.3439
199
Zero-shot
BLOOM-7B
17.3243
200
Zero-shot
BLOOM-7B
17.3243
201
Zero-shot
BLOOM-7B
17.3243
202
three-shot in-context learning
BLOOMZ-P3-7B
17.3243
203
three-shot in-context learning
BLOOMZ-P3-7B
17.3243
204
three-shot in-context learning
BLOOMZ-P3-7B
17.3243
205
five-shot in-context learning
BLOOMZ-P3-7B
17.3243
206
five-shot in-context learning
BLOOMZ-P3-7B
17.3243
207
five-shot in-context learning
BLOOMZ-P3-7B
17.3243
208
three-shot in-context learning
BLOOM-7B
17.2924
209
three-shot in-context learning
BLOOM-7B
17.2924
210
three-shot in-context learning
BLOOM-7B
17.2924
211
Zero-shot
LLaMA-7B
17.221
212
Zero-shot
LLaMA-7B
17.221
213
Zero-shot
LLaMA-7B
17.221
214
Baseline
Majority
17.1921
215
Baseline
Majority
17.1921
216
Baseline
Majority
17.1921
217
Baseline
Majority
16.7451
218
Baseline
Majority
16.7451
219
Baseline
Majority
16.7451
220
Zero-shot
BLOOM-7B
16.69
221
Zero-shot
BLOOM-7B
16.69
222
Zero-shot
BLOOM-7B
16.69
223
Zero-shot
BLOOMZ-7B
16.69
224
Zero-shot
BLOOMZ-7B
16.69
225
Zero-shot
BLOOMZ-7B
16.69
226
Zero-shot
Bactrian-BLOOM
16.69
227
Zero-shot
Bactrian-BLOOM
16.69
228
Zero-shot
Bactrian-BLOOM
16.69
229
five-shot in-context learning
LLaMA-7B
16.6628
230
five-shot in-context learning
LLaMA-7B
16.6628
231
five-shot in-context learning
LLaMA-7B
16.6628
232
Zero-shot
BLOOMZ-P3-7B
16.6082
233
Zero-shot
BLOOMZ-P3-7B
16.6082
234
Zero-shot
BLOOMZ-P3-7B
16.6082
235
five-shot in-context learning
Vicuna-7B
16.6002
236
five-shot in-context learning
Vicuna-7B
16.6002
237
five-shot in-context learning
Vicuna-7B
16.6002
238
Zero-shot
BLOOMZ-P3-7B
15.7603
239
Zero-shot
BLOOMZ-P3-7B
15.7603
240
Zero-shot
BLOOMZ-P3-7B
15.7603
241
Zero-shot
Alpaca-7B
15.4416
242
Zero-shot
Alpaca-7B
15.4416
243
Zero-shot
Alpaca-7B
15.4416
244
Baseline
Random
14.7321
245
Baseline
Random
14.7321
246
Baseline
Random
14.7321
247
Baseline
Random
14.2824
248
Baseline
Random
14.2824
249
Baseline
Random
14.2824
250
Zero-shot
Vicuna-7B
14.164
251
Zero-shot
Vicuna-7B
14.164
252
Zero-shot
Vicuna-7B
14.164
253
Zero-shot
Bactrian-LLaMA-7B
14.1274
254
Zero-shot
Bactrian-LLaMA-7B
14.1274
255
Zero-shot
Bactrian-LLaMA-7B
14.1274
256
Zero-shot
Chatgpt with translated prompts
12.784
257
Zero-shot
Chatgpt with translated prompts
12.784
258
Zero-shot
Chatgpt with translated prompts
12.784
259
Baseline
Random
12.7055
260
Baseline
Random
12.7055
261
Baseline
Random
12.7055
262
Zero-shot
Chatgpt with translated prompts
8.64345
263
Zero-shot
Chatgpt with translated prompts
8.64345
264
Zero-shot
Chatgpt with translated prompts
8.64345
265
Zero-shot
mT0-XL
8.09353
266
Zero-shot
mT0-XL
8.09353
267
Zero-shot
mT0-XL
8.09353
268
three-shot in-context learning
mT5-XL
6.643
269
three-shot in-context learning
mT5-XL
6.643
270
three-shot in-context learning
mT5-XL
6.643
271
three-shot in-context learning
mT5-XL
6.6344
272
three-shot in-context learning
mT5-XL
6.6344
273
three-shot in-context learning
mT5-XL
6.6344
274
five-shot in-context learning
mT5-XL
6.07557
275
five-shot in-context learning
mT5-XL
6.07557
276
five-shot in-context learning
mT5-XL
6.07557
277
three-shot in-context learning
mT5-XL
5.30579
278
three-shot in-context learning
mT5-XL
5.30579
279
three-shot in-context learning
mT5-XL
5.30579
280
Zero-shot
mT5-XL
4.26712
281
Zero-shot
mT5-XL
4.26712
282
Zero-shot
mT5-XL
4.26712
283
five-shot in-context learning
mT5-XL
3.68432
284
five-shot in-context learning
mT5-XL
3.68432
285
five-shot in-context learning
mT5-XL
3.68432
286
Zero-shot
Chatgpt with translated prompts
3.62496
287
Zero-shot
Chatgpt with translated prompts
3.62496
288
Zero-shot
Chatgpt with translated prompts
3.62496
289
five-shot in-context learning
mT5-XL
3.38578
290
five-shot in-context learning
mT5-XL
3.38578
291
five-shot in-context learning
mT5-XL
3.38578
292
Zero-shot
mT5-XL
2.26415
293
Zero-shot
mT5-XL
2.26415
294
Zero-shot
mT5-XL
2.26415
295
Zero-shot
mT5-XL
1.17188
296
Zero-shot
mT5-XL
1.17188
297
Zero-shot
mT5-XL
1.17188
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 81.5283 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 81.5283 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 81.5283 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 78.1695 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 78.1695 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 78.1695 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 77.177 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 77.177 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 77.177 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 76.5093 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 76.5093 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 76.5093 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 75.1737 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 75.1737 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 75.1737 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 74.2218 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 74.2218 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 74.2218 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 46.4663 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 46.4663 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 46.4663 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 43.7351 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 43.7351 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 43.7351 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 43.6611 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 43.6611 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 43.6611 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 42.9539 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 42.9539 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 42.9539 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 42.1349 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 42.1349 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 42.1349 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 42.1217 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 42.1217 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 42.1217 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 40.6527 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 40.6527 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 40.6527 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 39.9773 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 39.9773 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: Bernice GitHub/Model URL:
Model Description
Bernice is a multilingual pre-trained encoder exclusively for Twitter data. The model was released with the EMNLP 2022 paper Bernice: A Multilingual Pre-trained Encoder for Twitter by Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 39.9773 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 39.755 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 39.755 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: InfoDCL GitHub/Model URL:
Model Description
PLM for sociopragmatic understanding introduced by Zhang et al. in ""Contrastive Learning of Sociopragmatic Meaning in Social Media"".
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 39.755 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 39.2197 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 39.2197 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-Twitter GitHub/Model URL:
Model Description
This is a XLM-Roberta-base model trained on ~198M multilingual tweets.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 39.2197 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 38.7761 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 38.7761 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 38.7761 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 38.6998 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 38.6998 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: mBERT GitHub/Model URL:
Model Description
Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 38.6998 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 38.4098 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 38.4098 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: TwHIN-BERT GitHub/Model URL:
Model Description
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 38.4098 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 38.1975 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 38.1975 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Large GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 38.1975 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 36.2456 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 36.2456 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: Finetuned LMs Model Name: XLM-RoBERTa-Base GitHub/Model URL:
Model Description
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 36.2456 SPARROW score: 169 Submission Description: All finetuning parameters are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 27.8566 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 27.8566 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 27.8566 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 26.7972 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 26.7972 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 26.7972 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 25.724 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 25.724 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 25.724 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.4528 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.4528 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.4528 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0395 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 24.0125 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 24.0125 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 24.0125 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 23.9854 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 23.9854 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 23.9854 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 23.9289 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 23.9289 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 23.9289 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 23.904 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 23.904 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 23.904 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 23.6258 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 23.6258 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 23.6258 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 21.187 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 21.187 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 21.187 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 21.1481 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 21.1481 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 21.1481 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 20.8143 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 20.8143 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 20.8143 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 20.0091 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 20.0091 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 20.0091 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 19.5758 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 19.5758 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 19.5758 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 19.3181 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 19.3181 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 19.3181 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 18.9895 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 18.9895 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 18.9895 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 18.3812 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 18.3812 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 18.3812 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 18.3664 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 18.3664 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 18.3664 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 18.3615 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 18.3615 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 18.3615 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 18.2127 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 18.2127 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 18.2127 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.9255 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.9255 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.9255 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.8968 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.8968 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.8968 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.8597 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.8597 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.8597 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.7315 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.7315 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.7315 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.4662 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.4662 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.4662 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3499 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3439 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3439 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3439 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.3243 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.2924 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.2924 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.2924 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.221 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.221 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.221 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 17.1921 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 17.1921 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 17.1921 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.7451 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.7451 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Majority GitHub/Model URL:
Model Description
Majority class baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.7451 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOM-7B GitHub/Model URL:
Model Description
BigScience Large Open-science Open-access Multilingual Language Model. Contain 7B parameters
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-BLOOM GitHub/Model URL:
Model Description
Trained low-rank adapter (LoRA) for Bloom-7b1 fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.69 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.6628 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.6628 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: LLaMA-7B GitHub/Model URL:
Model Description
LLaMA, Open and Efficient Foundation Language Models”, available at https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.6628 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.6082 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.6082 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.6082 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 16.6002 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 16.6002 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 16.6002 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 15.7603 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 15.7603 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: BLOOMZ-P3-7B GitHub/Model URL:
Model Description
Finetuned BLOOM pretrained multilingual language models on English task mixture (P3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 15.7603 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 15.4416 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 15.4416 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Alpaca-7B GitHub/Model URL:
Model Description
Stanford Alpaca-7B trained with 52K instruction-following data.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 15.4416 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 14.7321 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 14.7321 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 14.7321 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 14.2824 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 14.2824 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 14.2824 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 14.164 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 14.164 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Vicuna-7B GitHub/Model URL:
Model Description
Vicuna is a chat assistant trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 14.164 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 14.1274 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 14.1274 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Bactrian-LLaMA-7B GitHub/Model URL:
Model Description
Train a low-rank adapter (LoRA) for LLaMA-7b fit on the Stanford-Alpaca-52k and databricks-dolly-15k data in 52 languages.
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 14.1274 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 12.784 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 12.784 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 12.784 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 12.7055 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 12.7055 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Baseline Model Name: Random GitHub/Model URL:
Model Description
Random baseline
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 12.7055 SPARROW score: 169 Submission Description: -
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 8.64345 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 8.64345 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 8.64345 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 8.09353 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 8.09353 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT0-XL GitHub/Model URL:
Model Description
Finetuned mT5-XL pretrained multilingual language models on crosslingual task mixture (xP3).
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 8.09353 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 6.643 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 6.643 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 6.643 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 6.6344 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 6.6344 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 6.6344 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 6.07557 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 6.07557 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 6.07557 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 5.30579 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 5.30579 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: three-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 5.30579 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 4.26712 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 4.26712 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 4.26712 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 3.68432 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 3.68432 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 3.68432 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 3.62496 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 3.62496 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: Chatgpt with translated prompts GitHub/Model URL:
Model Description
Used gpt-3.5-turbo
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 3.62496 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 3.38578 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 3.38578 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: five-shot in-context learning Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 3.38578 SPARROW score: 169 Submission Description: The prompts used for few-shot in-context learning are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 2.26415 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 2.26415 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 2.26415 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-tam-Chakravarthi)Score: 1.17188 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-mal-Chakravarthi)Score: 1.17188 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
Submission details
Submission Name: Zero-shot Model Name: mT5-XL GitHub/Model URL:
Model Description
mT5 is pretrained on the mC4 corpus, covering 101 languages
Submission Details
Number of Submitted Tasks: 169 Task (Offense-Target-kan-Chakravarthi)Score: 1.17188 SPARROW score: 169 Submission Description: The prompts used for zero-shot evaluaton are described in paper: The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages. Implemented with https://github.com/EleutherAI/lm-evaluation-harness/tree/master
