SPARROW is a evaluation benchmark for sociopragmatic meaning understanding. SPARROW comprises 169 datasets covering 13 task types across six primary categories (e.g., anti-social language detection, emotion recognition). SPARROW datasets encompass 64 different languages originating from 12 language families representing 16 writing scripts.


(1) Anti-Social

Task Name Identifier
Language
Score MetricURL
BibTeX
Aggressive-hin-Kumaraggressive-2018-kumar-hin
hin
Macro-F1
Dangerous-ara-Alshehridangerous-2020-alshehri-ara
ara
Macro-F1
Hate-ara-Alakrothate-2018-alakrot-ara
ara
Macro-F1
Hate-ara-Mubarakhate-2020-mubarak-ara
ara
Macro-F1
Hate-ara-Mulkihate-2019-mulki-ara
ara
Macro-F1
Hate-eng-Basilehate-2019-basile-eng
eng
Macro-F1
Hate-eng-Davidsonhate-2017-davidson-eng
eng
Weight-F1
Hate-eng-Waseemhate-2016-waseem-eng
eng
Weight-F1
Hate-fil-Cabasaghate-2019-cabasag-fil
fil
Macro-F1
Hate-Group-ara-Ousidhoumhate-group-2019-ousidhoum-ara
ara
Macro-F1
Hate-Group-fre-Ousidhoumhate-group-2019-ousidhoum-fre
fre
Macro-F1
Hate-ita-Boscohate-2018-bosco-ita
ita
Macro-F1
Hate-kor-Jeonghate-2022-jeong-kor
kor
Macro-F1
Hate-kor-Moonhate-2020-moon-kor
kor
Macro-F1
Hate-pol-Ptaszynskihate-2019-ptaszynski-pol
pol
Macro-F1
Hate-por-Fortunahate-2019-fortuna-por
por
Macro-F1
Hate-spa-Basilehate-2019-basile-spa
spa
Macro-F1
Hate-Target-ara-Ousidhoumhate-target-2019-ousidhoum-ara
ara
Macro-F1
Hate-Target-ben-Karimhate-target-2020-karim-ben
ben
Macro-F1
Hate-Target-fre-Ousidhoumhate-target-2019-ousidhoum-fre
fre
Macro-F1
Hate-Target-kor-Jeonghate-target-2022-jeong-kor
kor
Macro-F1
Hate-tel-Marreddyhate-2022-marreddy-tel
tel
Macro-F1
Hate-zho-Denghate-2022-deng-zho
zho
Macro-F1
Offense-Target-kan-Chakravarthioffense-target-2022-chakravart
kan
Macro-F1
Offense-Target-mal-Chakravarthioffense-target-2022-chakravart
mal
Macro-F1
Offense-Target-tam-Chakravarthioffense-target-2022-chakravart
tam
Macro-F1
Offensive-ara-Mubarakoffensive-2020-mubarak-ara
ara
Macro-F1
Offensive-ara-Zampierioffensive-2020-zampieri-ara
ara
Macro-F1
Offensive-dan-Zampierioffensive-2020-zampieri-dan
dan
Macro-F1
Offensive-ell-Zampierioffensive-2020-zampieri-ell
ell
Macro-F1
Offensive-eng-Zampierioffensive-2019-zampieri-eng
eng
Macro-F1
Offensive-Group-eng-Zampierioffensive-group-2019-zampieri-
eng
Macro-F1
Offensive-slv-Novakoffensive-2021-novak-slv
slv
Macro-F1
Offensive-Target-eng-Zampierioffensive-target-2019-zampieri
eng
Macro-F1
Offensive-tur-Zampierioffensive-2020-zampieri-tur
tur
Macro-F1
Sexism-fre-Chirilsexism-2020-chiril-fre
fre
Macro-F1


(2) Emotion Recognition

Task Name Identifier
Language
Score MetricURL
BibTeX
Emotion-ara-Abdulemotion-2020-abdul-ara
ara
Macro-F1
Emotion-ara-Mohammademotion-2018-mohammad-ara
ara
Macro-F1
Emotion-ben-Iqbalemotion-2022-iqbal-ben
ben
Macro-F1
Emotion-deu-Bianchiemotion-2022-bianchi-deu
deu
Macro-F1
Emotion-eng-Demszkyemotion-2020-demszky-eng
eng
Macro-F1
Emotion-eng-Mohammademotion-2018-mohammad-eng
eng
Macro-F1
Emotion-eng-Plazaemotion-2020-plaza-eng
eng
Macro-F1
Emotion-eng-Wallbottemotion-1986-wallbott-eng
eng
Macro-F1
Emotion-fas-Sabriemotion-2021-sabri-fas
fas
Macro-F1
Emotion-fin-Kajavaemotion-2018-kajava-fin
fin
Macro-F1
Emotion-fin-Ohmanemotion-2020-ohman-fin
fin
Macro-F1
Emotion-fre-Bianchiemotion-2022-bianchi-fre
fre
Macro-F1
Emotion-fre-Kajavaemotion-2018-kajava-fre
fre
Macro-F1
Emotion-hin-Debadityaemotion-2021-debaditya-hin
hin
Macro-F1
Emotion-ind-Saputriemotion-2019-saputri-ind
ind
Macro-F1
Emotion-ind-Wilieemotion-2020-wilie-ind
ind
Macro-F1
Emotion-ita-Bianchiemotion-2021-bianchi-ita
ita
Macro-F1
Emotion-ita-Kajavaemotion-2018-kajava-ita
ita
Macro-F1
Emotion-por-Cortizemotion-2021-cortiz-por
por
Macro-F1
Emotion-ron-Ciobotaruemotion-2021-ciobotaru-ron
ron
Macro-F1
Emotion-rus-Sboevemotion-2021-sboev-rus
rus
Macro-F1
Emotion-spa-Mohammademotion-2018-mohammad-spa
spa
Macro-F1
Emotion-spa-Plazaemotion-2020-plaza-spa
spa
Macro-F1
Emotion-tur-Guvenemotion-2020-guven-tur
tur
Accuracy
Emotion-vie-Hoemotion-2019-ho-vie
vie
Weight-F1
Emotion-zho-Leeemotion-2015-lee-zho
zho
Accuracy


(3) Humor Detection

Task Name Identifier
Language
Score MetricURL
BibTeX
Humor-eng-Meaneyhumor-2021-meaney-eng
eng
Macro-F1
Humor-hin-Aggarwalhumor-2020-aggarwal-hin
hin
Accuracy
Humor-rus-Blinovhumor-2019-blinov-rus
rus
Macro-F1
Humor-spa-Chiruzzohumor-2021-chiruzzo-spa
spa
Macro-F1


(4) Irony & Sarcasm

Task Name Identifier
Language
Score MetricURL
BibTeX
Irony-ara-Ghanemirony-2019-ghanem-ara
ara
Macro-F1
Irony-eng-Heeirony-2018-hee-eng
eng
F1-irony
Irony-fas-Golazizianirony-2020-golazizian-fas
fas
Accuracy
Irony-hin-Vijayirony-2018-vijay-hin
hin
Macro-F1
Irony-ita-Basileirony-2014-basile-ita
ita
Macro-F1
Irony-ita-Cignarellairony-2018-cignarella-ita
ita
Macro-F1
Irony-spa-Barbieriirony-2016-barbieri-spa
spa
Macro-F1
Irony-spa-Ortegairony-2019-ortega-spa
spa
Macro-F1
Irony-Type-eng-Heeirony-type-2018-hee-eng
eng
Macro-F1
Irony-zho-Xiangirony-2020-xiang-zho
zho
Macro-F1
Sarcasm-ara-Abufarhasarcasm-2020-abufarha-ara
ara
Macro-F1
Sarcasm-ara-Farhasarcasm-2021-farha-ara
ara
Macro-F1
Sarcasm-ces-Ptaceksarcasm-2014-ptacek-ces
ces
Macro-F1
Sarcasm-eng-Bammansarcasm-2015-bamman-eng
eng
Accuracy
Sarcasm-eng-Orabysarcasm-2016-oraby-eng
eng
Macro-F1
Sarcasm-eng-Ptaceksarcasm-2014-ptacek-eng
eng
Macro-F1
Sarcasm-eng-Rajadesingansarcasm-2015-rajadesingan-eng
eng
Accuracy
Sarcasm-eng-Riloffsarcasm-2013-riloff-eng
eng
F1-sarcasm
Sarcasm-eng-Walkersarcasm-2012-walker-eng
eng
Macro-F1
Sarcasm-zho-Gongsarcasm-2020-gong-zho
zho
Macro-F1


(5) Sentiment Analysis

Task Name Identifier
Language
Score MetricURL
BibTeX
Sentiment-5-eng-Sochersentiment-5-2013-socher-eng
eng
Accuracy
Sentiment-ace-Winatasentiment-2022-winata-ace
ace
Macro-F1
Sentiment-amh-Muhammadsentiment-2022-muhammad-amh
amh
Weight-F1
Sentiment-ara-Abdulsentiment-2021-abdul-ara
ara
Macro-F1
Sentiment-ara-Muhammadsentiment-2022-muhammad-ara
ara
Accuracy
Sentiment-arq-Muhammadsentiment-2022-muhammad-arq
arq
Weight-F1
Sentiment-ary-Muhammadsentiment-2022-muhammad-ary
ary
Weight-F1
Sentiment-bam-Diallosentiment-2021-diallo-bam
bam
Macro-F1
Sentiment-ban-Winatasentiment-2022-winata-ban
ban
Macro-F1
Sentiment-bbc-Winatasentiment-2022-winata-bbc
bbc
Macro-F1
Sentiment-ben-Islamsentiment-2021-islam-ben
ben
Macro-F1
Sentiment-ben-Patrasentiment-2018-patra-ben
ben
Macro-F1
Sentiment-bjn-Winatasentiment-2022-winata-bjn
bjn
Macro-F1
Sentiment-bos-Mozeticsentiment-2016-mozetic-bos
bos
Macro-F1
Sentiment-bug-Winatasentiment-2022-winata-bug
bug
Macro-F1
Sentiment-bul-Mozeticsentiment-2016-mozetic-bul
bul
Macro-F1
Sentiment-deu-Mozeticsentiment-2016-mozetic-deu
deu
Macro-F1
Sentiment-deu-Reisentiment-2016-rei-deu
deu
Macro-F1
Sentiment-eng-Mozeticsentiment-2016-mozetic-eng
eng
Macro-F1
Sentiment-eng-Pangsentiment-2005-pang-eng
eng
Accuracy
Sentiment-eng-Rosenthalsentiment-2017-rosenthal-eng
eng
Macro-Recall
Sentiment-eng-Sochersentiment-2013-socher-eng
eng
Accuracy
Sentiment-fas-Ashrafisentiment-2020-ashrafi-fas
fas
Macro-F1
Sentiment-fin-Kajavasentiment-2018-kajava-fin
fin
Macro-F1
Sentiment-fre-Kajavasentiment-2018-kajava-fre
fre
Macro-F1
Sentiment-hau-Muhammadsentiment-2022-muhammad-hau
hau
Weight-F1
Sentiment-heb-Amramsentiment-2018-amram-heb
heb
Accuracy
Sentiment-hin-Patrasentiment-2018-patra-hin
hin
Macro-F1
Sentiment-hrv-Mozeticsentiment-2016-mozetic-hrv
hrv
Macro-F1
Sentiment-hun-Mozeticsentiment-2016-mozetic-hun
hun
Macro-F1
Sentiment-ibo-Muhammadsentiment-2022-muhammad-ibo
ibo
Weight-F1
Sentiment-ind-Wiliesentiment-2020-wilie-ind
ind
Macro-F1
Sentiment-ita-Basilesentiment-2014-basile-ita
ita
Macro-F1
Sentiment-ita-Basilesentiment-2016-basile-ita
ita
Macro-F1
Sentiment-ita-Kajavasentiment-2018-kajava-ita
ita
Macro-F1
Sentiment-ita-Reisentiment-2016-rei-ita
ita
Macro-F1
Sentiment-jav-Winatasentiment-2022-winata-jav
jav
Macro-F1
Sentiment-jpn-Suzukisentiment-2022-suzuki-jpn
jpn
Accuracy
Sentiment-kan-Chakravarthisentiment-2022-chakravarthi-ka
kan
Macro-F1
Sentiment-kin-Muhammadsentiment-2022-muhammad-kin
kin
Weight-F1
Sentiment-kor-Jangsentiment-2013-jang-kor
kor
Macro-F1
Sentiment-mad-Winatasentiment-2022-winata-mad
mad
Macro-F1
Sentiment-mal-Chakravarthisentiment-2022-chakravarthi-ma
mal
Macro-F1
Sentiment-mar-Kulkarnisentiment-2021-kulkarni-mar
mar
Accuracy
Sentiment-min-Winatasentiment-2022-winata-min
min
Macro-F1
Sentiment-mlt-Dinglisentiment-2016-dingli-mlt
mlt
Macro-F1
Sentiment-nij-Winatasentiment-2022-winata-nij
nij
Macro-F1
Sentiment-nor-Velldalsentiment-2018-velldal-nor
nor
Macro-F1
Sentiment-pcm-Muhammadsentiment-2022-muhammad-pcm
pcm
Weight-F1
Sentiment-pcm-Oyewusisentiment-2020-oyewusi-pcm
pcm
Macro-F1
Sentiment-pol-Koconsentiment-2019-kocon-pol
pol
Macro-F1
Sentiment-pol-Mozeticsentiment-2016-mozetic-pol
pol
Macro-F1
Sentiment-pol-Rybaksentiment-2020-rybak-pol
pol
Macro-F1
Sentiment-por-Brumsentiment-2018-brum-por
por
Macro-F1
Sentiment-por-Mozeticsentiment-2016-mozetic-por
por
Macro-F1
Sentiment-ron-Dumitrescusentiment-2020-dumitrescu-ron
ron
Macro-F1
Sentiment-rus-Mozeticsentiment-2016-mozetic-rus
rus
Macro-F1
Sentiment-slk-Mozeticsentiment-2016-mozetic-slk
slk
Macro-F1
Sentiment-slv-Mozeticsentiment-2016-mozetic-slv
slv
Macro-F1
Sentiment-spa-Mozeticsentiment-2016-mozetic-spa
spa
Macro-F1
Sentiment-spa-Reisentiment-2016-rei-spa
spa
Macro-F1
Sentiment-sqi-Mozeticsentiment-2016-mozetic-sqi
sqi
Macro-F1
Sentiment-srp-Mozeticsentiment-2016-mozetic-srp
srp
Macro-F1
Sentiment-sun-Winatasentiment-2022-winata-sun
sun
Macro-F1
Sentiment-swe-Mozeticsentiment-2016-mozetic-swe
swe
Macro-F1
Sentiment-swh-Muhammadsentiment-2022-muhammad-swh
swh
Weight-F1
Sentiment-tam-Chakravarthisentiment-2022-chakravarthi-ta
tam
Macro-F1
Sentiment-tel-Marreddysentiment-2022-marreddy-tel
tel
Macro-F1
Sentiment-tha-Suriyawongkulsentiment-2019-suriyawongkul-t
tha
Macro-F1
Sentiment-tso-Muhammadsentiment-2022-muhammad-tso
tso
Weight-F1
Sentiment-Tw-eng-Thelwallsentiment-tw-2012-thelwall-eng
eng
Accuracy
Sentiment-twi-Muhammadsentiment-2022-muhammad-twi
twi
Weight-F1
Sentiment-yor-Muhammadsentiment-2022-muhammad-yor
yor
Weight-F1
Sentiment-yor-Shodesentiment-2022-shode-yor
yor
Macro-F1
Sentiment-Yt-eng-Thelwallsentiment-yt-2012-thelwall-eng
eng
Accuracy
Sentiment-zho-Tansentiment-2008-tan-zho
zho
Macro-F1
Sentiment-zho-Wansentiment-2020-wan-zho
zho
Macro-F1


(6) Subjectivity Analysis

Task Name Identifier
Language
Score MetricURL
BibTeX
Subject-ita-Basilesubject-2014-basile-ita
ita
Macro-F1
Subject-ita-Basilesubject-2016-basile-ita
ita
Macro-F1
Subjective-ces-Pribansubjective-2022-priban-ces
ces
Accuracy
Subjective-kor-Jangsubjective-2013-jang-kor
kor
Macro-F1
Subjective-spa-Barbierisubjective-2016-barbieri-spa
spa
Macro-F1
Subjevtive-eng-Pangsubjevtive-2004-pang-eng
eng
Accuracy