Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest. (arXiv:2209.06293v1 [cs.CL])

We challenge AI models to "demonstrate understanding" of the sophisticated
multimodal humor of The New Yorker Caption Contest. Concretely, we develop
three carefully circumscribed tasks for which it suffices (but is not
necessary) to grasp potentially complex and unexpected relationships between
image and caption, and similarly complex and unexpected allusions to the wide
varieties of human experience; these are the hallmarks of a New Yorker-caliber
cartoon.

We investigate vision-and-language models that take as input the cartoon
pixels and caption directly, …

arxiv benchmarks contest humor understanding