Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization

arXiv:2405.02517v1 Announce Type: new
Abstract: Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability …

abstract adversarial arxiv creative cs.cl datasets generate iterative language language models large language large language models logic optimization performance prompt research solutions tasks tests thinking thought type