Automatic design optimization of preference-based subjective evaluation with online learning in crowdsourcing environment

arXiv:2403.06100v1 Announce Type: cross
Abstract: A preference-based subjective evaluation is a key method for evaluating generative media reliably. However, its huge combinations of pairs prohibit it from being applied to large-scale evaluation using crowdsourcing. To address this issue, we propose an automatic optimization method for preference-based subjective evaluation in terms of pair combination selections and allocation of evaluation volumes with online learning in a crowdsourcing environment. We use a preference-based online learning method based on a sorting algorithm to identify …

abstract arxiv crowdsourcing cs.cl cs.hc cs.lg design eess.as environment evaluation generative however issue key media online learning optimization scale stat.ml type