[Discussion] What are frameworks used for Human-In-The-Loop (Active) learning ?

We have a new request from our product team : "Develop a human in the loop binary classifier for a fashion application. The classifier should learn the themes of photos and classify the future data with high degree of confidence. "

The human in the loop will be an annotator (or) expert whom we can query to label the examples (of mini-batches). The number of queries is limited by confidence score and budget.

Are there any tools/frameworks for building such …

frameworks human learning loop machinelearning