Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions. (arXiv:2211.00688v1 [cs.AI])

The adoption of pre-trained language models to generate action plans for
embodied agents is a promising research strategy. However, execution of
instructions in real or simulated environments requires verification of the
feasibility of actions as well as their relevance to the completion of a goal.
We propose a new method that combines a language model and reinforcement
learning for the task of building objects in a Minecraft-like environment
according to the natural language instructions. Our method first generates a
set …

arxiv building language natural natural language voxel