RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. (arXiv:2203.13296v2 [cs.CV] UPDATED)

We propose a transformer-based neural network architecture for multi-object
3D reconstruction from RGB videos. It relies on two alternative ways to
represent its knowledge: as a global 3D grid of features and an array of
view-specific 2D grids. We progressively exchange information between the two
with a dedicated bidirectional attention mechanism. We exploit knowledge about
the image formation process to significantly sparsify the attention weight
matrix, making our architecture feasible on current hardware, both in terms of
memory and computation. …

3d arxiv cv objects ray transformers videos