Web: https://www.reddit.com/r/MachineLearning/comments/xfbo2w/d_using_a_lipreading_network_as_a_training_loss/

Sept. 15, 2022, 11:46 p.m. | /u/zergling103

Machine Learning reddit.com

One thing I noticed with facial performance transfer networks is often the reenactment tends to poorly imitate the mouth movements of the source actor. The overall shape is preserved fairly well, but subtle "visemes" (visual analog of "phonemes") like those produced when pronouncing consonants seem to regress to the mean.

In 2016, a project called LipNet ([https://www.youtube.com/watch?v=wg3upHE8qJw](https://www.youtube.com/watch?v=wg3upHE8qJw)) had shown impressive results in predicting dialogue from viseme inputs only (no audio inputs).

Therefore, perhaps a similar network could be used to …

face loss machinelearning network performance reading training

Postdoctoral Fellow: ML for autonomous materials discovery

@ Lawrence Berkeley National Lab | Berkeley, CA

Research Scientists

@ ODU Research Foundation | Norfolk, Virginia

Embedded Systems Engineer (Robotics)

@ Neo Cybernetica | Bedford, New Hampshire

2023 Luis J. Alvarez and Admiral Grace M. Hopper Postdoc Fellowship in Computing Sciences

@ Lawrence Berkeley National Lab | San Francisco, CA

Senior Manager Data Scientist

@ NAV | Remote, US

Senior AI Research Scientist

@ Earth Species Project | Remote anywhere

Research Fellow- Center for Security and Emerging Technology (Multiple Opportunities)

@ University of California Davis | Washington, DC

Staff Fellow - Data Scientist

@ U.S. FDA/Center for Devices and Radiological Health | Silver Spring, Maryland

Staff Fellow - Senior Data Engineer

@ U.S. FDA/Center for Devices and Radiological Health | Silver Spring, Maryland

Senior Data Engineer

@ HealthVerity | United States

Data Analyst - Business Insights

@ Sertis | Bangkok

Biomarker Data Management Specialist

@ Precision Medicine Group | Remote, United States