[handbyhand] Virtual hands

Watch These AI Hands Attempt to Play an Unplayable Song on the Piano

But the hands in this video aren't real. They're a 3D rendering that’s programmed to move across the keys of a piano accurately to the tune of any song—except for ones that aren’t humanly possible to play. 

Created by Canadian-based Massive Technologies, the AI pianist is trained to listen to musical compositions and recreate them with virtual hands—and the results are pretty good. 

To generate the AI pianist’s movements, Massive Technologies gave the algorithm an audio file as an input, and it outputs an animation file “complete with the correct playing technique and kinematically accurate hand and body animation,” Salka said. The engineers can then apply that file to a virtual avatar in a virtual environment—videos like what’s on their YouTube channel, augmented reality that lets the user place a pianist at a real-life piano, or virtual reality. 

To train the AI, Salka said his team brought professionally trained pianists to their labs in Helsinki, where they were asked to simply play the piano for hours. The AI observed their playing “through special hardware and sensors,” he said, and throughout the process the pianist and machine learning engineers would check the AI’s work and give it feedback or corrections.

“We would then take that feedback and use it as the curriculum for the AI for our next session with the pianist,” he said. “We repeated that process until the AI results closely matched the human playing technique and style.” 

Massive Technologies launched an AR pianist app in 2017 that puts a virtual piano player in the room with you, but that app only offered a selection of pre-set songs. The company is releasing a new app that incorporates the AI pianist’s listening skills—it’s in the Apple Store now, but still in beta. Salka said that soon users will be able to point their phone’s microphone at any audio source, and the app will show a virtual piano player performing that song. He said he also sees potential for piano teachers to create interactive virtual lessons for remote teaching—or for virtual piano concerts, and film or games creators who want to incorporate a super-realistic pianist in their scenes.  

AI Generated Piano Performance - Sia Chandlier [Concert Creator AI]

MANO model

Conversion from skeleton to hand/body model? #1935

Thank you for your work!
Is there a way to convert the skeleton to a MANO/SMPL model?
That is compute the Hand mesh parameters according to the joint's position (suppose we do not need the skeleton length info)?

Thank you!

We don't have immediate plans to provide ways to generate a hand mesh from the 21 hand landmarks. 

基于参数化模型(MANO)的手势姿态估计---全面剖析

目前, 主流的手部姿态估计的技术方案是使用马普所于2017年提出的MANO参数化模型, 在此基础上回归3D坐标, 这是因为MANO有很合理的结构以及定义好的前向动力学树。本文的目的在于为大家介绍，基于MANO的手部姿态估计的流程：包括并不限于: 数据处理，MANO的推理流程(与论文对齐)，手的解剖学和生物学特点。

https://github.com/otaheri/MANO

关键点变成mesh，可以直接用于显示。

LeapMotion 

JavaScript SDK Documentation

Hello World

https://github.com/leapmotion/leapjs