PiaMuscle: Improving Piano Skill Acquisition by Estimating and Visualizing Activity of Hand Muscles
RUOFAN LIU
[{'start': '1.440', 'end': '5.680', 'text': "Hi everyone. I'm Ruofan, from\xa0\nInstitute of Science Tokyo."}, {'start': '5.680', 'end': '10.960', 'text': "And today I'm super excited to present\xa0\nPiaMuscle: Improving Piano Skill"}, {'start': '10.960', 'end': '18.480', 'text': 'Acquisition by Cost-effectively Estimating and\xa0\nVisualizing Activities of Miniature Hand Muscles.'}, {'start': '18.480', 'end': '23.560', 'text': 'This work was done with my awesome\xa0\ncollaborators, Yichen, Takanori,'}, {'start': '23.560', 'end': '30.960', 'text': 'Chen-Chieh, and Erwin, as well as my amazing\xa0\nsupervisors, Shinichi Furuya and Hideki Koike.'}, {'start': '31.560', 'end': '37.960', 'text': 'Skillful human movement encompasses interactions\xa0\nbetween the muscular and skeletal systems.'}, {'start': '37.960', 'end': '43.600', 'text': 'Quantitative assessment of muscles\xa0\ncould facilitate various applications,'}, {'start': '43.600', 'end': '53.080', 'text': 'including rehabilitation, fatigue prevention,\xa0\nand enhancement of skill acquisition.'}, {'start': '53.080', 'end': '57.840', 'text': 'But contemporary muscle measurement\xa0\neither requires electromyography'}, {'start': '57.840', 'end': '65.840', 'text': 'sensors and lab environments, or confines\xa0\nto gross motors involving large muscles.'}, {'start': '65.840', 'end': '74.200', 'text': 'For dexterous movements like piano performance,\xa0\nneither of which is really practical for pianists.'}, {'start': '74.200', 'end': '80.080', 'text': 'So instead in this research, we proposed\xa0\nPiaMuscle to explore the efficacy of using'}, {'start': '80.080', 'end': '88.040', 'text': 'readily accessible data such as videos\xa0\nand keystroke motions to infer muscle EMG.'}, {'start': '88.040', 'end': '92.560', 'text': "And subsequently, compare pianists'\xa0\nhand muscle activation with the"}, {'start': '92.560', 'end': '97.400', 'text': 'reference to enhance force\xa0\ncontrol in piano training.'}, {'start': '97.400', 'end': '104.000', 'text': "So there's been a lot of work done in evaluating\xa0\nthe muscle activities with external instruments."}, {'start': '104.000', 'end': '109.520', 'text': 'For example, sensor-based systems\xa0\nlike Delsys and MuscleRehab use'}, {'start': '109.520', 'end': '115.400', 'text': 'prohibitive sensors in controlled lab\xa0\nsettings to record muscle engagement.'}, {'start': '115.400', 'end': '119.840', 'text': 'However, these systems require high\xa0\ncost or high computational demands'}, {'start': '120.360', 'end': '124.240', 'text': 'and are inaccessible for most consumers.'}, {'start': '124.240', 'end': '128.640', 'text': 'On the other hand, data-driven and\xa0\nmachine-learning-based methods like'}, {'start': '128.640', 'end': '134.120', 'text': 'OpenCap demonstrate high accuracy when the\xa0\ninput data falls within the distribution'}, {'start': '134.120', 'end': '141.720', 'text': 'of the training set, which means it just\xa0\nfocuses on gross motor dynamic prediction.'}, {'start': '141.720', 'end': '145.200', 'text': 'Thus, overall, cost-effective methods for'}, {'start': '145.200', 'end': '149.840', 'text': 'estimating miniature muscles\xa0\nremain largely unexplored.'}, {'start': '150.640', 'end': '157.160', 'text': "To address these issues, we proposed a workflow\xa0\nto estimate miniature muscles' activities from"}, {'start': '157.160', 'end': '163.920', 'text': 'human-centric data and more accurately\xa0\nwhen combined with tool-centric data.'}, {'start': '163.920', 'end': '168.640', 'text': 'And we also visualized inferred\xa0\nmuscles and conducted user study'}, {'start': '168.640', 'end': '174.240', 'text': 'regarding the estimation accuracy\xa0\nand piano training effectiveness.'}, {'start': '174.240', 'end': '180.200', 'text': 'So first, we invited professional\xa0\npianists to collect a multimodal dataset.'}, {'start': '180.200', 'end': '183.960', 'text': 'Six muscles of a single hand were specified.'}, {'start': '183.960', 'end': '191.840', 'text': 'And an RGB camera was fixed to the\xa0\ntop-middle of the piano to record the videos.'}, {'start': '191.840', 'end': '198.840', 'text': 'As for keystroke motions, we utilized an\xa0\noptical sensing system from previous work.'}, {'start': '198.840', 'end': '201.080', 'text': 'After preprocessing all collected data,'}, {'start': '201.080', 'end': '209.760', 'text': 'we compiled 8 million frames of keystroke\xa0\nand EMG, along with 480k frames of videos.'}, {'start': '211.040', 'end': '214.960', 'text': "Here's a detailed look at\xa0\nour network architecture."}, {'start': '214.960', 'end': '220.080', 'text': 'To avoid potential distortions\xa0\nin direct estimations,'}, {'start': '220.080', 'end': '224.720', 'text': 'we trained two VQ-VAE at the first stage.'}, {'start': '224.720', 'end': '229.160', 'text': 'Rather than directly downsampling\xa0\nthe EMG, we constructed a discrete'}, {'start': '229.160', 'end': '235.960', 'text': 'codebook for EMG representation\xa0\nto reduce EMG temporal complexity.'}, {'start': '235.960', 'end': '242.760', 'text': 'Meanwhile, a codebook for pose to\xa0\ncompress it in spatial dimension.'}, {'start': '242.760', 'end': '246.640', 'text': 'In the second stage, a\xa0\ncross-attention-based transformer,'}, {'start': '246.640', 'end': '253.360', 'text': 'EMGFormer, was introduced to predict\xa0\nEMG from keystroke and video inputs.'}, {'start': '253.360', 'end': '260.880', 'text': 'We encoded the keystroke with the TCN and\xa0\nthe pose with the pre-trained pose encoder.'}, {'start': '260.880', 'end': '263.760', 'text': 'During training, we froze the parameters from'}, {'start': '263.760', 'end': '269.720', 'text': 'the first stage and optimized the loss\xa0\nby minimizing the EMG embedding loss.'}, {'start': '270.960', 'end': '276.480', 'text': 'Ablation studies were conducted to\xa0\nvalidate model estimation performance.'}, {'start': '276.480', 'end': '283.080', 'text': 'Interestingly, we found the experiment using\xa0\nonly video demonstrated notable accuracy,'}, {'start': '283.080', 'end': '289.960', 'text': 'while more modal inputs like keystroke\xa0\nmotions can further improve accuracy.'}, {'start': '289.960', 'end': '296.680', 'text': "Here's an example focusing on 4DI Muscle. The\xa0\nblue line represents the ground truth EMG,"}, {'start': '296.680', 'end': '299.640', 'text': 'while the orange displays our estimated values.'}, {'start': '301.120', 'end': '306.560', 'text': 'And we also listed more performance results\xa0\nhere. The top figure shows the ground truth,'}, {'start': '306.560', 'end': '312.080', 'text': "while the bottom presents our model's predictions."}, {'start': '312.080', 'end': '315.880', 'text': 'Based on the network, PiaMuscle\xa0\nvisualized the inferred muscle'}, {'start': '315.880', 'end': '322.960', 'text': 'activity after temporarily aligning the\xa0\nusers with the reference using K-TCC.'}, {'start': '323.480', 'end': '329.680', 'text': 'Here K-TCC refers to a custom network\xa0\nto achieve accurate synchronization'}, {'start': '329.680', 'end': '334.600', 'text': 'specifically tailored for piano performances.'}, {'start': '334.600', 'end': '337.560', 'text': 'Then the aligned videos, synchronized hands,'}, {'start': '337.560', 'end': '343.480', 'text': 'and discrepancy prompts on the timeline were\xa0\ndisplayed to show the muscle difference.'}, {'start': '343.480', 'end': '350.520', 'text': 'And click the "flip model" button\xa0\nhere to see the palmar side.'}, {'start': '350.520', 'end': '355.920', 'text': 'With PiaMuscle, we conducted a user\xa0\nstudy with five highly skilled pianists,'}, {'start': '355.920', 'end': '359.960', 'text': 'without and with muscle activity visualization.'}, {'start': '359.960', 'end': '367.080', 'text': 'Specifically, they experienced two comparative\xa0\nconditions, each session for 30 minutes.'}, {'start': '367.080', 'end': '374.600', 'text': 'And their pre-study and post-study performances\xa0\nwere recorded for quantitative analyses.'}, {'start': '374.600', 'end': '377.320', 'text': 'A comprehensive evaluation of the system can be'}, {'start': '377.320', 'end': '382.920', 'text': 'found in the paper but here let me\xa0\njust go over a couple of highlights.'}, {'start': '382.920', 'end': '388.440', 'text': 'So here, higher MAI represents\xa0\nbetter force control, and overall,'}, {'start': '388.440', 'end': '394.560', 'text': 'pianists show greater improvement\xa0\nin muscle accuracy under PiaMuscle.'}, {'start': '394.560', 'end': '401.040', 'text': 'Higher KAI and KCI represent more accurate\xa0\nand closer keystrokes to the reference,'}, {'start': '401.040', 'end': '406.600', 'text': 'and our system achieves better results too.'}, {'start': '406.600', 'end': '413.920', 'text': 'As for questionnaires, PiaMuscle obtains higher\xa0\nscores in all dimensions from UEQ results.'}, {'start': '413.920', 'end': '419.520', 'text': 'Regarding helpfulness, intuitiveness, and\xa0\noperability, PiaMuscle is better than only videos.'}, {'start': '421.400', 'end': '427.400', 'text': "Also based on participants' rating\xa0\nfor the visualized muscle accuracy,"}, {'start': '427.400', 'end': '433.600', 'text': "it's similarly accurate from\xa0\ntheir previous experience."}, {'start': '433.600', 'end': '437.240', 'text': 'In interviews, all pianists preferred PiaMuscle,'}, {'start': '437.240', 'end': '443.400', 'text': 'because piano muscle information\xa0\nis intuitive and objective.'}, {'start': '443.400', 'end': '446.560', 'text': 'They acknowledged that estimating and visualizing'}, {'start': '446.560', 'end': '454.400', 'text': 'muscle activities is an effective method\xa0\nto facilitate force control training.'}, {'start': '454.400', 'end': '458.440', 'text': "From previous results, we\xa0\ncan confirm it's feasible to"}, {'start': '458.440', 'end': '465.080', 'text': 'cost-effectively estimate miniature muscle\xa0\nactivations from readily accessible data.'}, {'start': '465.080', 'end': '473.640', 'text': 'And furthermore, this workflow can be adapted for\xa0\nother skills, especially for dexterous skills.'}, {'start': '473.640', 'end': '479.440', 'text': 'We also conclude that PiaMuscle can be a\xa0\nsupplement tool for conventional piano training.'}, {'start': '480.600', 'end': '482.720', 'text': 'Because it can provide objective'}, {'start': '482.720', 'end': '488.680', 'text': 'muscle information for reference\xa0\ninstead of absolute correctness.'}, {'start': '488.680', 'end': '499.400', 'text': 'And it can also address some video limitations,\xa0\nsuch as hand occlusion and audio distortion.'}, {'start': '499.400', 'end': '504.760', 'text': 'To summarize, we present PiaMuscle,\xa0\na designed workflow that democratizes'}, {'start': '504.760', 'end': '511.960', 'text': 'muscular activity monitoring by cost-effective\xa0\nestimation from readily accessible data, and'}, {'start': '511.960', 'end': '519.720', 'text': 'visualization to enhance acquisition of various\xa0\ndexterous skills that rely on small muscles.'}, {'start': '519.720', 'end': '524.680', 'text': 'Please refer to our paper\xa0\nfor more details and results.'}, {'start': '524.680', 'end': '530.040', 'text': 'Thanks! See you around the conference!'}]