Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech

Taras Kucherenko, Rajmund Nagy, Patrik Jonell, Michael Neff, Hedvig Kjellström, Gustav Eje Henter

[Project proposal (IVA'21 Honorable Mention)]
[Analysis of gesture property predictability (AAMAS'22)] [SaGA++ Dataset] [Code]

 

 

---

 

ABSTRACT

We propose a new framework for gesture generation, aiming to allow data-driven approaches to produce more semantically rich gestures. Our approach first predicts whether to gesture, followed by a prediction of the gesture properties. Those properties are then used as conditioning for a modern probabilistic gesture-generation model capable of high-quality output. This empowers the approach to generate gestures that are both diverse and representational.

 

---

 

Citation format for the general idea:

@inproceedings{kucherenko2021speech2properties2gestures,
      title={Speech2{P}roperties2{G}estures: {G}esture-Property Prediction as a Tool for Generating Representational Gestures from Speech}, 
      author={Kucherenko, Taras and Nagy, Rajmund and Jonell, Patrik and Neff, Michael and Kjellstr{\"o}m, Hedvig and Henter, Gustav Eje},
      year={2021},
      isbn = {9781450366724},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      url = {https://doi.org/10.1145/3472306.347833},
      doi = {10.1145/3472306.347833},
      booktitle = {Proceedings of the  21th ACM International Conference on Intelligent Virtual Agents},
      location = {Virtual Event, Japan},
      series = {IVA '21}
}

Citation format for the gesture property predictibility analysis:

@inproceedings{kucherenko2022multimodal,
  title={Multimodal analysis of the predictability of hand-gesture properties},
  author={Kucherenko, Taras and Nagy, Rajmund and Neff, Michael and Kjellstr{\"o}m, Hedvig and Henter, Gustav Eje},
  booktitle = {Proceedings of the  21st International Conference on Autonomous Agents and Multiagent Systems},
  series = {AAMAS '22}
  year={2022}
}