Gentle Gestures of Control: On the Somatic Sensibilities of an IoT Remote App

Main Article Content

Jūra Miniotaitė
Vaida Pakulytė
Ylva Fernaeus


The design of user experiences for physical appliances increasingly involves connection, monitoring, and control via smartphone applications. Despite the rich possibilities for interaction provided by smartphones, the current standard mode of engagement with such apps is through graphical user interface manipulations. To explore new felt experiences for this use context, a remote-control app for a robotic vacuum cleaner was designed, enabling participants to have their gaze focused on the robot, while steering it by gently tilting the phone. This particular interaction is used as a case to emphasize the role of somatic sensibilities when designing smartphone applications in the context of IoT. Through a phenomenologically-inspired analysis, we describe the user experience in terms of physical manipulation, perception, effort, and utility, and through social and emotional engagement. An important attribute was how the interaction, through its subtleness, created a somatically connected experience.

Article Details

How to Cite
Miniotaitė, J., Pakulytė, V., & Fernaeus, Y. (2022). Gentle Gestures of Control: On the Somatic Sensibilities of an IoT Remote App. Diseña, (20), Article.1.
Original articles
Author Biographies

Jūra Miniotaitė, KTH Royal Institute of Technology, Department of Media Technology and Interaction Design

BA in Media Technology, KTH Royal Institute of Technology. M.Sc. in Interactive Media Technology, KTH Royal Institute of Technology. She is a Doctoral student in Multimodal Machine Learning at KTH. As a researcher, she explores Human-robot Interaction, social robotics, and embo­died interaction. She is the co-author of ‘A Case Study of Deep Enculturation and Sensorimotor Synchronization to Real Music’ (with O. Misgeld, T. Gulz, and A. Holzapfel; in International Society for Music Information Retrieval Conference 2021).

Vaida Pakulytė, Electrolux CX, Software Team

BA in International Communication and Mul­timedia, Aarhus Universitet. MA in Digital Marketing & Social Media, London School of Business and Finance. She works on user experience and interaction design as part of the Consu­mer Experience Software team at Electrolux, specializing on applications related to robotic vacuum cleaners and air purifiers.

Ylva Fernaeus, KTH Royal Institute of Technology, Department of Media Technology and Interaction Design

BA in Computer and Systems Sciences, Stoc­kholm University. M.Sc. in Interactive Multimedia, University of Westminster. Ph.D. in Human-Machine Interaction, Stockholm University. Associate Professor in the Department of Media Technology and Interaction Design at KTH and guest researcher at Umeå Institute of Design. Her research is focuses on physical interaction design, embodied programming, preciousness in interaction, and hybrid crafting. Her latest publications include ‘Envisioning Arduino Action: A Collaborative Tool for Physical Computing in Educational Settings’ (with G. J. Roumen, Inter­national Journal of Child-Computer Interaction, Vol. 29, Issue 100277); ‘Co-watching 360-Films in Nursing Homes’ (with A. Lundström and S. Ghebremikael; in Proceedings of 13th IFIP Conference on Human-Computer Interaction); and ‘Troubling Care: Four Orientations for Wickedness in Design’ (with K. Helms; in Designing Interactive Systems Conference 2021).


Aloi, G., Caliciuri, G., Fortino, G., Gravina, R., Pace, P., Russo, W., & Savaglio, C. (2017). Enabling IoT Interoperability through Opportunistic Smartphone-based Mobile Gateways. Journal of Network and Computer Applications, 81(C), 74–84.

Ando, T., Isomoto, T., Shizuki, B., & Takahashi, S. (2018). Press & Tilt: One-handed Text Selection and Command Execution on Smartphone. Proceedings of the 30th Australian Conference on Computer-Human Interaction, 401–405.

Benford, S., Magerkurth, C., & Ljungstrand, P. (2005). Bridging the Physical and Digital in Pervasive Gaming. Communications of the ACM, 48(3), 54–57.

Bentley, F., & Barrett, E. (2012). Building Mobile Experiences. MIT Press.

Buur, J., Jensen, M. V., & Djajadiningrat, T. (2004). Hands-only Scenarios and Video Action Walls: Novel Methods for Tangible User Interaction Design. Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, 185–192.

Candau, Y., Françoise, J., Alaoui, S. F., & Schiphorst, T. (2017). Cultivating Kinaesthetic Awareness through Interaction: Perspectives from Somatic Practices and Embodied Cognition. Proceedings of the 4th International Conference on Movement Computing, 1–8.

Coronado, E., Villalobos, J., Bruno, B., & Mastrogiovanni, F. (2017). Gesture-based Robot Control: Design Challenges and Evaluation with Humans. 2017 IEEE International Conference on Robotics and Automation, 2761–2767.

Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction. MIT Press.

Eriksson, S., Höök, K., Shusterman, R., Svanes, D., Unander-Scharin, C., & Unander-Scharin, Å. (2020). Ethics in Movement: Shaping and Being Shaped in Human-Drone Interaction. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14.

Fernaeus, Y., & Jacobsson, M. (2009). Comics, Robots, Fashion and Programming: Outlining the Concept of ActDresses. Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, 3–8.

Fernaeus, Y., & Sundström, P. (2012). The Material Move How Materials Matter in Interaction Design Research. Proceedings of the Designing Interactive Systems Conference, 486–495.

Fernaeus, Y., Tholander, J., & Jonsson, M. (2008). Beyond Representations: Towards an Action-centric Perspective on Tangible Interaction. International Journal of Arts and Technology, 1(3/4), 249–267.

Forlizzi, J. (2007). How Robotic Products Become Social Products: An Ethnographic Study of Cleaning in the Home. Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, 129–136.

Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007). Gaze Cueing of Attention: Visual Attention, Social Cognition, and Individual Differences. Psychological Bulletin, 133(4), 694–724.

Hallnäs, L., & Redström, J. (2001). Slow Technology – Designing for Reflection. Personal and Ubiquitous Computing, 5(3), 201–212.

Hardy, R., & Rukzio, E. (2008). Touch & Interact: Touch-based Interaction of Mobile Phones with Displays. Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services, 245–254.

Hendriks, S., Mare, S., Gamboa, M., & Baytaþ, M. A. (2021). Azalea: Co-experience in Remote Dialog through Diminished Reality and Somaesthetic Interaction Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Article 261). Association for Computing Machinery.

Höök, K. (2018). Designing with the Body: Somaesthetic Interaction Design. MIT Press.

Hung, C.-H., Bai, Y.-W., & Wu, H.-Y. (2016). Home Outlet and LED Array Lamp Controlled by a Smartphone with a Hand Gesture Recognition. 2016 IEEE International Conference on Consumer Electronics, 5–6.

La Delfa, J., Baytas, M. A., Patibanda, R., Ngari, H., Khot, R. A., & Mueller, F. F. (2020). Drone Chi: Somaesthetic Human-Drone Interaction. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13.

Loke, L., & Núñez-Pacheco, C. (2018). Developing Somatic Sensibilities for Practices of Discernment in Interaction Design. The Senses and Society, 13(2), 219–231.

Miniotaité, J. (2021). JoyTilt: Beyond GUI App Design for Embodied Experience of Controlling a Robot Vacuum Cleaner [Master´s Thesis]. KTH, School of Electrical Engineering and Computer Science.

Nintendo. (n.d.). Nintendo History. Nintendo of Europe GmbH. Retrieved March 24, 2020, from

Norman, D. A. (1993). Cognition in the Head and in the World: An Introduction to the Special Issue on Situated Action. Cognitive Science, 17(1), 1–6.

Soma, R., Dønnem Søyseth, V., Søyland, M., & Schulz, T. (2018). Facilitating Robots at Home: A Framework for Understanding Robot Facilitation. The Eleventh International Conference on Advances in Computer-Human Interactions, 1–6.

Sung, J., Grinter, R. E., & Christensen, H. I. (2010). Domestic Robot Ecology. International Journal of Social Robotics, 2(4), 417–429.

Székely, M., & Michael, J. (2020). The Sense of Effort: A Cost-Benefit Theory of the Phenomenology of Mental Effort. Review of Philosophy and Psychology. Advance Online Publication, 1–16.

Vertesi, J. (2008). “Seeing Like a Rover”: Embodied Experience on the Mars Exploration Rover Mission. CHI ’08 Extended Abstracts on Human Factors in Computing Systems, 2523–2532.

Wu, L., Alqasemi, R., & Dubey, R. (2020). Development of Smartphone-Based Human-Robot Interfaces for Individuals with Disabilities. IEEE Robotics and Automation Letters, 5(4), 5835–5841.