Shangyun Wu

< I, AI CREATE TOGETHER - Shangavatar > is an experimental media and sound project aiming to clone Shangyun herself as an avatar for replacing the social activities and performance activities which need to face a wider audience in her life. 

As a sound and media researcher, designer and engineer, I have spent years in these two specialties. During my time at the RCA I chose one of the intersections between sound and media which is voice. With synthetic media technology, the AI voice with emotion could offer the capability of suggesting identity, and this could be applied in creating digital avatars and in different mediums. 

In this project, I trained my voice into an AI voice that carries different emotions. The voice becomes the signature of the avatar when it engages with humans in small talk. As a memory of the academic life experience in the UK, the dialog is designed with British daily conversation, that is, it’s English with a slightly Taiwanese accent. Also the AI voice is designed as a part of the interactive music system while Shangavatar plays the role as a musician. 

The showcase of Shangavatar will be on 25th July in iklectik. You are welcome to come and join us! 

API Support : Resemble AI 

copyright © 2022 ShangYunWu all rights reserved 

One of the key points of defining the avatar as Shangyun ( Me ) includes the facial expression while I talk with others. It’s a long process for mapping the real data of facial expression with my real 3D scan model. The negotiation between a realistic facial expression and the preset animation took me lots of time to decide. Through the whole research and development process, I decided to take my real facial expression as the physical feature is a scientific choice while designing the identity of the avatar.

copyright © 2022 ShangYunWu all rights reserved 

The previous Shangavatars are shown in the above videos. According to  “ Self, Identity, and Social Institutions'' by D. Heise, N. MacKinnon, it is the emotional voice and realistic facial expressions that carry the sense of personal identity. To test this, Shangavatar was designed as different iterations with varying appearances and variations on the AI voice. These include a cartoon version, a realistic photo version and a few 3d model versions..etc.

The starting point of designing Shangavatar is due to a shangyun’s personal need. That is, the whole interaction between the avatar and other humans is based on Shangyun’s ( my )  social interactions and experiences based on a human’s visual and auditory perception.

I endowed the avatar with a function based on the concept of a “meaningful glance”, to detect human presence and emotion in a specific spatial area . Once we enter the area, the avatar will attempt to engage us in conversion. The avatar can talk with appropriate emotion and give the relevant feedback which is what humans do most of the time in a social venue. 

This project is based upon research in the following books,  “ Self, Identity, and Social Institutions'' by D. Heise, N. MacKinnon, “ Musical Robots and Interactive Multimodal Systems '' by Jorge Solis, Kia Ng (auth.) and “Designing Sociable Robots” by Cynthia L. Breezee. 

copyright © 2022 ShangYunWu all rights reserved 

The project is designed based on Shangyun's personal experience. Thus, it is designed to have a small talk before and after a performance. In order to convey the whole experience to the audience, the shown mock-up sketch is designed with a scenario from a conversation to a facial detection screen till a performative installation. 

However, Shangavatar, only recently born, will be shown in the RCA’s physical graduate show as a social character role. At the same time, it will participate in a music showcase in Ikletic on 25th, July. You are welcome to save your day to join its showcase. ;) 

copyright © 2022 ShangYunWu all rights reserved 

Creating Shangavatar is a long process, I would like to thank for those who helps me to create Shangavatar with their professional skills and time, especially my friends and family for mental supports with love. This project is for those who love freedom.

Creator - ShangYunWu

Sound and Voice Technician - ShangYunWu

Music Producer - ShangYunWu

Small Talk Consultant - Sophie Maria Vickers

AI Voice Model and API Technical Supporter - Resemble AI 

First Avatar Visual Designer - Lara Pfennigdorff

3D Model Design - ShangYunWu

3D Model Technician - Mike Faulknr

3D Model Consultant - Mike Faulknr, Kam Raoofi , Kyle Ramsey

3D Scan Technician - Justin Calder, Kar Leung Wai

Interaction Designer - ShangYunWu

Interaction Technician - ShangYunWu

Installation Designer - ShangYunWu

Installation Consultant - Rodrigo Canas 

Concept Film Director - ChunTienHsiao

Concept Film Producer - ShangYunWu

Documentary Film Supporters - Chun Tien, Wen Pay, Jun Jun, Shi Ya, Yushen Wu, Ziqin Hou, Wen Hui

Special Thanks - Resemble AI , Professor Robert Rowe , Dr Matt Lewis