Machine readable emotions

Standardization work in the field of affective computing

The science fiction movie Her tells the story of a man who falls in love with the virtual assistant on his computer’s operating system. The assistant is personified through a female voice and, using artificial intelligence, can learn and evolve as typified through the complexity of the conversations. While this movie is based upon technology that does not exist, it does address questions about human relationships with machines.

Image of four emojis faces expressing different emotions
(Photo: Gino Crescoli from Pixabay)

Research is making a tremendous progress in a new area known as affective computing which seeks to bring together computational technology and human emotions. Efforts include finding ways to forecast a user’s mood (happiness, anger, depression, etc.) and enabling robots and computers to modulate their responses based upon the emotional input received.

The joint IEC and ISO technical subcommittee that provides standardization in the field of user system interfaces (ISO/IEC JTC 1/SC 35) has recently set up a new working group on affective computing. The Chair of ISO/IEC JTC 1/SC 35, Khalid Choukri, spoke with e-tech about the group’s current work.

Using affective computing

Call centres can be a source of frustration for many people. Long wait times, automated responses and endless rounds of soothing music can result in a negative customer experience. Attempts to reduce costs and customer wait times have resulted in the use of automated bots to respond to customer queries.

Choukri explains: "Imagine contacting a calling centre because you are upset about an error with your invoice. Managing this conversation would be very different from a conversation where you call to say how happy you are because of the extra gigabytes that they have given to you. The mood of the person calling, whether happy or upset, should be taken into consideration by the interface".

Affective computing can allow for chatbots and virtual assistants that have enhanced empathy, greater interaction and the ability to transform emotions into data. Research seeks to enable computer technology to better understand and manage the conversation.

As Choukri notes, "a very good customer call system should be able to understand if a customer is unhappy and needs to be put in touch with a senior manager. Or if the customer just needs a basic service. It should also be able to understand if the caller is sweet, cynical or has a sense of humour".

An additional complexity for affective computing is understanding cultural differences. Choukri explains that "we need to ensure that machines are culturally and linguistically adjusted to understand such differences as a happy or unhappy Norwegian compared with an Arab American".

Another issue may arise with the difference in language skills. For example, Choukri notes that machines will need to be able to decipher the language of native and non-native speakers. "A non-native English speaker cannot be expected to enunciate like the British".

Challenges in collecting data

One issue that needs to be addressed is the collection of the data that will be needed to train algorithms in affective computing. Much data will be needed but it must be collected in an ethical manner. As Choukri explains, "we cannot really push people to be upset".

He further notes that "we also need to be ethical in the way we process data. Whatever decision made by the system must be explainable. We need to include trustworthiness in our systems".

The SC 35 working group is currently examining emotional characteristics and specifically how data is gathered and identified. This work is expected to take several years. As Choukri notes, "there is a lot of research in affective computing but also a lot of controversy on issues like what kinds of emotions we can identify in human interaction. It is a very challenging project".

Further challenges include the diversity of affective characteristics and the way to interpret and reply to these characteristics.

Development of a new standard

SC 35 is currently in the final stages of developing the first part of a standard, ISO/IEC 30150, on the user interface for affective computing. This first part provides a model for affective computing user interfaces and areas where further standardization will be needed. While it does not specify the implementation of affective computing, it is intended for developers of systems that want to meet the needs of diverse users.