In recent years, challenging problems such as human-computer interaction, information retrieval systems, and personal assistants have led to an increased interest in providing machines with the ability to learn and use models of human behaviour. Of particular interest is the modeling and recognition of human intents in their daily routines. Recently a new class of personal assistants that are capable of addressing users’ information needs proactively is emerging. These systems can assist people by recommending the right information at just the right time and help them in accomplishing tasks.
Knowledge workers spend the majority of their working hours processing and manipulating information. These users face continual costs as they switch between tasks to retrieve and create information. While users interact with their digital devices (desktop computer, mobile), they leave behind fine-grained traces of interaction patterns. The ability to learn from user interactions promises pathways for solving a number of problems and improving user satisfaction and in case of office workers, improving their work efficiency. In the COADAPT project, researchers from Aalto University investigated the possibility of a system that will record in detail how users (specifically knowledge workers) perform their digital tasks, and intelligently leverage that information to increase their efficiency and productivity. The work done in the entity recommender system, combines human-computer interaction and machine learning to understand the user intent of doing the tasks at each time instant and recommend more relevant entities to the user related to each task.
An interesting research question in this domain is whether it is possible to build a system that can foresee what information objects and entities users are likely to need while they are doing their tasks, hence setting proactive recommendations to assist users with them. Such augmentation of human memory can effectively serve as a solution in preventing failure to recall previous information they used.
Together with the University of Helsinki, we considered how we can design intelligent user interfaces to make knowledge work less challenging and more productive. This system will be able to continuously monitor the digital activity of the users and based on the current content of the user’s digital activity it can model the user intention and then recommend different types of entities (e.g., people they contacted, tools and applications they used, files and documents they opened, and topics and keywords of their task) that can have a concrete influence on users’ tasks. The development work in COADAPT will include the creation of an actual deployable system, scalable in types of entities that can be recommended. This system is able to predict the future information needs of the user given previous behaviour and the feedback received from the user.
Machine learning for user modeling
For the successful implementation of such functionalities, the Entity Recommender system first needs to understand what is the user context. We define the context of the user as all entities extracted from the digital activities of a user’s computer at each time instant. Imagine you are writing an email to your colleague about a specific task. The user context at that specific time instant, is: email application, contact information of the colleague, and keywords extracted from your email. Since the number of extracted entities from the user’s screen at the monitoring phase is huge ( in our case study about tens of thousands), the context vector needs to be converted into a lower-dimensional latent space. Using Machine Learning techniques together with observed data, researchers have developed methodologies for training computers to get this ability. There are different techniques in natural language processing that can be used for this purpose, such as Latent Dirichlet Allocation, Latent Semantic Analysis, etc.
Next, computers need to infer the user intent from contemporaneous entities for a computational understanding through mathematical analysis. We connect the lower-dimensional contexts to entities of interest (intent) by making an intuitive linearity assumption between relevance of entities and contexts. In addition to the context, we also utilize the user feedback on recommended entities as another learning component. However, this learning signal is very rare and the users are not providing explicit feedback very often. Finally, using Bayesian analysis and defining appropriate priors and likelihood functions for different learning signals (the user’s context as well as user’s feedback) the system can recommend more relevant entities to the user. In the above example, imagine if you and your colleague are working on a document together and you want to attach the file of that document to your email, based on the current context, the recommender system should be able to recommend that file to you along with other relevant entities.
The implemented Entity Recommender system is able to anticipate user’s needs and respond with information appropriate to the current context. The results achieved so far demonstrate the viability of the approach by showing the added value provided by entity recommendation in terms of discovery of novel and relevant entities. More importantly, it is to discover how usefulness and influence of recommendation on the user real-life tasks, leading to improved user experience in completing the tasks.
