Project title: Microsoft RMIT Cortana Intelligence Institute
Project dates: January 2018-January 2020
Grants and funding: Microsoft Research USA
Description
Cortana Intelligence Institute is a co-funded initiative between Microsoft Research, Cortana Research and RMIT University, which will drive the next-generation of capabilities for Microsoft’s digital assistant, Cortana.
The Institute focused on researching work-related tasks, an area that will help make Cortana a more proactive and context-aware digital assistant that truly amplifies human capabilities.
Using sensors in mobile phones, the team built a complex multidimensional data set, which will be used to model and predict a person’s work-related tasks. This includes the physical activity and location of a user, their online and app behaviour, and their interactions with their social groups or peers.
Microsoft’s researchers collaborated with RMIT team members on the development of algorithms that use the novel dataset to improve Cortana.
Research strategy
This research project contributes to a greater understanding of the contextual factors that may characterize, or even influence, the tasks being performed by professionals and non-professionals. We take rich contextual factors derived from spatial, temporal, and online activities to better understand participants’ task habits. We use repeated patterns of tasks as cues in characterizing implicit task habits. We examine task batching behaviours and predict the tasks participants are about to undertake. Our work is evaluated on the CII dataset. Our initial research observations from the dataset include findings on that the tasks participants performed when commuting to/from work, were often personalised and may also be influenced by their job roles and demographic attributes. A qualitative analysis of the weekly interview data has provided nuanced insight from participants on how an ideal intelligent task assistant could be.
Rationale
The work of the Institute will enhance the work done by Microsoft Research and Cortana on task intelligence, enabling Cortana to support more complex work tasks: tracking a person progress on a task, reminding them about important task deadlines, or assisting with completing a task. The ultimate goal for a system like Cortana is to create a digital executive assistant for a user: a system that can manage a calendar, understand the user, be aware of context, and have a rich interactions with the user.
Key people
CI
- Mark Sanderson
- Flora Salim
- Yongli Ren
- Falk Scholer
Former Postdoc and PhD students
- Jonathan Liono
- Mohammad Saiedur Rahaman
- Damiano Spina
- Johanne Trippas
Associated journal publications
The research team has so far generated four research papers, with several more in the pipeline.
The research team created a set of potential patent applications during an in-person patent harvesting session at RMIT. This resulted in an initial set of 10 or so ideas, for each of which we wrote a paragraph describing the core invention. We distilled and prioritized the ideas in conjunction with the Microsoft legal team. We have so far submitted four patents for review by that team. One patent application has been approved and lodged:
Patent (pending)
- R. White, O. Shaya, K. Carter, Y. Ren, J. Liono, F. Salim, Method and system for scheduling tasks based on cyber-physical-social contexts (408146-US-NP)
List of publications
- J. R. Trippas, D. Spina, F. Scholer, A. H. Awadallah, P. Bailey, P. N. Bennett, R. W. White, J. Liono, Y. Ren, F. D. Salim, and M. Sanderson. Learning about work tasks to inform intelligent assistant design. In Proceedings of Conference on Information Interaction and Retrieval (CHIIR), pages 5–14, 2019.
- J. Liono, F.D. Salim, N. van Berkel, V. Kostakos, and A.K. Qin, 2019, March. Improving Experience Sampling with Multi-view User-driven Annotation Prediction. In 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom (pp. 1-11). IEEE.
- J. Liono, J. R. Trippas, D. Spina, M. S. Rahaman, Y. Ren, F. D. Salim, M. Sanderson, F. Scholer, and R. W. White. Building a benchmark for task progress in digital assistants. In Proceedings of WSDM’19 Task Intelligence Workshop (TI@WSDM19), 2019. 6 pages.
- J. Liono, M. S. Rahaman, F. D. Salim, Y. Ren, D. Spina, F. Scholer, J. R. Trippas, M. Sanderson, P. N. Bennett, and R. White. Intelligent Task Recognition: Towards Enabling Productivity Assistance in Daily Life. In International Conference on Multimedia Retrieval (ICMR’20).
- I. Holcombe-James, M. S. Rahaman, P. Bailey, P. N. Bennett, Y. Ren, M. Sanderson, F. Scholer, R. White, and F. D. Salim. Imagining a Workplace Digital Assistant: Identifying the Tasks Professionals Want Managed. In Computer Supported Collaborative Work 2020, ACM Proceedings of HCI. Under Review
