TEam Copilot: Strategic Concept Research series for AI

Overview

Over that last year at Microsoft, I have led research for Copilot (AI) in Teams meetings, the first to do so. The lead PM for Copilot in meetings approached me earlier in the year for guidance on a new AI concept that helps a group of users instead of one user at a time. They asked for actionable feedback and recommendations to determine if Satya would announce at Build, an upcoming conference. Ultimately, I planned and executed a series of studies that provided the strategic guidance needed for the team to confidently introduce Team Copilot at the conference.

The main research questions were:

• What is the perceived value of AI in meetings to align the team on the most valuable scenarios to pursue?

• How do user expect to engage with Copilot as a group during meetings?

• What are the potential usability issues to ensure the team is building an easy-to-use experience?

Approach

I developed a multi-method approach to evaluate Team Copilot concepts across several studies with a focus on getting rich in-depth feedback.

I partnered closely with a fellow research lead who was working with analogous Team Copilot capabilities in Teams chat, to ensure our research was aligned and to expand impact across multiple teams of stakeholders.

The initial plan called for concept and in-code feedback prior to Build conference, but I demonstrated adaptability by modifying the plan when it became clear that the in-code experience would not be ready.

Ultimately, I planned and ran three studies:

1. Individual interviews + Cognitive walkthrough: In depth discussions covering the Team Copilot interaction model, expectations, and proposed capabilities (12 users, mix of admins and end users)

2. Group session: Follow-up group interview to revisit and gather further feedback on the interaction model (18 end users)

3. Co-design session at a conference: Due to NDA limits with the audience, focused on taking a step back to evaluate the idea of Copilot helping the group (10 users)

1. Individual Interviews + Cognitive Walkthrough

I designed and executed a critical concept feedback study to guide and inform stakeholders on the ways to build and integrate Team Copilot, highlighting important ‘need to knows’ and potential pitfalls. Using a mix of interview and cognitive walkthrough methods, I covering the Team Copilot interaction model, expectations, and proposed capabilities. (12 users with Copilot experience, mix of admins and end users).

Research Goals:

• Understand perceptions and overall perceived value of Team Copilot concept

• Understand perceived value for the proposed MVP capabilities

• Understand the expected level of presence/representation of Team Copilot in meetings

• Understand the expected level of Copilot participation (i.e., proactive vs reactive)

• Understand expected controls, including skill management

Process & Reporting:

Ongoing ‘initial’ insights were regularly shared with key stakeholders during the interview stage, and updated as insights were refined. This ongoing sharing provided timely directional guidance while I worked on a final report.

For the readout, I included stakeholders across the entire Team Copilot effort so the team could be thinking holistically about the overall framework (i.e., beyond meetings).

Key portions to the final report—Research POV, Executive Summary, and Detailed Findings—can be seen below:

Impact:

Following the readout, I participated in cross-team discussions and brainstorms regarding best practices and the overall framework for Team Copilot, and prepared the team for high-level leadership meetings. The ‘What to keep in mind’ checklist I created for understanding, interacting with and managing Team Copilot was particularly impactful across multiple teams and has continued to be a point of reference for design considerations and best practices.

Lots of good insights here. … The other feedback that also resonated with me and is a hard problem to solve is the … Would love to understand/discuss what we can do here to simplify or mitigate these problems. These are not easy problems given the path we have chosen. - CVP for Teams Meetings

There was still ongoing debate on the approach towards aspects like entry points and the co-existence of the current Copilot, and stakeholders opted to make some compromises on the design to meet the deadline for a Build announcement. However, I continued to be vocal in this area to hold stakeholders accountable and make sure the designs would be updated as part of a fast-follow during private preview.

2. Group Session

To gather additional concept feedback around the interaction model for Team Copilot, I leveraged a group call with 18 external end users. This call was the first of a new customer series I helped set up with customer PMs to gather Copilot feedback. I quickly put together a plan that provided feedback which addressed some of the top outstanding items following the individual interviews.

Key topics addressed:

• Interacting with a shared Copilot

• Adding and removing a shared Copilot during a meeting

• Adding and removing a shared Copilot prior to a meeting

Impact:

As can be the case with larger studies, I was not able to fully cover some of the interaction details in the previous individual interviews. By leveraging this group session, I focused in on the needed details and provided key insights for the Team Copilot interaction model that I shared across multiple teams.

Thank you Nathan! There was so much great feedback! I gave <lead PM> and the core team a quick breakdown of the overtones that validated our concerns but we need to address this with a larger and heavier audience as soon as possible. - Lead Designer for Meetings Copilot

3. Co-design Session

For a M365 conference, I had planned to use the session to get Team Copilot feedback leveraging an in-code demo, which was a natural progression from the concept testing already completed. However, since this session was run prior to the announcement at Build, I was asked to pivot to a design-free session focused on brainstorming and co-design and gathered fantastic feedback for the team. I successfully managed stakeholder expectations so that they were still very excited in spite of these changes.

Process:

I spoke with 10 experienced users of Copilot during Teams meetings during a 75 minute group session. After setting the stage with a discussion of their current experience, I asked participants to brainstorm around the following prompt: 

  "Imagine Copilot is available in Teams meetings and everyone can engage with it."

Without showing any designs or even mentioning the word ‘Group Copilot,' I asked them to individually think about:

• Skills: Scenarios where Copilot could help everyone in the meeting (including $100 exercise for prioritization)

• Interactions: How this AI would work and how everyone would interact with it

• Current Copilot: What they would do about the current Copilot experience 

• Controls/Managing: How they may enable or add this AI to meetings, and what controls they expect 

After the individual brainstorming, participants got together in small groups for a co-design exercise. In addition to lively discussions that included key leadership stakeholders, participants presented their designs during the session. 

Impact:

Key insights were quickly shared by way of a news post, and the session itself was recognized as the top session among all the other research sessions at the conference (over 30). In spite of the ‘no design’ constraints imposed on the group session, the output was a nice wrap of the concept research series. It generated needed energy and customer empathy for the team, provided insights that strengthened earlier findings, and confirmed that the general Copilot concept was on the right track. The co-design study, along with the previous two studies, provided the strategic guidance the team needed to tweak the designs and the confidence to proceed to coding. Shortly thereafter, Microsoft CEO Satya Nadella announced Team Copilot to the world at the Microsoft Build Conference (May 2024).

That was phenomenal! - Partner GPM for Teams Meetings