AI/ML Execution Monitoring

Context

Engineers need to know what's going on with their models and processes as they run (execute) to ensure they're meeting SLA and business obligations. Execution monitoring is just that: it helps modelers react when things go wrong and make proactive improvements.

‍

Teams were building one-off solutions, making the process hard to make repeatable, templatized, and governable.
‍

To be well managed and mitigate risks, Capital One needed to build an enterprise solution that could ensure compliance and scale to other ML pipeline applications, such as model scoring, cost monitoring and system status alerting.

Role

Lead Designer

Team

  • 1 Product Manager
  • 4 Developers (Overseas)
  • 1 UX Designer

TimeLine

6 Months

skills

UX / UI
Information Architecture
Figma

Impact

300+
Model developers in the pilot release of the monitoring dashboard
10
Interviews moderated with model developers and data scientists
5
Workshops led to refine the prototype and data elements
AI/ML Monitoring — Discover 1
AI/ML Monitoring — Discover 2
AI/ML Monitoring — Define 1
AI/ML Monitoring — Define 2
AI/ML Monitoring — Define 3
AI/ML Monitoring — Design 1
AI/ML Monitoring — Design 2
AI/ML Monitoring — Design 3
AI/ML Monitoring — Synthesize 1
AI/ML Monitoring — Synthesize 2
AI/ML Monitoring — Deliver 1
AI/ML Monitoring — Deliver 2
AI/ML Monitoring — Deliver 3

Discover

Objective: Research and develop a UX/UI solution that allows users to view all services and jobs running on Capital Ones machine learning platform, information about their status, the creation of dashboard views and alerts at different scopes.

Research and develop a UX/UI solution that lets users view all services and jobs running on Capital One's machine learning platform, their status, and configurable dashboard views and alerts at different scopes.

‍

Contribution: Translated business needs into project brief, build moderator/user guide, recruited model developers and documented existing constraints.

‍

Result: Aligned on project scope and roadmap with partners, kicked off research, and established design objectives and success metrics around adoption.

Design Process

Leveraged design thinking process to analyze the AI/ML space, empathizing with data modelers, data scientists and engineers on their JTBD and the required UX solution

Problem Statement

This project was started as part of a larger Enterprise wide initiative to standardize the AI/ML process and governance. Defined problem space and context in a pitch deck as a HMW and outlined the projected goal

Project Brief

Worked with cross-functional partners to create project brief, outlining RACI chart, delivery phases, milestones, dependencies and the problem statement. This allowed for alignment and sign-off of objectives

Moderator Guide

Collaborated with research to build a moderator guide to understand user sentiment. Questions included identifying the user role, current state, use cases, opportunities and troubleshooting experiences

Read Details
AI/ML Monitoring — Discover 1
AI/ML Monitoring — Discover 2

Define

Objective: Synthesize collected interviews into prioritized set of recommendations and prototype
‍

Contribution: Moderated and led 10 interviews, distilled information into research report, documented findings and provided  strategic recommendations
‍

Result: Research report was leveraged in prioritization conversations with partners

Read Details
AI/ML Monitoring — Define 1
AI/ML Monitoring — Define 2
AI/ML Monitoring — Define 3

Design

Objective: Create a scalable design and back-end system that allows for incremental roll out of features.
‍

Contribution: Built an object-oriented UX (OOUX) framework and approach to collaborate with product and tech partners. Delivered multiple data requirements and hierarchy of components.
‍

Result: Prioritized a set of features, like anomaly detection, email notifications, rules engine, etc. and the technical data requirements needed to support those features.

Building an Scalable Framework

In this stage of research, I took user quotes and sentiments and translated/mapped them to the features and solutions required to build. These were foundational to the OOUX approach, which included framing content as data, calls to actions, core content and objects

Deconstructing insights into the OOUX framework

Metadata

Defines attributes and properties of objects, offering contextual details to users for better understanding, such as a job’s metadata

Objects

The core components in the system that users interact with. Objects represent tangible or conceptual items, such as a “Job”

Calls to Action

CTAs are linked to objects and influence user decisions, such as restart a job, create an alert, view troubleshooting logs etc.

Core Content

What users come to the platform for, such as job status, completion time, or user historical runs and trends

Existing Requirements

To gather requirements, I looked at existing bespoke solutions across platforms and mapped the metadata, actions and CTAs to make a gap analysis in comparison to our research findings

OOUX Model Framing

Built a OOUX mapping, where the information architecture is mapped to user actions and data points uncovered during interviews. Notes in from the research are tied into the various data elements.

The research report also included the framework and construction of the OOUX model to help document understanding in a clear way with cross functional partners

Read Details
AI/ML Monitoring — Design 1
AI/ML Monitoring — Design 2
AI/ML Monitoring — Design 3

Synthesize

Objective: Test the OOUX framework as a low fidelity design, gather feedback from users on the data elements and information architecture
‍

Contribution: Led 5 workshop sessions with model developers to gather feedback on early prototype design, synthesized findings into refined visual
‍

Result: Developed a clear, shared understanding of the UX/UI design and the data elements it needed to support.

Site Architecture

Leveraged information from OOUX exercise to build a site architecture and connections between data elements. This helped the tech team begin working on building the necessary components

The second version focused on answering some of the questions from engineers; such as the relationship between jobs running on different back-end instances and the associated metadata

Driving Conversations

The low fidelity design was quickly mocked up in Lucidspark, and drove conversations with data scientists and modelers as to different features, such as the starting dashboard, which data elements should be included in a single job run and how job rules would be constructed

Defining Opportunities

This also provided to an opportunity to test future ideas such as a “Profile” section and “Trend and History” which would provide users the ability to configure specific monitoring alerts and see data most pertinent for reporting and regulatory needs

Read Details
AI/ML Monitoring — Synthesize 1
AI/ML Monitoring — Synthesize 2

Deliver

Objective

‍

Refine the design based on feedback and design expertise, moving it into a QA pilot.

‍

  • Built a high fidelity, click through prototype of the experience in Figma. Created a hand off document. Worked with product and tech to prioritize features based on data feasibility.
  • Shipped a limited release of central dashboard for execution monitoring compute jobs for 300+ model developers

High Fidelity Design

Translated the low fidelity flows and OOUX framework into high fidelity screens, aligning the dashboard, job details, alerts, and notification patterns with the existing MLX design system.

Engineering Handoff

Partnered with engineers to walk through the site architecture and rules component, clarifying how job metadata, thresholds, and notifications map to the underlying platform services and back-end instances.

QA & Release

Supported QA validation of the finalized execution monitoring flows ahead of release, confirming job states, alerts, and error handling matched the documented rules and thresholds.

Read Details
AI/ML Monitoring — Deliver 1
AI/ML Monitoring — Deliver 2
AI/ML Monitoring — Deliver 3

Final product

Reflection

I'd usability-test the high-fidelity prototype before the pilot, not just workshop the low-fidelity one.

With developers overseas, I'd set up async design reviews earlier to shorten feedback loops.

Next, I'd use adoption data from the 300+ pilot users to decide which features to build first.