AI/ML Execution Monitoring
Engineers need to know what's going on with their models and processes as they run (execute) to ensure they're meeting SLA and business obligations. Execution monitoring is just that: it helps modelers react when things go wrong and make proactive improvements.
To be well managed and mitigate risks, Capital One needed to build an enterprise solution that could ensure compliance and scale to other ML pipeline applications, such as model scoring, cost monitoring and system status alerting.
Role
Team
- 1 Product Manager
- 4 Developers (Overseas)
- 1 UX Designer
TimeLine
skills
Impact
- Defined scope from business needs, led user research, and developed UX/UI design for an internal AI/ML monitoring tool. Collaborated with product partners to deliver pilot design to 300+ model developers
- Strategically defined experience of multiple features, like AI anomaly detection and automated notifications, for future releases
Discover
Objective: Research and develop a UX/UI solution that allows users to view all services and jobs running on Capital Ones machine learning platform, information about their status, the creation of dashboard views and alerts at different scopes.
Contribution: Translated business needs into project brief, build moderator/user guide, recruited model developers and documented existing constraints.
Result: Aligned on project scope with partners, kicked off research and established design objectives
Design Process
Leveraged design thinking process to analyze the AI/ML space, empathizing with data modelers, data scientists and engineers on their JTBD and the required UX solution
Problem Statement
This project was started as part of a larger Enterprise wide initiative to standardize the AI/ML process and governance. Defined problem space and context in a pitch deck as a HMW and outlined the projected goal
Project Brief
Worked with cross-functional partners to create project brief, outlining RACI chart, delivery phases, milestones, dependencies and the problem statement. This allowed for alignment and sign-off of objectives
Moderator Guide
Collaborated with research to build a moderator guide to understand user sentiment. Questions included identifying the user role, current state, use cases, opportunities and troubleshooting experiences
Define
Objective: Synthesize collected interviews into prioritized set of recommendations and prototype
Contribution: Moderated and led 10 interviews, distilled information into research report, documented findings and provided strategic recommendations
Result: Research report was leveraged in prioritization conversations with partners
User Interviews
Interviews with model developers and data scientist to understand how execution monitoring is done. Tracking included spreadsheets, custom dashboards and exploring troubleshooting logs
Research Report
The research report compiled observations from the 10 interviews, synthesized as user statements, quotes and recommendations. This allowed for conversations with PM partners
Design
Objective: Create a scalable design and back-end system that allows for incremental roll out of features.
Contribution: Built an object-oriented UX (OOUX) framework and approach to collaborate with product and tech partners. Delivered multiple data requirements and hierarchy of components.
Result: Prioritized a set of features, like anomaly detection, email notifications, rules engine, etc. and the technical data requirements needed to support those features.
Building an Scalable Framework
In this stage of research, I took user quotes and sentiments and translated/mapped them to the features and solutions required to build. These were foundational to the OOUX approach, which included framing content as data, calls to actions, core content and objects
Deconstructing insights into the OOUX framework
Metadata
Defines attributes and properties of objects, offering contextual details to users for better understanding, such as a job’s metadata
Objects
The core components in the system that users interact with. Objects represent tangible or conceptual items, such as a “Job”
Calls to Action
CTAs are linked to objects and influence user decisions, such as restart a job, create an alert, view troubleshooting logs etc.
Core Content
What users come to the platform for, such as job status, completion time, or user historical runs and trends
Existing Requirements
To gather requirements, I looked at existing bespoke solutions across platforms and mapped the metadata, actions and CTAs to make a gap analysis in comparison to our research findings
OOUX Model Framing
Built a OOUX mapping, where the information architecture is mapped to user actions and data points uncovered during interviews. Notes in from the research are tied into the various data elements.
Research Report (Continued)
The research report also included the framework and construction of the OOUX model to help document understanding in a clear way with cross functional partners
Synthesize
Objective: Test the OOUX framework as a low fidelity design, gather feedback from users on the data elements and information architecture
Contribution: Led 5 workshop sessions with model developers to gather feedback on early prototype design, synthesized findings into refined visual
Result: Developed clear understanding of UX/UI design and necessary data elements.
Site Architecture
Leveraged information from OOUX exercise to build a site architecture and connections between data elements. This helped the tech team begin working on building the necessary components
The second version focused on answering some of the questions from engineers; such as the relationship between jobs running on different back-end instances and the associated metadata
Driving Conversations
The low fidelity design was quickly mocked up in Lucidspark, and drove conversations with data scientists and modelers as to different features, such as the starting dashboard, which data elements should be included in a single job run and how job rules would be constructed
Defining Opportunities
This also provided to an opportunity to test future ideas such as a “Profile” section and “Trend and History” which would provide users the ability to configure specific monitoring alerts and see data most pertinent for reporting and regulatory needs
Deliver
Refine the design based on feedback and design expertise into QA pilot:
- Built a high fidelity, click through prototype of the experience in Figma. Created a hand off document. Worked with product and tech to prioritize features based on data feasibility.
- Limited release of central dashboard for execution monitoring compute jobs for 300+ model developers
.png)
High Fidelity Design
Translated the low fidelity flows and OOUX framework into high fidelity screens, aligning the dashboard, job details, alerts, and notification patterns with the existing MLX design system.
Engineering Handoff
Partnered with engineers to walk through the site architecture and rules component, clarifying how job metadata, thresholds, and notifications map to the underlying platform services and back-end instances.
QA & Release
Supported QA validation of the finalized execution monitoring flows ahead of release, confirming job states, alerts, and error handling matched the documented rules and thresholds.
.svg)