AI/ML Execution Monitoring
Engineers need to know what's going on with their models and processes as they run (execute) to ensure they're meeting SLA and business obligations. Execution monitoring is just that: it helps modelers react when things go wrong and make proactive improvements.
Teams were building one-off solutions, making the process hard to make repeatable, templatized, and governable.
To be well managed and mitigate risks, Capital One needed to build an enterprise solution that could ensure compliance and scale to other ML pipeline applications, such as model scoring, cost monitoring and system status alerting.
Role
Team
- 1 Product Manager
- 4 Developers (Overseas)
- 1 UX Designer
TimeLine
skills
Impact
Discover
Objective: Research and develop a UX/UI solution that allows users to view all services and jobs running on Capital Ones machine learning platform, information about their status, the creation of dashboard views and alerts at different scopes.
Research and develop a UX/UI solution that lets users view all services and jobs running on Capital One's machine learning platform, their status, and configurable dashboard views and alerts at different scopes.
Contribution: Translated business needs into project brief, build moderator/user guide, recruited model developers and documented existing constraints.
Result: Aligned on project scope and roadmap with partners, kicked off research, and established design objectives and success metrics around adoption.
Design Process
Leveraged design thinking process to analyze the AI/ML space, empathizing with data modelers, data scientists and engineers on their JTBD and the required UX solution
Problem Statement
This project was started as part of a larger Enterprise wide initiative to standardize the AI/ML process and governance. Defined problem space and context in a pitch deck as a HMW and outlined the projected goal
Project Brief
Worked with cross-functional partners to create project brief, outlining RACI chart, delivery phases, milestones, dependencies and the problem statement. This allowed for alignment and sign-off of objectives
Moderator Guide
Collaborated with research to build a moderator guide to understand user sentiment. Questions included identifying the user role, current state, use cases, opportunities and troubleshooting experiences


Define
Objective: Synthesize collected interviews into prioritized set of recommendations and prototype
Contribution: Moderated and led 10 interviews, distilled information into research report, documented findings and provided strategic recommendations
Result: Research report was leveraged in prioritization conversations with partners



Design
Objective: Create a scalable design and back-end system that allows for incremental roll out of features.
Contribution: Built an object-oriented UX (OOUX) framework and approach to collaborate with product and tech partners. Delivered multiple data requirements and hierarchy of components.
Result: Prioritized a set of features, like anomaly detection, email notifications, rules engine, etc. and the technical data requirements needed to support those features.
Building an Scalable Framework
In this stage of research, I took user quotes and sentiments and translated/mapped them to the features and solutions required to build. These were foundational to the OOUX approach, which included framing content as data, calls to actions, core content and objects
Deconstructing insights into the OOUX framework
Metadata
Defines attributes and properties of objects, offering contextual details to users for better understanding, such as a job’s metadata
Objects
The core components in the system that users interact with. Objects represent tangible or conceptual items, such as a “Job”
Calls to Action
CTAs are linked to objects and influence user decisions, such as restart a job, create an alert, view troubleshooting logs etc.
Core Content
What users come to the platform for, such as job status, completion time, or user historical runs and trends
Existing Requirements
To gather requirements, I looked at existing bespoke solutions across platforms and mapped the metadata, actions and CTAs to make a gap analysis in comparison to our research findings
OOUX Model Framing
Built a OOUX mapping, where the information architecture is mapped to user actions and data points uncovered during interviews. Notes in from the research are tied into the various data elements.
The research report also included the framework and construction of the OOUX model to help document understanding in a clear way with cross functional partners



Synthesize
Objective: Test the OOUX framework as a low fidelity design, gather feedback from users on the data elements and information architecture
Contribution: Led 5 workshop sessions with model developers to gather feedback on early prototype design, synthesized findings into refined visual
Result: Developed a clear, shared understanding of the UX/UI design and the data elements it needed to support.
Site Architecture
Leveraged information from OOUX exercise to build a site architecture and connections between data elements. This helped the tech team begin working on building the necessary components
The second version focused on answering some of the questions from engineers; such as the relationship between jobs running on different back-end instances and the associated metadata
Driving Conversations
The low fidelity design was quickly mocked up in Lucidspark, and drove conversations with data scientists and modelers as to different features, such as the starting dashboard, which data elements should be included in a single job run and how job rules would be constructed
Defining Opportunities
This also provided to an opportunity to test future ideas such as a “Profile” section and “Trend and History” which would provide users the ability to configure specific monitoring alerts and see data most pertinent for reporting and regulatory needs


Deliver
Objective
Refine the design based on feedback and design expertise, moving it into a QA pilot.
- Built a high fidelity, click through prototype of the experience in Figma. Created a hand off document. Worked with product and tech to prioritize features based on data feasibility.
- Shipped a limited release of central dashboard for execution monitoring compute jobs for 300+ model developers
High Fidelity Design
Translated the low fidelity flows and OOUX framework into high fidelity screens, aligning the dashboard, job details, alerts, and notification patterns with the existing MLX design system.
Engineering Handoff
Partnered with engineers to walk through the site architecture and rules component, clarifying how job metadata, thresholds, and notifications map to the underlying platform services and back-end instances.
QA & Release
Supported QA validation of the finalized execution monitoring flows ahead of release, confirming job states, alerts, and error handling matched the documented rules and thresholds.



Final product
Reflection
I'd usability-test the high-fidelity prototype before the pilot, not just workshop the low-fidelity one.
With developers overseas, I'd set up async design reviews earlier to shorten feedback loops.
Next, I'd use adoption data from the 300+ pilot users to decide which features to build first.