Page 2 · Jobs
Where I turned messy operations and data into product direction.
Three jobs, from the engine room to the product team: an AI product management internship at IDeaS, an operations internship at Daikin, and a summer keeping tour boats running in Alaska.
AI Chatbot Product Discovery & Alignment
- Product Roadmap
- RAG & Multi-Agent
- Requirements
- Team Alignment
- AI Evaluation
Main project
Business requirements and the AI roadmap
Situation
- At IDeaS I needed to figure out what users would find valuable for an AI chat bot in our revenue management system.
- Align product on the requirements needed from the engineering teams.
- Align the SAS analytics team with the IDeaS AI engineers on the RAG and multi agent architecture.
Task
- Make an AI product roadmap and supporting business requirements documents.
- Lead alignment meetings.
Action
- Read Inspired by Marty Cagan to learn about product discovery and product alignment, read AI Engineering with Foundation Models by Chip Huyen.
- Aligned teams on an evaluation pipeline to increase operational efficiency and quality assurance.
- Analyzed 8000 customer case files along with other user data.
- Made an AI system requirements document and an Analytics requirements document.
- Made a prioritized AI capabilities roadmap.
Results
- The company shifted resources to build the evaluation system.
- The company shifted resources to create human SME data for model context and evaluation.
37 pages
Business requirements document, authored
720 ideas
Product ideas analyzed for the roadmap
9 user stories
Added from my own user research
Studied up
Inspired
Marty Cagan
Product discovery and alignment, used directly at IDeaS.
Self-driven project
Evaluation-driven AI development
Business problem: outdated PM procedures
Step 4Current state
Back to the start every time, by hand
Future state
- PM
- Test cases
- Evaluation rubrics
- Clear acceptance criteria
First step: learn about AI evaluation
Step 4
AI Engineering · Chip Huyen
Best practice:
- Evaluation pipeline
- Prompt engineering
- RAG / agent systems
Solution: team alignment
Step 4Proposal: evaluation pipeline
Solution design recommendations
Jack Callinan · 7-15-2026
PMs
- Evaluation test cases (linked to BRD)
- Rubrics for evaluation metrics (Confluence folder)
Engineers
- AI as a judge (future roadmap)
- System prompt filing
Conservative ROI = 2,000%
*assuming 3 engineers and 1 PM
$300k
Cost savings from the AI evaluation pipeline and PM communication system
2,000%
Conservative ROI (20×), assuming 3 engineers and 1 PM
32 pages
Evaluation-driven development alignment document
What I delivered
2
Business requirements documents.
AI roadmap
Prioritized gen-AI capabilities for development.
8,000
Customer support cases analyzed.
20
User interviews analyzed.