Research proposal · Intelligent Robot Learning Lab · University of Alberta

Owner in the loop: learning a business owner's reward when he is only sometimes watching

My vending robots, a vision-based store and a team of LLM agents run seven small businesses. Each needs my approval, and I am rarely available. How can these agents learn my reward from ratings and corrections, measure how aligned they are with me, and decide what to do while I am not watching?

Why this lab, why me

The lab's questions, my research line, and the work that brought me here

For 20 years I have built businesses that must keep running when I am not there: a distribution company, rental properties, vending machines, and now AI agents. Every one of them hit the same limit: decisions waited for me. That is the problem this lab studies from the other side, how an agent learns from a human who teaches it, and how much it can do alone.

What the IRL Lab works on

  • Learning rewards from human ratings and preferences (R4, 2026)
  • Measuring alignment with a stakeholder (TAC, 2025)
  • Agents that do not always observe the reward (Monitored MDPs, 2025)
  • Foundation models and LLM decision-making with RL
  • Human-AI teams and robots learning in the real world

My research line

  • Agents that learn their owner's way of deciding from approvals, ratings and corrections
  • Error measured as "the agent decided differently from the owner"
  • Knowing when to act alone and when to ask, while the owner is away
  • Keeping the owner's profile when someone else approves
  • Tested on robots in the field and LLM agents

What I have built

  • An LED distribution company grown from my living room to about 200 employees, with quality-inspection offices in Hong Kong and Shenzhen
  • SAP data-migration processes for large manufacturing go-lives
  • Rental and short-term rental properties in Mexico, operated remotely
  • Vending robots in Ontario, a vision store, and seven businesses run by AI agents
2007 – 2012

Data migration engineer. Designed SAP data-migration processes (legacy extraction, cleansing and transformation rules, SQL) for manufacturing plants.

2012 – 2020

Founder, LED lighting distributor. Grew it to about 200 employees and US$6–7M in yearly sales, with offices across Mexico and in China. Learned how hard it is to delegate decisions.

2020 – 2022

Real estate investor. Built and ran rental and short-term rental properties, managed at a distance.

2022 – today

Canada. Vending robots, AI agents for content, SEO, investing and sourcing, a cashierless vision store, and Applied AI & Machine Learning at Conestoga College.

A deployed testbed

Robots and agents already working in the field, with one human teacher

Most human-in-the-loop learning is tested in simulation or in short lab studies. This testbed runs every day, earns real money, and is taught by one real owner whose availability is limited and measurable.

Embodied agents

4 vending robots

Deployed at a clinic, an auto-parts yard, a hotel and a car dealership, with sales telemetry. The owner corrects every restock plan; his rules cover his time and driving distance, which the textbook optimum ignores.

The robot, virtualized

A vision-based store

GrabGo turns a shelf into a store that charges from images: a shelf camera, a customer's phone photo, or both. The owner's privacy and cost trade-offs decide the setup at each location.

LLM agents

Content, SEO, investing, sourcing

Language-model agents propose posts, site changes, market theses and supplier decisions. Every approval, rejection and correction with its reason is logged.

Real decision log
9 of 24

site changes from the SEO agent ended their first week waiting for the owner. Waiting, not error, was the main cost.

A second teacher

Specifiers

Other people sometimes approve on the owner's behalf. Their answers are a second, misaligned teacher: in simulation, learning from them replaces the owner's profile without anyone noticing.

A limited monitor

A full-time student owner

The owner is in class most weekdays. His availability is a real, scheduled monitor: feedback exists only when he is watching.

Where this meets the IRL Lab

One testbed, several of the lab's current questions

I read the lab's recent work and the research interests listed on its team page. These are the points where my data and my operation could be useful, member by member.

Matthew E. TaylorDirector · human-in-the-loop RL, multi-agent systems, roboticsArgues that RL is fundamentally human in the loop, and studies how agents learn from advice, demonstrations and feedback in the real world.Offer: a long-running, real deployment where the human is the bottleneck, to test HITL methods outside the lab.
Calarina MuslimaniPhD · reward alignment, HITL RLTrajectory Alignment Coefficient: agreement between a stakeholder's preferences and a reward's rankings.Offer: TAC measured against a real owner's choices across several businesses and over months, as his situation changes. Used in Experiment 1.
Chaitanya KharyalPhD · vision, robotics, human decision-makingR4: learning rewards from ordinal ratings (bad, neutral, good) with a ranking mean squared error.Offer: real ordinal ratings from an owner, plus a vision robot (the store) where ratings come from images. Experiment 1 tests the rating idea on business decisions.
Sheila Schoepp · Yingyue CaoPhD / MSc · foundation models with RL, LLM decision-makingIntegrating LLMs and vision-language models with RL; using RL to improve LLM decisions.Offer: LLM agents that already make real business decisions and need a learned reward from their owner instead of hand-written rules.
Ethan Towsley · Marzieh GhayourMSc · human-AI teams, adaptive systemsMetrics and rewards for human-AI team collaboration; RL for decision-making in real-world settings.Offer: a real owner + specifier + agents team, with logged hand-offs and the cost of each wait.
Sahand Somi · Matthew SchlegelMSc / Postdoc · RL in industry, explainable HITL RLRL applied to construction and to power systems.Offer: a new applied domain for the lab, automated retail (vending robots and cashierless stores), run by someone who has operated companies for 25 years.
Alireza KazemipourMonitored MDPs (with Parisi, Taylor and Bowling)Agents that do not always observe the reward, for instance because a human observer is not always available.Offer: a natural Mon-MDP where the monitor is the owner's real schedule. Experiment 2.

Experiment 1 · building on R4 and TAC

Ratings or preferences: which teaches an agent the owner's reward?

The owner reviews spending proposals. With a preference, he picks the better of two proposals. With a rating, he labels one proposal bad, neutral or good. Both cost him one answer. I fit a linear reward from each kind of feedback (a ranking-MSE loss in the spirit of R4 for ratings, Bradley–Terry for preferences) and score it two ways: alignment with the owner (Kendall τ-b, as in TAC) and the share of proposals the agent would approve or reject differently from him.

40
Ratings (bad · neutral · good)
Alignment 0.90 · decides differently 8.1%
Preferences (A or B)
Alignment 0.86 · decides differently 19.1%

Alignment = Kendall τ-b between the owner's true utility and the learned reward on 600 new proposals (1 = same ranking). 20 runs per point.

Share of new proposals where the agent's approve/reject differs from the owner's. This owner approves about a third of proposals.

1

Both kinds of feedback learn the owner's ranking well; preferences catch up with ratings after about 160 answers. But preferences never tell the agent where the owner's bar is: even with 160 answers the agent still approves or rejects about 19% of proposals differently from him, against about 5% with ratings. For an agent that must decide whether to act, ordinal ratings carry the information that matters. This supports R4's motivation in a non-robotic, money-valued domain.

Experiment 2 · the owner as a monitor

What should the agent do while the owner is not watching?

Decisions arrive every hour, because vending robots and an online store never close. The owner is available a few hours a week. Only then does he review the queue (up to six items an hour) and rate them; the agent refits its reward every week from the ratings it has seen. Three policies: always wait for him, always act alone, or act alone only when the learned reward is confident and the action is reversible.

20 h
Always wait
Average wait 290 h
Errors 0 per 100
Always act alone
Average wait 0 h
Errors 9.3 per 100
Alone if confident and reversible
Average wait 65 h
Errors 0.35 per 100 · alone 25%

Average hours a decision waits for the owner (log scale), 12 weeks, 8 runs per point. Below about 30 hours a week the queue never clears when the agent always waits.

2

When the owner is the monitor, waiting is the dominant cost, not error. Acting alone on everything makes about 7 to 13 mistakes per 100 decisions. Acting alone only on confident, reversible decisions (about a quarter of them) keeps mistakes near 0.3 per 100 and, at 20 owner-hours a week, cuts the average wait from about 290 to 65 hours. Choosing which decisions to keep for the owner, and when to ask him, is exactly the open question of learning under a monitor.

Proposed MSc thesis

Learning an owner's reward from ratings and corrections under a monitored, misaligned-teacher setting

  1. Rating-based reward learning on a deployed system. Does R4-style learning from ordinal ratings, plus corrections with reasons, reach the owner's decision bar with fewer owner answers than preferences, on vending, store and LLM-agent decisions?
  2. The owner as a monitor. Framing the owner's schedule as the monitor of a Mon-MDP, which decisions should the agent act on alone, which should it queue, and when should it ask?
  3. Alignment that drifts and teachers that disagree. Using TAC over months, how fast does alignment decay when the owner's situation changes, and how should an agent weigh a specifier's ratings against the owner's?

What I bring to the lab

Testbed

A live system the lab could study without building it: robots in the field, a vision store, LLM agents, logs and a real teacher.

Experience

25 years founding and operating companies (up to about 200 employees, offices in Mexico and China), plus SAP data-migration engineering. I can run an applied project end to end.

Flexibility

I am open to a funded position or an unfunded research collaboration.

Both experiments use a simulated owner built from the portfolio's decision profile (case 01); the comparisons, not the absolute numbers, are the result. Code: github.com/antoniosg2025-stack/owner-in-the-loop

Projects I am working on

Seven businesses, seven case studies

Each case is a real operation with its own notebook and one main finding. Together they are the evidence behind this proposal. The full interactive portfolio is at research.antonio-sainz.com, and the code is on GitHub.

References

  1. Kharyal, C., Muslimani, C., & Taylor, M. E. (2026). Reward Learning through Ranking Mean Squared Error. ICML.
  2. Muslimani, C., Johnstonbaugh, K., Chandramouli, S., Booth, S., Knox, W. B., & Taylor, M. E. (2025). Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners. Reinforcement Learning Journal.
  3. Kazemipour, A., Parisi, S., Taylor, M. E., & Bowling, M. (2025). Model-Based Exploration in Monitored Markov Decision Processes. ICML.
  4. Muslimani, C., & Taylor, M. E. (2025). Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning. ICLR.
  5. Arabneydi, J., Islam, S., Das, S., et al., Taylor, M. E., Guzdial, M. (2025). A Systematic Approach to Design Real-World Human-in-the-Loop Deep Reinforcement Learning.
  6. Baraka, K., et al., Taylor, M., et al. (2026). Human-Interactive Robot Learning: Definition, Challenges, and Recommendations. ACM Transactions on Human-Robot Interaction.
  7. Taylor, M. E. (2023). Reinforcement Learning Requires Human-in-the-Loop Framing and Approaches. HHAI.