Building an AI Video Editor Agent: Architecture Lessons from Production UX Work

TL;DR

• We explored an agentic AI video editor — users describe intent ("make it punchier," "add a hook"), the agent edits, user approves or refines. No default timeline. See the full case study.
• Architecture = three nested loops: Ingest (understand footage) → Propose (first cut + stated intent) → Refine (natural-language iterations with changelog).
• Agentic beats assistive when iteration speed is the bottleneck — assistive tools help with one step; agents own the outcome until you reject it.
• Tool orchestration maps user intents to atomic edit operations (trim, restyle, caption, music) — LLM plans, deterministic tools execute, preview renders synchronously where possible.
• Shipping agents in 2026? Nova Sidera builds agentic product UX from €4,500 fixed-scope — €50 architecture audit at /consultation/ if you want feedback before hiring.

What Problem Were We Solving?

Short-form mobile video is everywhere — Reels, TikTok, YouTube Shorts — but editing a 30-second clip still takes 30 minutes in traditional tools. Creators bounce between apps, abandon drafts, and ship inconsistent quality.

Existing "AI video" features are assistive, not agentic:

• Auto-captions — one step.
• Beat sync — one step.
• Background removal — one step.

The user still operates a timeline, still chooses transitions, still exports manually. Responsibility stays with the user.

Our design question: What if the agent takes initiative — and you only approve?

Discovery work mapped real creator conversations. Messages like "shorter," "remove the boring part," "try a different vibe," "add a hook at the start" are not editor commands — they are agent intents. That reframing drove every architectural decision. Full UX exploration: AI Video Editor case study.

What Is the Difference Between Agentic and Assistive AI?

Assistive is safer to ship — less can go wrong visibly. Agentic wins when cognitive load and iteration count hurt more than occasional wrong proposals.

Not every product should be agentic. If your users are professional editors who want precision, keep the timeline. If your users are creators who hate editing, agentic is the wedge.

How Does the Three-Loop Architecture Work?

We structured the agent as three nested loops — each loop has clear inputs, outputs, and failure handling.

Loop 1 — Ingest

Goal: Build a structured model of raw footage before any edit decision.

Ingest runs once per upload — cached aggressively. Re-ingesting on every refine request destroys latency.

Technical note: combine ffmpeg probes for deterministic metadata with ML classifiers for semantic tags. Do not ask the LLM to guess timestamps — tools measure, models interpret.

Loop 2 — Propose

Goal: Produce a first cut plus a stated intent the user can accept or reject.

The planner reads ingest artifacts and a style brief (platform: Reels, tone: energetic, target length: 25s). It outputs:

1. Edit decision list (EDL) — ordered atomic operations.
2. Natural-language summary — "I cut two pauses, added a hook from your second take, synced to the beat drop at 0:04."
3. Preview render — watchable video, not a storyboard.

Propose is where agentic UX lives. The user sees result + explanation together — transparency builds trust for autonomous decisions.

Loop 3 — Refine

Goal: Map natural-language feedback to new EDL deltas.

User: "Shorter opener, keep the joke at the end."

Refine loop:

1. Intent parser — classifies request against ~30 atomic intents (trim, reorder, replace music, restyle captions, etc.).
2. Delta planner — modifies EDL, does not regenerate from scratch unless needed.
3. Changelog — "Removed 1.2s from shot 1; joke clip unchanged."
4. Re-render — incremental preview update.

Refine loops must be fast — target under 10 seconds for mobile UX. If render takes 60 seconds, the conversation dies.

Upload → [Ingest] → artifacts cache
                         ↓
              User opens project → [Propose] → preview v1
                         ↓
              User message → [Refine] → preview v2…vN
                         ↓
              Export → final render queue

How Should You Orchestrate Tools and the LLM?

The LLM is a planner, not an editor. Never let it output pixel data or cut timestamps without tool validation.

Atomic edit tools

Each tool is deterministic, testable, idempotent where possible:

The LLM selects tools and parameters; code executes and validates bounds (no negative durations, no shots outside source range).

Patterns from OpenAI function calling and Anthropic tool use apply directly — schema strict, reject malformed calls, retry with error context.

State management

Maintain:

• EDL version history — every refine creates a revertible snapshot.
• Artifact store — ingest outputs, intermediate renders, final export.
• Conversation thread — user messages linked to EDL versions for auditability.

When the user says "undo," revert EDL pointer — do not re-run LLM unless necessary.

Model selection

• Planner / intent parser: capable multimodal or strong text model with tool support.
• Ingest classifiers: specialized models or APIs (transcription, scene detection) — cheaper and more reliable than one giant model call.
• Do not multimodal-plan from raw video every turn — cost and latency explode. Plan from ingest artifacts.

Reference architecture aligns with emerging agent orchestration patterns described in LangGraph and production writeups from teams shipping coding agents — stateful graphs, explicit nodes, human-in-the-loop on propose/refine boundaries.

What Mobile UX Patterns Actually Work?

Our case study collapsed the UI to two surfaces:

1. Preview — always shows latest cut, full-screen friendly.
2. Conversation — thread where user nudges agent; every agent action is reversible by replying.

Optional third surface for longer projects: Shots board — agent-maintained, user marks "must keep" / "cut" without manual timeline editing.

UX rules we validated

• Show changelog with preview — never silent edits.
• Suggest reply chips — "Shorter," "Different music," "More energy" reduce typing friction.
• Progress for render — honest seconds remaining; fake spinners erode trust.
• Export is explicit — agent does not auto-post; user confirms resolution and destination.

Accessibility: preview controls need keyboard and VoiceOver labels; conversation chips are buttons, not divs — same WCAG 2.1 AA baseline as any consumer app facing the EU market.

What Failure Modes Should You Plan For?

Agentic systems fail differently from CRUD apps.

Human-in-the-loop boundary: first Propose requires tap-to-accept. Refine can be more autonomous once trust establishes — tunable per product.

Log every planner decision with EDL diff for debugging. When users report "it ruined my video," you need replay, not vibes.

How Should Founders Ship an Agent Like This?

Practical sequence if you are building in 2026:

1. Concierge MVP — human editor behind chat UI validates intents before you automate tools. One week, cheap learning.
2. Automate Ingest + one Refine intent — e.g., trim only. Ship narrow.
3. Add Propose loop — first cut template-driven before full planner freedom.
4. Expand atomic tools — caption styles, music, hooks — one per sprint.
5. Performance pass — incremental render, edge cache, preview resolution tiers.

Budget realism: a focused agent prototype (one platform, 5–8 intents, no export to every social network) fits €15,000–€30,000 fixed-scope with a nearshore team. Full production mobile app with billing, teams, and 4K export sits higher — often T&M at €55–€95/hr or dedicated team from €6,500/month once scope expands.

Read our pricing models guide before you sign — agent projects almost always start fixed for v1 prototype, then switch to T&M for tool expansion.

What Should You Do Next?

Agentic video is one domain — the architecture patterns transfer to support agents, data copilots, devtools, creative suites anywhere users iterate in language and expect visible outcomes fast.

If you are designing something similar:

• Study the AI Video Editor case study for UX flows.
• Map your domain's atomic intents before you pick models.
• Book €50 consultation at /consultation/ — we review agent architecture, latency risks, and scope; fee credited toward build.
• Fixed-scope agent prototypes from €4,500 — see pricing or contact us.

The timeline is not coming back for most creator tools. The agents that win will be the ones that take responsibility for outcomes — with changelogs, rollback, and previews fast enough to keep the conversation alive.

Nova Sidera — agentic UX, web, mobile, AI integrations from Montenegro. Projects from €4,500 · €50 audit at /consultation/