For the last two weeks I’ve been working on an AI video pipeline. It takes a story and turns it into a two minute animated episode.
There’s a harness around it so Claude Code can help with the work that feeds it. The harness is a small MCP server I wrote that wraps my local ComfyUI. One file, no telemetry, no auto-update. It only ever talks to 127.0.0.1. The third party options all carried a lot more surface area than I wanted.
I picked a simple series to build against. A dog and a bird travel around, visit new places, and learn about them. The dog’s name is Pesto. It’s an easy story to define, which makes it a good test bed while the pipeline and the harness are still rough. There are 14 episodes in the repo now.
Everything runs on one machine. An RTX 4080 SUPER with 16 GB of VRAM and 64 GB of RAM. VRAM is the real bottleneck and it shapes most of the decisions below.
Characters Come First
Character generation is the first step, and it matters more than I expected. The character defines the artistic style. Everything downstream inherits it. The world, the scenes, the animation, every episode after this one. Get the character wrong and you’re fighting it in every frame.
So I spend real time here generating characters and styles before anything else moves.
Training LoRAs
Once I have styles I like, I fine tune with LoRAs so I can reproduce the same characters. That’s the whole point. Without them you get a slightly different dog every time you generate a frame.
There’s a ladder here and I use the lightest rung that holds. Reusing one still is free, but you get one pose. Reference conditioning needs no training and gets you multiple poses. A trained LoRA is the best of the three, and it’s what the heroes use.
Each episode now loads two of them. One for the character and one for the show’s look. Each character also gets a trigger word and a locked seed, and per shot the harness applies the LoRA of whichever character the key frame prompt names.
The curation step matters more than the training settings. A few off-model images hurt a character LoRA more than a small dataset does. So the flow is generate a dataset, open the contact sheet, delete everything that isn’t on model, then train.
The Base Model Changed Under Me
I started on Flux and the key frames looked great. Then the licensing caught up with me.
Flux.1-dev is non-commercial. A LoRA inherits the license of whatever generated its training images, so a character trained on Flux output is non-commercial too, and so is every frame rendered with it. That’s the kind of thing that’s cheap to fix early and expensive to fix late.
The fix was a clean-room rebuild on Z-Image-Base, which is Apache-2.0. The captions are text I wrote and carry no license, so I re-rendered the training art from the same captions on the new base, retrained the LoRAs, and re-rendered the whole catalogue. The Flux versions are archived beside the new ones. Every episode spec is on Z-Image now.
The model you pick isn’t only a quality decision.
The Script
Next is the script and the story. I keep it small. Two characters, two minutes, one place they visit.
An episode is a JSON spec. Six scenes, and somewhere between 18 and 37 shots depending on the episode. Each shot carries its key frame prompt, its motion, and the narration line it’s pinned to.
Three checks run before any GPU work happens, and all three are mechanical. One validates the spec. One flags robotic narration and phrases I’ve already used in other episodes. One does the arithmetic on runtime and pacing against a target length.
None of them can tell me whether the story is any good. That reading is Claude’s, and it happens in a Claude Code session rather than through an API key, so nothing leaves the rig. Seven dimensions scored 1 to 5. Hook, escalation, turn, button, voices, reads-aloud, something-to-see. A 3 is the bar the shipped episodes already hit.
Every score has to be attached to the work. It cites a line that literally appears in the narration, or a shot number that exists. Anything scored 3 or below has to name one concrete fix. A verdict that quotes a line nobody wrote gets rejected instead of stored. It’s easy to get a model to hand you a confident score. Getting it to point at something real took the rest of the work.
The Audio Sets The Clock
The narration gets synthesized before anything renders, and it decides the timing for everything else.
Every line for the whole film is synthesized up front. Then each line is measured, and the shot pinned to it is planned from that measurement. The animator is asked for exactly that many frames. Runtime is the narration, and a shot’s max length is only a preference. When a gap outruns every shot’s max, the shots stretch rather than the narration running past the picture.
That’s been the cleanest part of the design. No guessing at scene durations. The house voice is Chatterbox now, and the TTS is pinned to CPU so it can never take VRAM from the model rendering the video.
Key Frames
Key frames come out of Z-Image. One image per shot.
I review them on a contact sheet and against the scenes. When I’m away from my desk Claude publishes the review page to my phone as an artifact, which has been more useful than I expected.
Sometimes you get inconsistencies with the world or the characters. This is where the LoRAs matter most. They’re what lets me generate the same output again and again. I can also render a few candidates per shot and pick one without re-rendering, since picking copies the approved file and writes its seed back into the spec.
Motion
Once the key frames look right I apply motion to each one. This is the heavy GPU stage. LTX-2.5 animates each key frame in the style of its scene, and the motion prompt comes from the script.
The image model and the motion model can’t both be resident, so the render runs in two phases. Key frames on one ComfyUI instance, swap, motion on another. Not elegant, but it’s the constraint I actually have.
Every render writes a manifest with the prompt, seed, LoRA, motion level, frame count and realized duration for each shot. When a long render dies partway, resume reuses a clip only when all of that still matches. Anything changed re-renders. The failure direction is a wasted minute of GPU, never a stale shot.
Stitching
The last stage is a Python script. It stitches everything together, adds bookends for the intro and the closing, and lays a music background over the whole thing. The music comes from ACE-Step, generated per episode.
Four Kinds Of Model
No single model does any of this. Each stage needs its own kind.
Z-Image makes the key frames. LTX-2.5 makes the motion. Chatterbox reads the narration. ACE-Step writes the music. I write the script, and Claude judges it against the rubric.
Then the last stage isn’t a model at all. Stitching the video files together and applying the bookends is plain Python.
The Constraint Is The Point
Everything above runs on one machine with 16 GB of VRAM. That’s well below what the frontier video models can do, and for a while I treated it as the thing holding the project back.
It’s the opposite. AI video takes a lot of testing, and metered pricing makes the testing the expensive part.
A five second clip costs me about half a cent in electricity. The same clip is around $0.35 on Kling, $0.60 on Runway, and $2 on Veo. An episode is 18 to 37 shots. So one full render is under 20 cents locally, and somewhere between $6 and $75 on an API depending on the model.
But I don’t render an episode once. Here’s what the last two weeks looked like.
When I moved the base model I re-rendered the whole catalogue. 14 episodes, a few hundred clips, about seven GPU hours and nothing else.
One character deformed in five of his seven shots. I re-rendered him twice at different motion settings, both failed, so I cut him from the story and replaced him with a character shaped better for the animator. Three passes to learn one thing about design.
And my favorite one. Pesto’s paw kept morphing into a hand, so I added terms to the negative prompt and re-rendered at the same seed. The new file was byte identical to the old one. The sampler runs at a CFG where negative conditioning gets discarded entirely, so every negative prompt in that path had been dead code the whole time, including the anti-warp block that predated the bug. Nothing errored. The graph has a negative input, ComfyUI encodes the text, the sampler ignores it.
On a metered API I’d have looked at the new output, decided the fix probably helped, and closed the investigation. An inert knob doesn’t just fail to fix the defect, it feels like a fix and gets committed. Re-rendering to diff the bytes is what caught it, and that test is only free because the GPU is on my desk.
So the constraint bought the thing I actually needed. Unlimited attempts while the mechanics are still wrong.
What Swaps And What Doesn’t
The bet is that the pipeline outlives whatever model is best this month. I’ve tested that once now, moving off Flux, and it mostly held.
It took about a day. The harness didn’t change at all. The spec schema, the audio-led timing, the coverage roles, the narration, the assembly, the finish stage, all of it carried over untouched.
The rendering layer did not. That’s a per-model adapter and it got rebuilt. Different trainer, different LoRA format, different splice point in the graph, new workflow. So swapping the base is a day of real work, not a settings change. Worth budgeting it that way.
One thing didn’t transfer at all and it surprised me. Identity carried fine, since that’s what the LoRA holds. Prompt dialect didn’t. The same insert prompt that made Flux put a named object front and center made the new base weight the whole scene evenly, so the object ended up as background clutter. A LoRA carries who. Nothing carries how a model reads a sentence.
The split I trust now is that local testing locks the creative decisions, and those transfer. Composition, camera, motion beats, timing. What local can’t do is predict the final look, because that part is model-specific. I spent a while misusing local renders to preview quality when their real job is to direct.
Which makes the plan pretty simple. Compose locally for free, and render finals on whatever’s best that week once a story is worth it.
What I Underestimated
I thought the hard part would be motion, since that’s where all the GPU time goes. It’s the characters and the script. Both of those decide what every stage after them can be, and neither one gets fixed later.
The licensing was the other one. I had a working character and a catalogue of episodes rendered on a base I couldn’t ship, and the only real fix was to rebuild from captions on a clean base. Worth checking before the LoRA, not after.