Bulk Listing Video Generation at Scale

The Problem With Bulk

A brokerage signs up, connects their inventory, and expects videos for all 380 active listings. A photographer finishes a weekend of shoots and wants forty properties rendered by Monday. An MLS wants every new listing processed within an hour of going live. These are the requests that turn a working integration into a production system.

The naive approach — loop over listings, call the API for each photo, wait — fails in three predictable ways. It consumes your entire credit balance before the first customer sees a result. It saturates your worker capacity so interactive requests queue behind a batch nobody is watching. And when a deploy restarts the process halfway through, it has no idea what it already paid for.

All three are solvable with the same three ingredients: a credit budget computed before you start, a bounded worker pool, and per-listing state in your own database. This guide walks through each. It assumes you already understand the job lifecycle covered in our async jobs and polling guide.

Budget the Batch Before You Start It

The credit math is simple and worth internalizing, because it determines what you can honestly promise a customer. Each photo becomes one clip at one credit. Each listing needs one additional credit to merge those clips into a reel. So a listing costs photos + 1 credits.

The practical consequence: a brokerage with 380 listings averaging eight photos needs roughly 3,420 credits. That does not fit in a single month of any standard plan, which means the honest answer is a phased rollout, a photo cap per listing, or a custom volume plan. Deciding that before the import starts is far better than discovering it at listing 94.

const API = "https://app.photoaivideo.com/api/v1"
const HEADERS = { "X-API-Key": process.env.PHOTOAIVIDEO_API_KEY }

// Every listing costs (photos + 1): one credit per clip, one for the merge.
function batchCost(listings) {
  return listings.reduce((sum, l) => sum + l.photoUrls.length + 1, 0)
}

async function preflight(listings, { reserve = 50 } = {}) {
  const { remaining } = await fetch(API + "/credits", { headers: HEADERS }).then((r) => r.json())
  const needed = batchCost(listings)

  // Hold back a reserve so live customer requests keep working during the import.
  const usable = Math.max(0, remaining - reserve)
  if (needed > usable) {
    throw new Error(
      "Batch needs " + needed + " credits, only " + usable + " usable (balance " + remaining + ").",
    )
  }

  return { needed, remaining }
}

Pre-Flight Checklist

A Bounded Worker Pool

The core of a bulk pipeline is a pool that processes a fixed number of listings concurrently and no more. Six to ten is a reasonable starting point. The goal is steady throughput you can reason about, not maximum parallelism.

// Process listings N-at-a-time instead of all at once.
async function drainBatch(batchId, listings, concurrency = 6) {
  const queue = [...listings]
  let completed = 0

  async function worker() {
    while (queue.length > 0) {
      const listing = queue.shift()
      if (!listing) return

      try {
        await processListing(batchId, listing)
        completed++
      } catch (err) {
        // One bad listing must never kill the batch.
        await db.batchItems.update(
          { batchId, listingId: listing.id },
          { status: "failed", error: String(err) },
        )
      }

      await db.batches.update({ id: batchId }, { completed })
    }
  }

  // Exactly `concurrency` workers share the queue.
  await Promise.all(Array.from({ length: concurrency }, worker))
}

Two properties make this production-safe. The try/catch inside the worker means one unreachable photo URL marks a single listing failed instead of aborting all 400. And the running completed counter turns progress into a database query, which is what lets you show a real progress bar instead of a spinner.

Making the Batch Resumable

Long batches outlive processes. A deploy, a scale-down, or an unhandled exception will interrupt a three-hour import, and the only thing standing between that and a doubled bill is per-listing state you wrote down before spending the credit.

// Every listing in a batch gets a row BEFORE any credit is spent.
async function startBatch(customerId, listings) {
  await preflight(listings)

  const batch = await db.batches.insert({
    customerId,
    total: listings.length,
    completed: 0,
    estimatedCredits: batchCost(listings),
    status: "running",
  })

  await db.batchItems.insertMany(
    listings.map((l) => ({
      batchId: batch.id,
      listingId: l.id,
      status: "queued",   // queued -> rendering -> merged | failed
    })),
  )

  await queue.enqueue("drain-batch", { batchId: batch.id })
  return batch
}

// On restart, pick up only what never finished.
async function resumeBatch(batchId) {
  const pending = await db.batchItems.find({
    batchId,
    status: ["queued", "rendering"],   // terminal rows are skipped
  })

  const listings = await loadListings(pending.map((p) => p.listingId))
  await drainBatch(batchId, listings)
}

The distinction that matters is between queued and rendering. A queued listing has cost you nothing and can be started fresh. A rendering listing already has jobIds attached and paid-for clips in flight — so resuming it means polling those existing jobs, not submitting new ones. Conflating the two is how teams accidentally pay twice for the same listing.

Processing a Single Listing

Within one listing, start every clip concurrently, wait for all of them, then merge. Remember the twenty clip ceiling on a reel — a listing with more photos than that needs to be truncated or split into multiple reels.

const MAX_CLIPS_PER_REEL = 20

async function processListing(batchId, listing) {
  const photos = listing.photoUrls.slice(0, MAX_CLIPS_PER_REEL)

  await db.batchItems.update(
    { batchId, listingId: listing.id },
    { status: "rendering" },
  )

  // Start all clips at once — wall-clock time is the slowest clip, not the sum.
  const jobs = await Promise.all(
    photos.map((imageUrl) =>
      fetch(API + "/videos", {
        method: "POST",
        headers: { ...HEADERS, "Content-Type": "application/json" },
        body: JSON.stringify({ imageUrl, effect: "orbit_right", duration: 5 }),
      }).then((r) => r.json()),
    ),
  )

  // Persist jobIds immediately so a crash here is recoverable, not a lost charge.
  await db.batchItems.update(
    { batchId, listingId: listing.id },
    { videoJobIds: jobs.map((j) => j.jobId) },
  )

  const settled = await Promise.allSettled(jobs.map((j) => waitForVideo(j.jobId)))
  const ready = jobs.filter((_, i) => settled[i].status === "fulfilled")

  // A reel needs at least 2 clips. Below that, there is nothing to merge.
  if (ready.length < 2) {
    throw new Error("Only " + ready.length + " clip(s) succeeded for listing " + listing.id)
  }

  const reel = await fetch(API + "/reels", {
    method: "POST",
    headers: { ...HEADERS, "Content-Type": "application/json" },
    body: JSON.stringify({ videoJobIds: ready.map((j) => j.jobId) }),
  }).then((r) => r.json())

  const finished = await waitForReel(reel.jobId)

  await db.batchItems.update(
    { batchId, listingId: listing.id },
    { status: "merged", reelUrl: await copyToOwnStorage(finished.reelUrl) },
  )
}

Using Promise.allSettled rather than Promise.all is deliberate. If one photo in a twelve-photo listing fails to render, you still have eleven good clips worth merging. With Promise.all, that single failure would discard eleven credits you have already spent.

Per-Tenant Fairness

If you serve multiple customers from one API account, a global worker pool is not enough. One brokerage importing their full inventory will occupy every slot, and every other customer will experience your product as broken. The fix is a per-tenant cap layered on top of the global one.

const GLOBAL_CONCURRENCY = 10
const PER_TENANT_CONCURRENCY = 3

async function claimSlot(customerId) {
  const [globalActive, tenantActive] = await Promise.all([
    db.batchItems.count({ status: "rendering" }),
    db.batchItems.count({ status: "rendering", customerId }),
  ])

  if (globalActive >= GLOBAL_CONCURRENCY) return false
  if (tenantActive >= PER_TENANT_CONCURRENCY) return false
  return true
}

With those numbers, a single tenant can never use more than three of ten slots. Their large import still completes, just steadily, while seven slots remain available for everyone else. This is the difference between a platform that degrades gracefully and one where your biggest customer's onboarding looks like an outage to everybody else.

Five Failure Modes and Their Fixes

What to Show the Customer

A bulk import that takes three hours needs a status surface, or your support inbox becomes the status surface. Because you are writing per-listing rows, you already have everything required: a total, a completed count, and a per-listing state.

Show the count finished against the total, list the individual listings that failed with a readable reason, and let each finished listing be previewed the moment it is ready rather than making the customer wait for the whole batch. Videos that arrive progressively feel fast. An import that reveals nothing for three hours and then dumps 380 results feels broken even when it worked perfectly.

Frequently Asked Questions