How to Build a Camera-to-AI Drawing Assistant That Feels Real-Time

Today I want to walk through a practical architecture for turning a phone camera feed into AI-assisted drawing guidance without making the experience feel laggy or noisy.

You do not need a huge model stack to make this useful. You need three things working together:

• a reliable frame pipeline
• predictable rendering modes
• strict timing and fallback rules

1. Start with a frame contract, not raw images

If you stream random image blobs, latency will spike and your desktop or engine side cannot reason about ordering. Define a compact packet contract first.

import Foundation

struct FramePacket: Codable {
    let sessionId: UUID
    let frameIndex: Int
    let timestampMs: Int64
    let width: Int
    let height: Int
    let jpegData: Data
    let orientation: Int
}

Why this matters:

• frameIndex gives deterministic ordering
• timestampMs lets you detect stale generations
• dimensions + orientation prevent bad overlay alignment

2. Add backpressure so AI work never blocks capture

Your capture side should continue running even when generation is slower than camera FPS. Keep only the latest frame for inference and drop the rest.

final class InferenceQueue {
    private let queue = DispatchQueue(label: "ai.inference", qos: .userInitiated)
    private var latest: FramePacket?
    private var running = false

    func submit(_ frame: FramePacket) {
        queue.async {
            self.latest = frame
            guard !self.running else { return }
            self.running = true
            self.drain()
        }
    }

    private func drain() {
        guard let frame = latest else {
            running = false
            return
        }
        latest = nil

        runInference(on: frame) { [weak self] in
            self?.queue.async { self?.drain() }
        }
    }
}

This pattern keeps UI responsive and avoids generation queues that are several seconds behind the live camera.

3. Render overlays with explicit blend modes

Artists need different guidance styles. Some want exact tracing, others want structure hints only. Blend mode controls make the same model output usable for both.

import CoreImage

enum OverlayBlend {
    case normal, multiply, screen, softLight, difference
}

func composite(base: CIImage, overlay: CIImage, mode: OverlayBlend, alpha: CGFloat) -> CIImage {
    let weighted = overlay.applyingFilter("CIColorMatrix", parameters: [
        "inputAVector": CIVector(x: 0, y: 0, z: 0, w: alpha)
    ])

    switch mode {
    case .normal:
        return weighted.composited(over: base)
    case .multiply:
        return weighted.applyingFilter("CIMultiplyBlendMode", parameters: ["inputBackgroundImage": base])
    case .screen:
        return weighted.applyingFilter("CIScreenBlendMode", parameters: ["inputBackgroundImage": base])
    case .softLight:
        return weighted.applyingFilter("CISoftLightBlendMode", parameters: ["inputBackgroundImage": base])
    case .difference:
        return weighted.applyingFilter("CIDifferenceBlendMode", parameters: ["inputBackgroundImage": base])
    }
}

Expose this as a simple UI toggle and users immediately feel the system is more controllable.

4. Treat quality mode as a product feature

Use at least two generation profiles:

• Fast preview: lower resolution, frequent refresh
• Quality render: higher resolution, less frequent refresh

Users can sketch with preview mode, then request higher-quality guidance for difficult parts of the drawing.

5. Pitfalls to avoid

• Using camera FPS as generation target. Inference cadence should be independent.
• Applying overlays before perspective correction.
• Letting old AI output overwrite newer frames.
• Hiding failure states. Show when the engine drops to fallback mode.

6. Verification checklist

Before shipping, verify:

• overlay drift stays under a few pixels after perspective correction
• stale frames are dropped correctly under load
• switching between fast and quality mode does not freeze capture
• end-to-end latency remains stable over a 10-minute session

If these are stable, the assistant feels intentional instead of gimmicky.