AI Machine Learning Foundations for VFX Artists

Class Listing

Class 1: Understanding AI & Machine Learning in VFX

AI might be the buzzword on your social feeds, but machine learning is the real engine under the hood of today’s most powerful VFX tools. In this first session, Doug Hogan breaks down the core concepts of Artificial Intelligence and Machine Learning, helping you cut through the hype and understand what these terms _actually_ mean for working artists.

You’ll explore how supervised machine learning works in tools like Nuke’s CopyCat, BorisFX Silhouette, and Mocha Pro. You'll see how labeled data and artist-driven training examples allow these tools to assist (not replace) your roto, tracking, and cleanup workflows while often cutting task time in half. Doug gives you an overview of all the techniques, from generating depth passes with MiDaS and ZoeDepth to using ML-powered inpainting tools for rig removal and plate cleanup.

This class also introduces key distinctions between Machine Learning (ML) and Generative AI (GenAI), showing how the former excels at identification and classification (discriminative models), while the latter synthesizes new images, textures, and even facial performances (generative models). You’ll also hear from VFX Supervisor Eric Levy of Clean Plate FX, who shares how ML is reshaping beauty work and artist workflows across real productions.

Additional topics include:

- How AI, ML, and Deep Learning are related
- ML-powered techniques for roto, cleanup, and denoising in Nuke, Silhouette, and Photoshop
- Open-source AI tools (like SAM, ComfyUI, and Instant-NGP) already used in production
- Case studies: 3D tracking with KeenTools, face replacement in _The Mandalorian_, and real-time deepfakes
- The ethics and artistry behind ML-assisted workflows

By the end of this class, you’ll walk away with a grounded understanding of AI and ML, how they apply to modern VFX pipelines, and how to begin experimenting with these tools right now, no coding required! This is the foundation for the rest of the series, and the launchpad for any artist looking to future-proof their creative toolkit.

Class 2: Scene Reconstruction with NeRFs & Gaussian Splatting

Photogrammetry has long been the go-to for capturing real-world environments, but recent breakthroughs in neural rendering have changed the game. In this session, Doug Hogan explores the evolution of scene reconstruction, from Paul Debevec’s early image-based lighting to cutting-edge tools like NeRFs and Gaussian Splatting that can turn a phone capture into a fully relightable 3D environment in minutes.

You'll learn what makes **Neural Radiance Fields (NeRFs)** so powerful for volumetric rendering, and why **Gaussian Splatting** is quickly becoming the go-to method for real-time scene capture and playback. Doug breaks down these concepts visually, helping you understand the core technical differences and why Gaussian Splatting is such a win for artists: fast rendering, beautiful lighting, and direct integration with tools like Nuke.

This class also walks you through a modern, artist-friendly pipeline using **PostShot**, **Scaniverse**, **RealityCapture**, and **Luma AI**, showing how to go from mobile capture to fully orbitable splat scene and finally into Nuke with a powerful new OFX plugin for compositing.

Additional topics include:

- The historical impact of Paul Debevec’s _Campanile Movie_ on photogrammetry
- NeRFs as neural volumetric fog vs. Gaussian Splatting as glowing ellipsoids
- Fast, local training with PostShot and how it fits into a modern VFX pipeline
- How to capture and export splats from mobile apps like Scaniverse and Luma AI
- Using Gaussian Splats inside Nuke’s 3D workspace for lighting and occlusion
- Future-forward tools like Meta’s Horizon Worlds AI generation, DreamGaussian, Blockade Labs, and Hunyuan's new World 1.0

By the end of this class, you’ll have a complete understanding of the workflow for capturing real-world spaces, generating real-time 3D splats, and compositing them seamlessly inside your VFX projects. So whether you're building set extensions, digital doubles, or immersive environments this is how the future gets captured.

Class 3: Markerless Motion Capture with Move AI and Alternatives

Motion capture used to mean a room full of cameras, markers, and high-end rigs. Today, all you need is a phone and the right software. In this session, Doug Hogan explores the rise of markerless motion capture and how tools like Move AI, Rokoko Vision, and AnimDif are giving indie creators the power to bring performances into 3D and 2D animation pipelines without the heavy overhead.

You'll learn how markerless mocap is changing the animation workflow, from rapid prototyping to full-blown character performance. Doug also traces the history of Wonder Studio, its surprising acquisition by Autodesk, and how that reflects a shift toward AI-assisted character pipelines in major studios.

The session includes a hands-on demo: generating a performance clip inside Wonder Studio using fxphd short footage from the Technicolor series, followed by an honest breakdown of where the tool shines and where it still struggles.

Additional topics include:

- Comparing Move AI and Rokoko Vision for 3D animation capture
- Using AnimDif for lightweight, indie-friendly 2D mocap
- Wonder Studio’s evolution and Autodesk’s move into AI mocap
- Best practices for integrating markerless mocap into production workflows
- The limitations of AI mocap tools and how to work around them

By the end of this class, you’ll understand how to set up and run your own markerless mocap captures, evaluate which tools fit your scale of production, and confidently decide when AI mocap is the right choice for your project.

Class 4: GenVFX – Compositing with AI

AI is making its mark on one of the oldest part of the VFX pipeline, Compositing. In this class, Doug Hogan introduces a new generation of AI-driven compositing techniques and shows how they fit into real-world VFX production. From background removal, roto, to relighting, AI is taking on the heavy lifting while still leaving the artistry in the hands of the compositor.

You’ll get a guided tour through some of the most exciting tools today: ComfyUI for building custom ML workflows, Runway Gen-4 for inpainting and clean plating, SwitchLight Studio for relighting passes, Higgsfield.ai for generative VFX-style simulations, and Skyglass, a new entrant pushing the iPhone into real-time compositing.

Doug demonstrates how to create a short scene using this toolset, evaluating each step for speed, quality, and usability. Along the way, you’ll see how to combine AI passes with traditional comp techniques to keep control of the final look.

Additional topics include:

- ComfyUI workflows for matte creation and background removal
- Clean plating and object removal with Runway Aleph
- Generating relight passes (diffuse, normal, depth) with SwitchLight Studio
- Generative particle and effects sims in Higgsfield.ai
- Early experiments with Skyglass and its creator-focused features

By the end of this class, you’ll know what’s hype and what’s production-ready, and you’ll have a practical understanding of how to use AI compositing tools to complement your existing Nuke workflows.

Class 5: Battle of the AI Generators

AI image and video models are evolving at a pace that feels impossible to keep up with. New models launch every week and older ones get major upgrades overnight. In this session, Doug Hogan runs a real production style showdown. Five of the top image generators and five of the leading video models go head to head. Same prompts. Same tests. Same criteria. Then we rank them. Best to worst. No guessing. No hype. Just results.

We begin by defining the evaluation criteria used throughout the session.
Accuracy to prompt. Ability to edit. Overall detail. Useful happy accidents. Does it feel like a collaboration or a fight. You will see how each of these metrics matters in real production tasks.

The image generator lineup includes Nano Banana Pro, Seedream, Wan 2.5, QWEN, and Flux. Each model gets the same prompt pack. Portraits. Multi character scenes. Hard objects. Fine text. Clocks. Perspective challenges. You will see how each model responds, where they fail, and where they surprise you.

The video generators go through the same test. Veo 3.1, Sora 2, Kling, Wan 2.5, and Moonvalley, the ethical favorite. We use the highest performing image model from the first half as the visual reference to keep the tests fair. You will see how each tool handles motion, consistency, lighting, and prompt control.

At the end of each section, Doug announces a ranked list. Then a full finale ranking covers all ten models together. We timestamp the results because this space moves fast and changes every month.

Additional topics include:

- How to judge an AI model the same way you judge an artist
- How to prompt these various models to achieve the results you want
- How “collaborative” a generator needs to feel to be useful in VFX
- Why happy accidents matter in ideation workflows
- The current state of ethical models in image and video generation

By the end of this class, you will know exactly which models to trust for production, which ones are best for ideation, and which ones are still experimental. This is your real world guide to choosing the right generator for the right moment in your pipeline.

Class 6: MLSharp Gaussian Splats and QWEN Image Layering

In this class, we take the highest performing image model from the previous session and push it into a full 3D reconstruction workflow. Using Nano Banana Pro as our image generator, we move beyond flat images and into high quality Gaussian splats using Apple’s MLSharp repository.

We start by generating a single ultra high resolution spherical image and converting it into a detailed Gaussian splat representation. This allows us to step into the scene, inspect depth, and treat the output more like a volumetric asset than a traditional image.

From there, we bring the splat into SuperSplat, a WebGL based splat viewer and editor, where we preview, inspect, and modify the result in real time. This is where the practical issues show up. Holes. Stretching. Occlusion errors. All the things you need to deal with if this is going to be usable in production.

To solve those problems, we introduce QWEN’s Image Layer model inside ComfyUI. We use it to intelligently extract separate depth and content layers from the original spherical image. Each layer is then converted into its own Gaussian splat. These splats are recombined inside SuperSplat to reduce artifacts, fill gaps, and produce a cleaner, more robust final result.

In the final section, we step out of pure ML and into tooling. Using Claude, we vibe code a lightweight custom UI that wraps the MLSharp command line workflow. This removes the need to remember complex CLI commands and shows how quickly you can prototype a real production facing tool that artists can actually use.

Additional topics include:

How Gaussian splats differ from meshes and NeRFs in real world workflows

Common failure cases when generating splats from single images

Why layered splat reconstruction produces cleaner results

How ComfyUI fits into non generative ML pipelines

Turning research code into usable artist tools

By the end of this class, you will understand how to generate, inspect, repair, and deploy Gaussian splats from a single image source. More importantly, you will see how to connect cutting edge research, node based workflows, and rapid UI prototyping into something that feels practical, flexible, and production ready.