Generative AI | Zerui "Jerry" Ma's personal website

High-Fidelity AI Video Generation Engine (LTX-2.5)

Sun, 01 Mar 2026 00:00:00 +0000

High-Fidelity AI Video Generation Engine (LTX-2.5)

Overview

Engineered an ultra-low-latency, memory-efficient video generation pipeline leveraging the LTX-2.5 Diffusion Transformer (DiT) architecture on a consumer NVIDIA GeForce RTX 5080 GPU (Blackwell architecture). By synthesizing custom low-precision quantized GEMM kernels and an asynchronous layer-wise weight offloading subsystem, this engine delivers high-resolution, multi-frame video synthesis within a 16GB VRAM budget.

flowchart LR A[Host System RAM\nPinned Memory] <-->|Async DMA Stream\nLayer-wise Weights| B[NVIDIA RTX 5080\n16GB GDDR7] subgraph GPU Computation B --> C[Custom NVFP4 GEMM Kernels\nDynamic FP4/FP8 Casting] C --> D[Spatio-Temporal DiT Layers\nFlashAttention + CUDA Graphs] D --> E[Latent Frame Synthesis] end E --> F[High-Fidelity Video Output]

Technical Architectural Innovations

1. Custom NVFP4 Quantized GEMM Compilation

Designed and compiled custom NVFP4 (NVIDIA 4-bit Floating Point) quantized GEMM kernels utilizing Blackwell tensor cores.
Implemented robust dynamic FP4/FP8 casting with adaptive block-level scale-factor management, maintaining numerical stability across extreme gradient dynamics.
Achieved a 65% reduction in model memory footprint with negligible visual fidelity degradation compared to standard FP16/BF16 baselines.

2. Asynchronous Pinned-Memory Layer-wise Offloading

Architected an asynchronous double-buffered weight streaming engine that exploits high-bandwidth PCIe Gen 5 DMA transfers.
Streams individual DiT transformer blocks into VRAM concurrently with the forward-pass computation of preceding layers, completely hiding memory transfer latency.
Enabled continuous multi-frame 1080p video generation within consumer 16GB VRAM, eliminating out-of-memory (OOM) exceptions.

3. FlashAttention & CUDA Graph Optimization

Integrated fused FlashAttention kernels tailored for 3D spatio-temporal attention matrices.
Captured static execution branches into CUDA Graphs, eradicating CPU-GPU launch overhead and cutting end-to-end inference latency by 42%.

Key Results & Benchmarks

Memory Footprint: Reduced from ~42 GB (BF16 unquantized) to <14.8 GB active VRAM.
Latency Acceleration: 42% end-to-end speedup over native PyTorch baseline implementations.
Visual Quality: Retained smooth temporal coherence and high perceptual score across complex multi-object video generations.

Conditional Diffusion Models for Protein Trajectories (PyTAO)

Thu, 15 Jan 2026 00:00:00 +0000

Conditional Diffusion Models for Protein Trajectories (PyTAO)

Overview

Sampling continuous molecular dynamics and conformational transitions of proteins typically requires computationally prohibitive quantum mechanical / molecular mechanics (QM/MM) calculations. In collaboration with Dr. Peng Tao’s Proteins and Computers Lab at SMU, we designed and trained conditional diffusion generative models capable of directly generating continuous biomolecular conformational trajectories from boundary structural coordinates, bypassing thousands of CPU/GPU hours of physics simulations.

This work culminated in a research publication in the Journal of Chemical Theory and Computation (ACS, 2026) and the open-source PyTAO computational chemistry and trajectory toolkit.

flowchart LR A[Initial & Target\nConformations] --> B[Conditional Diffusion Model\nNoise Scheduler & Denoiser] B --> C[Slurm HPC Training\nDistributed Data Parallel] C --> D[Continuous Molecular Dynamics\nTrajectory Reconstruction] D --> E[Validated Physical Coordinates\nBypassing QM/MM Cost]

Highlights & Engineering

1. Conditional Generative Diffusion

Implemented deep generative diffusion models conditioned on 3D Cartesian coordinates and learned invariant spatial representations.
Reconstructed high-dimensional protein conformational trajectories with realistic bond angles and non-bonded interactions.

2. High-Performance Distributed Training (DDP)

Developed Slurm parallel batch training scripts leveraging PyTorch Distributed Data Parallel (DDP) across high-performance GPU clusters.
Optimized tensor transformations and memory-mapped trajectory datasets for rapid epoch turnover.

3. Open-Source PyTAO Toolkit

Built, tested, and released the PyTAO open-source Python library, integrating ONIOM calculation automation, coordinate parsing, and Slurm HPC job monitoring for computational chemistry workflows.

Publication

Efficient sampling of short protein trajectories with conditional diffusion models
Chen Xiong, Praveen Kandhan, Dong Chen, Zerui Ma, Eric D. Smith, Peng Tao.
Journal of Chemical Theory and Computation (ACS), 22(1), 78 (2026).
DOI: .

Efficient sampling of short protein trajectories with conditional diffusion models

Thu, 15 Jan 2026 00:00:00 +0000

Published in the Journal of Chemical Theory and Computation (ACS), 2026, 22(1), 78.
DOI: .

Paper Published in Journal of Chemical Theory and Computation (ACS)

Thu, 15 Jan 2026 00:00:00 +0000

Our collaborative research paper, , has been published in the Journal of Chemical Theory and Computation (ACS), Vol. 22, Iss. 1 (2026).

Co-authored with Chen Xiong, Praveen Kandhan, Dong Chen, Eric D. Smith, and Dr. Peng Tao. Check out the open-source implementation in our repository.