Development history¶

This page records how the engine was built, phase by phase, up to 2.1.8-beta (for what comes next see the roadmap). UltrafastEngine 2.0 was developed in phases. Each phase ends with tests that pin the new physics to analytic results or another code. Status as of 2.1.8-beta (Stage 1 complete).

Order from here (decided 2026-10-16): Stage 1 finishes the core engine and its documentation (released as 2.1.8-beta, the stable core); Stage 2 is the fitting plugin with transform elements; Stage 3 is the Zemax plugin. The live-data plugin and ML/PINN plugins follow after the stable release. Fitting, Zemax and live data are plugins: installed separately, loaded through public interfaces, never required by the core.

Phase 0 -- the split and the contract (done)¶

[x] Zemax-free C# core (net10.0 + net48), zero dependencies; own JSON, expression compiler, FFT, NPZ writer
[x] System file (.ufs) and run spec formats, reader/writer, validation
[x] Model registry with data / expression / callback forms; overrides (run > system > data > material > engine) and provenance report
[x] Material library (Sellmeier/Ciddor/gases, absolute index), table/expression/callback indices
[x] v2 L0 physics ported into compile -> execute (group-delay linear operators, RK4IP, grid planner, diagnostics)
[x] C API (ultrafast.h, [UnmanagedCallersOnly], NativeAOT-ready), Python package (ctypes / hostfxr), CLI
[x] Zemax binding rebuilt on an IOpticStudio abstraction: sampler, system builder, fake OpticStudio, record/replay, stubs
[x] Test runner (C#), pytest suite, docs site, examples

Phase 1 -- the physics you see in the lab (done in this release)¶

[x] Fresnel s/p, thin-film transfer matrices (stacks, quarter-wave notation), ideal/AR/GDD/tabulated coatings, grating efficiency
[x] Losses with exact per-mechanism budget: Fresnel, coatings, clipping (Gaussian on apertures, vignetting), surface scatter (TIS), absorption
[x] Jones polarisation with 3D s/p basis transport from traced ray vectors; waveplates, polarisers, rotators
[x] LIDT margins and peak fluence/intensity per surface; critical-power warning
[x] Adaptive ERK4(3)-IP stepper (FSAL) besides fixed RK4IP
[x] Full vector chi(3) coupling (XPM 2/3 + coherent term)
[x] User time-domain responses n(t, I) with co-moving and lab time; replace/add modes
[x] Carrier-resolved real-field propagation (automatic where a model asks for it)
[x] Instruments: energy meter, spectrometer, scope/photodiode, AC/IAC, FROG (SHG/PG), d-scan, contrast, polarimeter, pulse summary, ideal compressor
[x] pyNLO cross-check
[x] Zemax export: AOI (RAID), normals (RENA/B/C), coating rules (+ optional operand read-out), semi-diameters, beam radii, environment

Phase 2 -- active media and spatial physics¶

[x] 2.1 foundations: per-pulse grids with exact carrier change, internal multi-field propagation, auxiliary beam ports (parameters / previous run / callback), by-products, signed energy budget, user elements
[x] 2.1 spatial: radial (Hankel) and cartesian transverse models with diffraction per element, profiles carried between elements, lenses and curved mirrors, beam diagnostics (M2, true peak fluence)
[x] 2.1 χ(2) / OPCPA: crystal element (SHG/SFG/DFG/OPA, non-collinear, type 0/I/II, QPM), crystal library, phase-matching calculator, superfluorescence seeding, pyNLO cross-check
[x] 2.2 gain: Yb:YAG, Ti:sapphire, Nd:YAG and user media, saturation (spectral Frantz-Nodvik, time-ordered), pump models and rate equations, pulse-train steady state, thermal lens, multipass and regenerative amplifiers, spatial gain; position-dependent material properties
[x] 2.3 solid-state oscillators (parametric, Haus and cavity models; Kerr-lens and SESAM mode-locking; soliton, multi-pulse, cw-breakthrough and QML diagnostics); broadband χ(2) in the carrier-resolved mode, ASE, thermal lens to OpticStudio (Gradient 1)

Phase 2.5 -- connecting to the lab (done; fitting moves to a plugin in Stage 1.1 and is rebuilt in Stage 2)¶

[x] Instrument sampling: near field / far field (focal plane of a lens), pinhole, iris, fibre (NA check), slit, point, off-axis; on carried profiles or a Gaussian beam model
[x] Measured data on any instrument: CSV / NPZ / JSON with units from column names, background, normalisation, per-point or relative σ; curves, 2-D traces (FROG, d-scan, cameras) and scalar readings; χ², rms error, plots
[x] Calibration fit: Levenberg-Marquardt over paths into the system / run files, frozen grid, covariance, correlation and identifiability warnings; predictions with 1σ for every instrument and element, measured or not

Stage 1 -- stable core (done: 2.1.8-beta)¶

Guiding use case: reproducing high-dynamic-range contrast measurements of a Yb amplifier -> telescope -> MPC -> compressor chain (energy-dependent pedestal, asymmetric post-pedestal), with every candidate mechanism switchable.

1.1 Architecture (done: 2.0.0-alpha.2)¶

[x] Plugin system: IPlugin + manifest, discovery (plugins/ folders, ULTRAFAST_PLUGIN_PATH), ultrafast plugins list|install|remove, Python discovery; plugins register elements, instruments, models, solvers, run modes and CLI commands
[x] Zemax binding moved to plugins/zemax; current fitting/comparison code moved to plugins/fitting (kept working until Stage 2)
[x] Option registry: every element / instrument / model declares its parameters (type, unit, default, description) once
[x] JSON Schema v2 for .ufs and run specs generated from the registry ($schema for editor autocompletion); ultrafast schema, ultrafast validate against it
[x] Schema-versioned migrations (v1 -> v2) in the loader and ultrafast migrate
[x] Unique instrument names (auto-suffix + warning); many identical instruments anywhere
[x] One camera instrument: imaging near / far / relay (lenses and gaps) + sensor model (pixels, QE(λ), exposure, gain, bit depth, read/dark noise, saturation, hot pixels, background); nearFieldCamera / farFieldCamera migrate

1.2 Optical components (non-ray-traced models of what each component does to the beam) (done: 2.1.8-beta)¶

[x] Gaussian beam tracking (astigmatic q-parameters x/y) through every element in 1-D: lenses, curved mirrors and gaps update the beam of later elements automatically (principal axes x/y only; thermal lenses of gain media not yet in the tracking)
[x] Mirrors: off-axis astigmatism of a curved mirror, Zernike aberrations and surface figure on spatial fields, GDD-ripple coating models
[x] ndFilter (OD or OD(λ), absorptive / reflective, substrate dispersion and nonlinearity, ghosts)
[x] Thick (multi-order / zero-order) waveplates with material dispersion; thin-film polarisers at AOI; Faraday rotator (built-in TGG data approximate)
[x] Ghost reflections of plates and windows (post-pulses, and their nonlinear pre-pulse partners)

1.3 Contrast and polarisation (done: 2.1.8-beta)¶

[x] Third-order cross-correlator instrument (SHG-gated 3ω correlation, 10^10-10^12 dynamic range, resolution, noise floor, scan grid, polarisation selection)
[x] Input pulse pedestal models: ASE (incoherent), spectral phase noise, pre/post pulses
[x] Vector (Jones) propagation on spatial grids (NER resolved across the beam)

1.4 Gases (done: 2.1.8-beta)¶

[x] Rotational Raman of molecular gases (N2, O2, air, CO2, H2), isotropic + anisotropic parts (acts on NER); molecular constants from the literature, dephasing times approximate
[x] refractiveindex.info database reader (all formulas + tabulated n,k), bundled subset, import tool, p/T scaling

1.5 L3 propagation (done: 2.1.8-beta)¶

[x] UPPE on radial / cartesian grids (full k_z dispersion, non-paraxial, space-time focusing)
[x] Ionisation (PPT / ADK, Keldysh, multiphoton) and plasma (Drude, avalanche, recombination, absorption)

1.6 Speed and studies (done: 2.1.8-beta)¶

[x] SIMD complex kernels (Vector128: SSE2 / NEON on .NET 10); mixed-radix and Bluestein FFT
[x] Compute back-end interface above IFft (whole segment on the device); GPU back-ends are left to plugins (none shipped yet)
[x] Parameter scans (grid, random, LHS, Sobol), optimisation, sensitivities (local, Sobol); prefix caching; parallel batches (pipelining of elements across points deferred)

1.7 Pulse retrieval (done: 2.1.8-beta)¶

[x] FROG (PCGPA, ePIE) and d-scan retrieval behind IRetrieval; retrieved fields usable as run inputs

1.8 Consolidation (done: 2.1.8-beta)¶

[x] docs/reference, docs/tutorials, docs/examples (runnable, CLI + Python, a few C++), docs/assets (realistic synthetic data from a hidden "true" system with an answer key)
[x] Every option, function, extension point and override documented; agent-readable docs (llms.txt, AGENTS.md)
[x] Python 3.10+: full type hints (inline, py.typed; no separate stubs needed), typed dicts from the schema (ultrafast.types), generated API reference

Stage 2 -- fitting plugin with `transform` elements¶

[ ] transform element: a parameterised operator on the complex field -- spectral amplitude / phase, temporal modulation, Jones matrix per frequency, energy, spatial phase (Zernike), pointing, general (low-rank) matrices, user / ML operators -- so spectrum, pulse, contrast and beam stay mutually consistent
[ ] Constraints: passivity, causality (Kramers-Kronig), smoothness; identifiability report
[ ] Replaceable estimators (LM, regularised GN, CMA-ES, ensemble Kalman, MCMC; user plugins); instrument-response corrections
[ ] Results in three layers: simulated -> inferred -> believed, with provenance and an influence map

Stage 3 -- Zemax plugin (OpticStudio depth)¶

[ ] Zernike aberrations exported from OpticStudio applied to spatial fields (IAberrationModel)
[ ] Paraxial ABCD matrices between elements from paraxial-ray operands (exact transverse transport)
[ ] Prism pairs and Martinez stretchers with lens aberrations as analytic elements
[ ] Output basis vectors in results (lab-frame polarisation after folded paths)

Stage 4 -- engine depth before branches (2026-10-29 to 30)¶

Done in this order, each stage building on the one before: review of tests with no external reference (T0); absorbing boundary, Hankel cache and cw source (T1.1); Gerchberg-Saxton and near fields (T1.4); ray-traced dispersion, prism, spaceTime, angular dispersion on profiles (T1.3); thermal lens, stress birefringence, partial coherence, resonators (T2). New tracker state stayed inside BeamState, so each edge of a graph could later own a clone, and partial coherence stayed a run mode around the executor, so branches do not depend on it.

[x] Diffraction validated against the Fresnel integral; absorbing boundary (linear and nonlinear elements); profile-only linear path; cheaper Hankel matrix
[x] Beam tracking: thermal lenses, Kostenbauder couplings, ray bundle for dispersion to fourth order, prism, angular dispersion on carried profiles
[x] Sources and masks: Hermite-Gauss, Laguerre-Gauss, Bessel-Gauss, Airy, measured near fields (NPZ, PNG), cw; axicon, zone plate, vortex, slits, maps; Gerchberg-Saxton
[x] Cavities and coherence: resonator (eigenmode, Fox-Li), Gaussian-Schell partial coherence, stress birefringence, amplifier thermal lens
[x] Trust: build/example_check.py, Linux CI, generated validation matrix

Stage 5 -- branched systems (from 2026-10-30)¶

[x] Ports on all element types, PathLedger, SystemGraph with verification, Sequential and Branched forms
[x] Interference in 1-D, packets, loops, crystal ports, beamSplitter, delay, virtual inputs
[x] An independent physics review of the T1 and T2 work, with its findings fixed
[x] Space, the splitter family, Gouy and crystal phases, tail per path, node cache, array studies, modelled instruments

Stage 6 -- review of T0 to T2 and branches (2026-10-31)¶

Every spatial model and the branched interferometers were compared with LightPipes (an independent Fresnel code, development use only), and branched arms with pyNLO, which turned out to run on Python 3.12 once jsonpickle is installed.

[x] Found and fixed: M² and second-moment radius of tilted, decentred and higher-order beams; steadyState loops claiming convergence while decaying; graph.energyOutJ hiding separate packets; unbooked packets at crystal ports; the second arm's tail lost when carried profiles merged
[x] Tests: test_lightpipes.py, test_pynlo_branched.py, ReviewRegressionTests
[x] Examples 32 to 34 moved to the Python API; 35 to 38 added; tutorial files moved to docs/tutorials/files/

Stage 7 -- adaptive optics, temporal shaping and the digital twin (2026-11-09 to 10; five sessions S1-S5, merged to `main`)¶

Owner decisions: transverse shaping first, temporal shaping too; "Zernike" means modes; the fibre array is a coherent tiled-aperture array; every device is parametric (no vendor models); up to a full 1920 x 1080 SLM; the twin is its own plugin on top of fitting; CI on Linux (x64, arm64) and Windows, macOS by hand; Apache-2.0.

[x] S1 foundation: Python package split by what things are (shims keep every 2.2.0 name), ultrafast.plotting with one theme and animate, Zernike decomposition, examples regrouped into folders with make_all.py / make_figures.py, gallery generator
[x] S2 core: modulator element (cells, fill factor, gaps, quantisation, cross-talk, chromatic rules), in-memory maps, transverse.frequencyBlock, outputs.keepProfiles, coherent tile source, slope maps from wavefrontSensor; scale spikes (a 1920 x 1080 panel runs; the memory peak is the .NET GC, 2.7-3.4 GB)
[x] S3 ao plugin: deformableMirror, slm, spectralShaper, aopdf; ultrafast_ao (DM bench, tiled array, combining metrics); holography, control and sensorless algorithms; a physics review (15 findings, 9 fixed)
[x] S4 fitting and twin: fitting across conditions (held-out conditions, dependence kinds up to network, surfaces), the twin plugin (project, validation, planner, what-if, drift tracker, goals, benchmark)
[x] S5 examples and docs: 7 AdaptiveOptics, 2 DigitalTwin, 3 Fibres examples; twin docs

Stage 8 -- closing gaps toward the first release (2026-11-11; sessions A-F on `dev/2026-11`)¶

[x] WP0 (A): UF_WITH_AO in the native library, grid-planner hook for shaper side pulses (widens the window or warns), unknown-element message names ao, tests for Littrow dispersion, Bolduc and double phase through an SLM, tiled-array lobe fractions
[x] WP1 (B) thermal lenses in beam tracking: distributed (quadratic-index) lens in the ABCD tracker, applied per pass in multipass and per round trip in regenerative amplifiers, effective focal length reported
[x] WP4 (B) plasma in envelope mode: frequency-resolved Drude operator (ionization.frequencyResolved, default true), electronDensityM3 for a prepared plasma; envelope against carrier-resolved within 1.2 % (blue shift) for MPI. The frequency-resolved shift is up to 5 % below the single-carrier formula for a 30 fs pulse at 1030 nm (the component check states both)
[x] WP2 (D) gain: aseModel (steady-state forward / backward ASE per transverse cell and frequency, depletes the inversion between pulses; checked against n_sp h nu dnu (G-1), Frantz-Nodvik, saturated-amplifier law, cw photon balance); four-lobed stress-birefringence Jones map on cartesian grids (beam average equals the lumped value to 1e-6)
[x] WP3 (C) material catalogue: all material, gas, Raman, gain and chi(2) data in embedded JSON with source, status and uncertainty; runs list every approximate / unverified value used (summary.json unverifiedInputs); physics.strictMaterials; TGG index refitted to the Schlarb-Sugg table (old fit up to 0.0106 off); 43 of 183 quantities verified; the YVO4 vendor sets stay (5e-4 / 9e-4 from Shi et al. 2001, recorded)
[x] WP5 (A) decisions: no compile cache (a 128 x 128 trial costs 10 ms including compile), no full streaming executor (revisit for 4096 x 4096), tiles checked against a Branched balanced tree in a test; triggers in docs/reference/speed.md
[x] WP6/WP7 (A, E) findings: ny = 1 grids; cw line built at the requested carrier (cw against pulse agree to 1e-9); SPGD gain normalised by the running maximum (default reaches >= 0.9 over metric scales 1e-3 to 1e3; Spall sequences not built); Raman soliton shift rate explained (Gordon's first-moment approximation; engine within 2 % of the exact-gain rate and 0.3 % of pyNLO); SLM against Fourier explained (panel half a pixel off the grid; new warning); tiled-array reference explained (under-sampled lens, lock now at f = 1 m)
[x] WP8 (F) release system: components and manifest, install.py (release and git sources, install, add, upgrade, uninstall), packager --components, ZOS-API gate, release.yml, 23 tests (see releasing)
[x] Polish: README picture grid and contact badge, logo palette and banner on every docs page, CI timeouts and apt retries, plasma component check split into carrier and frequency-resolved rows
[x] Follow-up (PR #1): avalanche / inverse-bremsstrahlung double count fixed; catalogue defaults for user gain media and solids without data; strictMaterialsAllow; partial-override notes; Keldysh far-wing NaN. Spall gain sequences judged unnecessary (the running-maximum normalisation covers the range 1e-3 to 1e3)