Example configs¶
Complete pipeline YAML for each analysis mode, pulled live from the
examples/ directory.
Each one runs end to end with Pipeline.from_yaml(...). Read the
YAML pipeline guide for the section-by-section
reference and the step reference for what each step does.
Static XES¶
Single-state emission spectra, no time or energy axis. Loads the ePix, filters
to x-ray shots, patches dead columns, rotates the dispersion axis, and sums to
one spectrum per run. examples/mfx101080524_static_xes.yaml.
# Static XES analysis for mfx101080524 (AlkB Fe Ka)
# Replicates the workflow in XSpect_XES_mfx101080524.ipynb
#
# Old pattern:
# xes = XESBatchAnalysisRotation()
# xes.rois = [[89, 105]]
# xes.adu_cutoff = 3.0
# xes.angle = -1.0
# xes.pixels_to_patch = [351, 352]
# xes.run_parser(['237-239'])
# xes.primary_analysis_parallel_range(4, experiment, method=xes.primary_analysis_static, increment=500)
#
# New pattern:
# pipe = Pipeline.from_yaml("mfx101080524_static_xes.yaml")
# pipe.run(cores=4, batch_size=500)
#
# Matches primary_analysis_static workflow:
# 1. union_shots(epix, ['xray','xray']) — keep only xray shots
# 2. filter_detector_adu(epix, 3.0) — zero pixels below threshold
# 3. hitfinding(epix, cutoff_multiplier=1) — remove low-signal shots
# 4. reduce_detector_shots(epix, sum) — sum shots → 2D (704 x 200)
# 5. patch_pixels(epix, axis=0, pixels=[351,352]) — fix bad rows on 2D
# 6. rotate(epix, -1.0) — rotate 2D image
# 7. reduce_detector_spatial(epix, rois=[[89,105]]) — ROI on last axis, sum → 1D (704,)
# 8. make_energy_axis — vonHamos geometry → energy in eV
experiment:
hutch: mfx
experiment_id: mfx101080524
lcls_run: 24
data:
runs: [237-239]
keys:
ipm_dg2/sum: ipm
lightStatus/xray: xray
lightStatus/laser: laser
detector_keys:
epix_1/ROI_0_area:
name: epix
transpose: false
pipeline:
# 1. Filter to xray-only shots (AND of xray mask with itself = xray mask)
- step: union_shots
on: epix
filter_keys: [xray, xray]
new_key: epix
# 2. ADU threshold (zero pixels below 3.0 ADU)
- step: filter_detector_adu
on: epix
adu_threshold: 3.0
# 4. Sum all shots → 2D image (704 x 200)
- step: reduce_detector_shots
on: epix
reduction: sum
# 5. Patch bad pixels on 2D summed image (axis 0 = dispersion/rows)
- step: patch_pixels
on: epix_reduced
pixels: [351, 352]
mode: polynomial
axis: 0
# 6. Rotate the summed 2D image
- step: rotate_detector
on: epix_reduced
angle: -1.0
# 7. Apply ROI on last axis (cross-dispersion, 200 px) and sum → 1D spectrum (704,)
- step: reduce_detector_spatial
on: epix_reduced
rois: [[89, 105]]
combine_rois: true
reduction: sum
# 8. Convert pixel axis to energy (vonHamos geometry for Fe Ka)
- step: make_energy_axis
detector_key: epix_reduced_ROI_1
crystal_detector_distance: 42.75
crystal_radius: 250.0
d_spacing: 0.981
mm_per_pixel: 0.05
name: xes
output:
format: hdf5
path: ./results/mfx101080524_static/
Time-resolved (pump-probe) XES¶
Laser-on minus laser-off emission, binned by pump-probe delay. Adds
time_binning and the laser masks, then reduce_detector_temporal and the
combine_runs reduction to build the transient.
examples/mfxl1027922_ultrafast_xes.yaml.
# Ultrafast time-resolved XES analysis for mfxl1027922
# Replicates the workflow in XSpect_XES_mfxl1027922.ipynb (cells 1-5)
#
# Old pattern:
# xes = XESBatchAnalysisRotation()
# keys = ['tt/ttCorr','epics/lxt','enc/lasDelay','ipm4/sum','tt/AMPL','epix_2/ROI_0_area']
# names = ['time_tool_correction','lxt_ttc','encoder','ipm','time_tool_ampl','epix']
# xes.rois = [[0, 50]]
# xes.adu_cutoff = 3.0
# xes.angle = 90
# xes.transpose = True
# xes.mintime = -0.9
# xes.maxtime = 0.9
# xes.numpoints = 40
# xes.add_filter('simultaneous','time_tool_ampl',0.05)
# xes.run_parser(['44-46'])
# xes.primary_analysis_parallel_range(8, xes_experiment, increment=1000)
#
# New pattern:
# pipe = Pipeline.from_yaml("mfxl1027922_ultrafast_xes.yaml")
# pipe.run(cores=8, batch_size=1000)
#
# Matches primary_analysis_parallel_range workflow (time-resolved):
# 1. filter_shots(simultaneous, time_tool_ampl > 0.05) — keep good timing shots
# 2. filter_detector_adu(epix, 3.0) — zero pixels below threshold
# 3. rotate_detector(epix, 90) — rotate detector image 90 degrees
# 4. reduce_detector_spatial(epix, rois=[[0,50]]) — ROI sum → 2D (shots x pixels)
# 5. union_shots(epix_ROI_1, [simultaneous, laser]) — laser-on shots
# 6. separate_shots(epix_ROI_1, [xray, laser]) — xray-only (laser-off) shots
# 7. time_binning([-0.9, 0.9, 40]) — create 40 time bins
# 8. union/separate timing indices — split timing for on/off
# 9. reduce_detector_temporal — bin spectra into time bins
# 10. normalize_xes — area-normalize each time bin
# 11. make_energy_axis — vonHamos geometry → energy
experiment:
hutch: mfx
experiment_id: mfxl1027922
lcls_run: 22
data:
runs: [44-46]
keys:
tt/ttCorr: time_tool_correction
enc/lasDelay: encoder
ipm_dg2/sum: ipm
tt/AMPL: time_tool_ampl
lightStatus/xray: xray
lightStatus/laser: laser
detector_keys:
epix_2/ROI_0_area:
name: epix
rois: [[0, 50]]
combine_rois: true
transpose: true
pipeline:
# 1. Filter simultaneous shots on time tool amplitude (keep > 0.05)
- step: filter_shots
on: simultaneous
filter_key: time_tool_ampl
threshold: 0.05
# 2. Create time bins from timing data (-0.9 to 0.9 ps, 40 points)
# This must happen early so we can filter masks by delay range
- step: time_binning
bins: [-0.9, 0.9, 40]
lxt_key: null
# 3. Filter shot masks to only include shots within timing range
- step: filter_shots
on: xray
filter_key: delays
threshold: [-0.9, 0.9]
- step: filter_shots
on: simultaneous
filter_key: delays
threshold: [-0.9, 0.9]
# 4. Split timing bin indices by laser status
- step: union_shots
on: timing_bin_indices
filter_keys: [simultaneous, laser]
- step: separate_shots
on: timing_bin_indices
filter_keys: [xray, laser]
# 5. ADU threshold (zero detector pixels below 3.0 ADU)
- step: filter_detector_adu
on: epix
adu_threshold: 3.0
# 6. Reduce spatial: sum cross-dispersion ROI [0:50] on axis 1 → 2D (shots x 705)
# Data is (shots, 705, 100) after transpose; ROI on axis=2 (cross-dispersion)
- step: reduce_detector_spatial
on: epix
rois: [[0, 50]]
combine_rois: true
reduction: sum
axis: 2
# 8. Select laser-on shots (simultaneous AND laser)
- step: union_shots
on: epix_ROI_1
filter_keys: [simultaneous, laser]
# 9. Select laser-off shots (xray AND NOT laser)
- step: separate_shots
on: epix_ROI_1
filter_keys: [xray, laser]
# 10. Bin detector data into time bins (laser-on)
- step: reduce_detector_temporal
on: epix_ROI_1_simultaneous_laser
timing_bin_key: timing_bin_indices_simultaneous_laser
# 11. Bin detector data into time bins (laser-off / xray-only)
- step: reduce_detector_temporal
on: epix_ROI_1_xray_not_laser
timing_bin_key: timing_bin_indices_xray_not_laser
# 12. Generate energy axis from vonHamos spectrometer geometry
- step: make_energy_axis
detector_key: epix_ROI_1
crystal_detector_distance: 50.6
crystal_radius: 250.0
d_spacing: 0.895
mm_per_pixel: 0.05
name: kbeta
reduction:
- step: combine_runs
detector_key: epix_ROI_1
output:
format: hdf5
path: ./results/mfxl1027922_ultrafast/
2D XAS (energy × delay)¶
Simultaneous incident-energy scan and pump-probe delay, producing a transient
absorption map Δμ(E, t). Uses make_ccm_axis, ccm_binning, time_binning,
and reduce_detector_ccm_temporal. examples/xcs101591326_2d_xas.yaml.
# 2D XAS: DCCM energy × time delay for xcs101591326
# Runs 187-216: simultaneous DCCM energy scan + pump-probe time axis
# Produces transient absorption map: Δμ(E, t) = μ_on(E,t) − μ_off_pre-t0(E)
#
# Axes:
# Energy: auto from dccm_E_setpoint (~0.25 eV resolution)
# Time: enc/lasDelay, -2 to 8 ps, 0.5 ps bins (21 points)
#
# Output arrays per run (shape: n_time × n_energy):
# epix_simultaneous_laser_time_energy_binned
# ipm_simultaneous_laser_time_energy_binned
# epix_xray_not_laser_time_energy_binned
# ipm_xray_not_laser_time_energy_binned
experiment:
hutch: xcs
experiment_id: xcs101591326
lcls_run: 26
data:
runs: ["187-216"]
keys:
ipm4/sum: ipm
epix_2/ROI_sum: epix
epicsUser/dccm_E_setpoint: ccm
enc/lasDelay: encoder
tt/ttCorr: time_tool_correction
lightStatus/xray: xray
evr/code_41: laser
detector_keys: {}
pipeline:
# 1. Filter shots on upstream I0 intensity
- step: filter_shots
on: simultaneous
filter_key: ipm
threshold: 50
# 2. CCM energy axis (auto-derived from setpoints, 0.25 eV resolution)
- step: make_ccm_axis
energies: auto
ccm_key: ccm
resolution: 0.00025
# 3. Digitize shots into CCM energy bins
- step: ccm_binning
ccm_key: ccm
ccm_bins_key: ccm_bins
# 4. Time binning: -2 to 8 ps, 0.5 ps bins (21 points via linspace)
- step: time_binning
bins: [-2, 8, 21]
lxt_key: null
fast_delay_key: encoder
tt_correction_key: time_tool_correction
# 5. Split by laser status — create filtered index arrays for both dimensions
- step: union_shots
on: epix
filter_keys: [simultaneous, laser]
- step: separate_shots
on: epix
filter_keys: [xray, laser]
- step: union_shots
on: ipm
filter_keys: [simultaneous, laser]
- step: separate_shots
on: ipm
filter_keys: [xray, laser]
- step: union_shots
on: ccm_bin_indices
filter_keys: [simultaneous, laser]
- step: separate_shots
on: ccm_bin_indices
filter_keys: [xray, laser]
- step: union_shots
on: timing_bin_indices
filter_keys: [simultaneous, laser]
- step: separate_shots
on: timing_bin_indices
filter_keys: [xray, laser]
# 6. 2D binning (time × energy) — laser-on
- step: reduce_detector_ccm_temporal
on: epix_simultaneous_laser
timing_bin_key: timing_bin_indices_simultaneous_laser
ccm_bin_key: ccm_bin_indices_simultaneous_laser
- step: reduce_detector_ccm_temporal
on: ipm_simultaneous_laser
timing_bin_key: timing_bin_indices_simultaneous_laser
ccm_bin_key: ccm_bin_indices_simultaneous_laser
# 7. 2D binning (time × energy) — laser-off reference
- step: reduce_detector_ccm_temporal
on: epix_xray_not_laser
timing_bin_key: timing_bin_indices_xray_not_laser
ccm_bin_key: ccm_bin_indices_xray_not_laser
- step: reduce_detector_ccm_temporal
on: ipm_xray_not_laser
timing_bin_key: timing_bin_indices_xray_not_laser
ccm_bin_key: ccm_bin_indices_xray_not_laser
output:
format: hdf5
path: ./results/xcs101591326_2d_xas/
Temporal XAS¶
Fluorescence-detected XAS at a fixed incident energy, scanned over delay
(I_f / I_0). examples/xcs101591326_temporal_xas.yaml.
# Temporal XAS analysis for xcs101591326
# Run 101: lxt_fast time-delay scan at fixed DCCM energy (7126.5 eV)
# Fluorescence-detected XAS using ipm5 (I0) and epix_2/ROI_sum (If)
#
# Workflow:
# 1. filter_shots — remove low-intensity shots (bad I0)
# 2. time_binning — bin shots by enc/lasDelay (1 ps bins, -40 to 100 ps)
# 3. union/separate_shots — split by laser status (pumped vs unpumped)
# 4. reduce_detector_temporal — bin I0 and If along time axis
#
# enc/lasDelay is in picoseconds.
# XAS signal: mu ∝ If/I0 (fluorescence yield) vs time delay.
experiment:
hutch: xcs
experiment_id: xcs101591326
lcls_run: 26
data:
runs: [339,340,341,342,343,344,347]
keys:
ipm5/sum: ipm
epix_2/ROI_sum: epix
enc/lasDelay: encoder
lightStatus/xray: xray
evr/code_41: laser
detector_keys: {}
pipeline:
# 1. Filter shots on upstream I0 intensity
- step: filter_shots
on: simultaneous
filter_key: ipm
threshold: 500
# 2. Time binning from lxt_fast (enc/lasDelay in ps) — 1 ps bins
- step: time_binning
bins: [-10, 25, 35]
lxt_key: null
fast_delay_key: encoder
tt_correction_key: null
# 3. Split shot masks by laser status
- step: union_shots
on: ipm
filter_keys: [simultaneous, laser]
- step: separate_shots
on: ipm
filter_keys: [xray, laser]
- step: union_shots
on: epix
filter_keys: [simultaneous, laser]
- step: separate_shots
on: epix
filter_keys: [xray, laser]
- step: union_shots
on: timing_bin_indices
filter_keys: [simultaneous, laser]
- step: separate_shots
on: timing_bin_indices
filter_keys: [xray, laser]
# 4. Bin I0 (ipm) along time axis — laser-on and laser-off
- step: reduce_detector_temporal
on: ipm_simultaneous_laser
timing_bin_key: timing_bin_indices_simultaneous_laser
- step: reduce_detector_temporal
on: ipm_xray_not_laser
timing_bin_key: timing_bin_indices_xray_not_laser
# 5. Bin If (fluorescence) along time axis — laser-on and laser-off
- step: reduce_detector_temporal
on: epix_simultaneous_laser
timing_bin_key: timing_bin_indices_simultaneous_laser
- step: reduce_detector_temporal
on: epix_xray_not_laser
timing_bin_key: timing_bin_indices_xray_not_laser
reduction:
- step: combine_runs
detector_key: epix
- step: combine_runs
detector_key: ipm
output:
format: hdf5
path: ./results/xcs101591326_temporal_xas/
Droplet / photon-counting XES¶
Per-shot XES from the MFX droplet2photon sparse layout. droplet_reconstruction
rebuilds dense frames from photon positions before the usual XES chain.
examples/mfx101609126_droplet_pershot_xes.yaml.
# Per-shot XES projections for mfx101609126 — DROPLET-RECONSTRUCTED variant.
#
# This is the droplet2photon analogue of mfx101609126_pershot_xes.yaml.
# Instead of loading the pre-processed epix100_*/ROI_area datasets, every
# detector image is reconstructed on-the-fly from the droplet2photon sparse
# photon coordinates (droplet_reconstruction step). Photon-counting
# reconstruction removes read noise and charge-sharing tails, which should
# improve the shot-to-shot signal-to-noise used by the stochastic RIXS solver.
#
# Three per-shot 1D spectra are produced, all filtered to xray-on shots so the
# shot index aligns across every array (required by spook):
# xrt_hproj (N_xray, 2048) — incident spectrum (XRT, feespec/hproj)
# epix_seer_ROI_1 (N_xray, 768) — SEER spectrometer (epix100_1), full panel width
# epix_spec_ROI_1 (N_xray, 300) — emission (epix100_0)
#
# Stochastic-RIXS comparison (see mfx101609126_spook_rixs.ipynb):
# * A = xrt_hproj — incident monitor #1 (current approach)
# * A = epix_seer_ROI_1 — incident monitor #2 (SEER, this is the new option)
# * B = epix_spec_ROI_1 — emission
# Cleaning up the SEER spectrometer (gap-patched, photon-counted) is what lets
# it stand in as the incident-energy monitor that XRT provides today.
#
# KEY DIFFERENCES from the ADU-based mfx101609126_pershot_xes.yaml:
# 1. No filter_detector_adu step. Droplet images are PHOTON COUNTS (1,2,3…),
# so an ADU threshold of 5 would zero all signal.
# 2. SEER (epix100_1) is reconstructed with FULL columns (0–768), only the
# rows are cropped. The smalldata ROI col_start was changed 80 -> 0, so the
# ROI is now the full panel width and reconstructed column indices equal
# full-panel indices — patch_pixels gap columns are given directly in
# full-panel coordinates (old ROI-local + 80).
# 3. SEER gap columns are patched from an explicit manual list (ASIC panel
# gaps), not auto_detect.
experiment:
hutch: mfx
experiment_id: mfx101609126
lcls_run: 24
data:
runs: [78, 79, 80, 81, 82]
# max_shots: 500 # uncomment to limit shots for a quick test
keys:
MfxDg2BmMon/totalIntensityJoules: ipm
lightStatus/xray: xray
feespec/hproj: xrt_hproj
# No detector_keys — epix images come from the droplet_reconstruction step.
pipeline:
# 1. Reconstruct per-shot images from the droplet2photon sparse arrays.
# epix100_0 (spectroscopy / emission): crop to the ROI_area window
# rows 270-330, cols 400-700 -> (60, 300)
- step: droplet_reconstruction
det: epix100_0
new_key: epix_spec
roi: [270, 330, 400, 700]
# epix100_1 (SEER / incident monitor): match the SMALLDATA ROI window
# rows 350-450, cols 0-768 -> (100, 768)
# This ROI comes from UserDataCfg/epix100_1/ROI__ROI_ROI = [350 450 0 768]
# (col_start was changed 80 -> 0, so the ROI is now the full panel width).
# The droplet2photon sparse coordinates are stored in FULL-PANEL coordinates
# (col 0-767), so with col_start=0 the reconstructed column index EQUALS the
# full-panel column index and the patch_pixels gap columns below are given in
# full-panel coordinates directly (old ROI-local indices + 80).
- step: droplet_reconstruction
det: epix100_1
new_key: epix_seer
roi: [350, 450, 0, 768]
# 2. Filter all arrays to xray-on shots so A and B share the same shot index
- step: union_shots
on: xrt_hproj
filter_keys: [xray, xray]
new_key: xrt_hproj
- step: union_shots
on: ipm
filter_keys: [xray, xray]
new_key: ipm
- step: union_shots
on: epix_spec
filter_keys: [xray, xray]
new_key: epix_spec
- step: union_shots
on: epix_seer
filter_keys: [xray, xray]
new_key: epix_seer
# 3. Patch ASIC panel-gap columns.
# epix100_0 (ROI-cropped 300-col frame): same gap columns as the
# ADU pipeline (cols are local to the 400-700 ROI window).
- step: patch_pixels
on: epix_spec
pixels: [80, 81, 82, 176, 177, 178, 272, 273, 274]
mode: polynomial
deg: 2
# epix100_1 SEER (768-col full-panel frame, cols 0-768): gap columns are
# now FULL-PANEL indices (= old ROI-local indices + 80, since the smalldata
# ROI col_start moved 80 -> 0). These are the ASIC tile-boundary gaps: a
# dark center column flanked by bright charge-sharing neighbors (verified
# against the reconstructed mean: dips at full-panel 95,191,287,381,...).
- step: patch_pixels
on: epix_seer
pixels: [94, 95, 96, 190, 191, 192, 286, 287, 288,
381, 382, 383, 384, 477, 478, 479, 480,
574, 575, 576, 634, 635, 636]
mode: polynomial
deg: 2
# 4. Rotate to align dispersion axis with detector columns
# Fixed angles from mfx101609126_rotation_diagnostic.ipynb:
# epix100_0 (spectroscopy): -2.0 deg
# epix100_1 (SEER): -1.6 deg
- step: rotate_detector
on: epix_spec
angle: -2.0
reshape: false
- step: rotate_detector
on: epix_seer
angle: -1.6
reshape: false
# 5. Project signal rows onto X -> per-shot 1D spectrum (N_xray, n_cols)
# epix_spec: local rows 10-40 (full panel rows 280-310)
# epix_seer: local rows 50-90 (full panel rows 400-440)
- step: reduce_detector_spatial
on: epix_spec
axis: 1
rois: [[10, 40]]
combine_rois: true
reduction: sum
purge: true
- step: reduce_detector_spatial
on: epix_seer
axis: 1
rois: [[50, 90]]
combine_rois: true
reduction: sum
purge: true
output:
format: hdf5
path: ./results/mfx101609126_droplet_pershot_xes/
RIXS¶
Incident-energy scan with 2D emission images, giving a RIXS plane (incident
energy from DCCM, emission energy from the von Hamos spectrometer).
examples/mfx101609126_static_rixs.yaml.
# Static RIXS plane for mfx101609126
# Run 97: DCCM energy scan with 2D epix images (emission spectra)
# Incident energy from DCCM, emission energy from vonHamos spectrometer
#
# Workflow:
# 1. union_shots — keep only xray shots
# 2. filter_detector_adu — zero pixels below ADU threshold
# 3. make_ccm_axis — define incident energy bins from DCCM readback
# 4. ccm_binning — digitize shots into incident energy bins
# 5. reduce_detector_ccm (ipm) — sum IPM per energy bin → normalization denominator
# 6. reduce_detector_ccm (epix) — sum 2D images at each incident energy point
# 7. reduce_detector_shots — total sum of all xray frames → 2D diagnostic image
# 8. reduce_detector_spatial — ROI + sum cross-dispersion → 1D per energy
#
# Result: 2D RIXS plane (incident energy x emission pixels)
experiment:
hutch: mfx
experiment_id: mfx101609126
lcls_run: 24
data:
runs: [97]
keys:
MfxDg2BmMon/totalIntensityJoules: ipm
lightStatus/xray: xray
epicsUser/dccm_E_setpoint: ccm
detector_keys:
epix100_0/ROI_area:
name: epix
transpose: false
pipeline:
# 1. Filter to xray-only shots
- step: union_shots
on: epix
filter_keys: [xray, xray]
new_key: epix
# 2. ADU threshold (remove pixels below 2 keV)
- step: filter_detector_adu
on: epix
adu_threshold: 2.0
# 3. Define incident energy axis from DCCM setpoints
- step: make_ccm_axis
energies: auto
ccm_key: ccm
resolution: 0.00025
# 4. Digitize shots into DCCM energy bins
- step: ccm_binning
ccm_key: ccm
ccm_bins_key: ccm_bins
# 5. Sum IPM per incident energy bin → (n_energy,) normalization denominator
- step: reduce_detector_ccm
on: ipm
ccm_bin_key: ccm_bin_indices
average: false
# 6. Sum 2D images at each incident energy → (n_energy x 60 x 300)
# (detector is already ROI-cropped to 60 rows x 300 cols in smalldata)
- step: reduce_detector_ccm
on: epix
ccm_bin_key: ccm_bin_indices
average: false
# 7. Total sum of all ADU-filtered xray frames → (60 x 300) diagnostic image
# Used to inspect signal distribution before choosing the spatial ROI.
- step: reduce_detector_shots
on: epix
reduction: sum
purge: true
# 8. Sum cross-dispersion axis (all 60 rows) → (n_energy x 300) RIXS plane
- step: reduce_detector_spatial
on: epix_energy_binned
rois: [[0, 60]]
combine_rois: true
reduction: sum
output:
format: hdf5
path: ./results/mfx101609126_static_rixs/