i build flare-indexer for the data intelligence lab at tcu, where i work as a software engineer and research assistant. it started out as solarflare-labeler, a python package i made from the scripts the lab used to prepare flare prediction datasets, and it has grown quite a bit since then.
the question it answers sounds simple. given a solar image taken at some moment, did a significant flare happen in the hours after it? every image gets a label, and that labeled table is what a prediction model trains on. flare-indexer takes an index of image timestamps and a flare catalog, looks forward from each image, and builds the table.
the first version had a small public api: an event matcher that finds flares inside a time window, a DatasetBuilder that walks through every image, and labeling strategies like BinaryThresholdStrategy, which labels an image 1 if any matched flare is at least a given goes class. flares are always compared by their numeric flux, never by their class names. sorted as text, “m9.0” looks stronger than “m10.0”, even though m10.0 is the stronger flare.
over the summer i added two more ways to slice the data. with sequences, several consecutive images become one sample, and the prediction window is measured from the sequence’s last image. with active-region labeling, only flares from the same region of the sun count, instead of any flare on the whole disk. one decision i cared about here: if an image’s active region is missing, it matches nothing. it never quietly falls back to whole-sun matching, because a wrong label that looks right is much harder to catch than an empty one.
in september the work shifted to real data, and most of the recent changes came from that. the package also got its new name, flare-indexer.
real data doesn’t arrive in tidy csv files. hmi solar images come as lists of file paths with the timestamp buried in the filename, and an integrated goes flare catalog has its own column names. i added loaders for both: one parses the timestamp out of each hmi filename, and the other adapts a goes catalog into the format the matcher expects. a raw file listing can also include files that aren’t images, or two paths with the same timestamp. the loader can either stop at the first bad row or skip bad rows, and when it skips, it reports exactly which rows it excluded and which timestamps were duplicated, so nothing disappears without a trace.
timestamps needed the same care. the catalog format i started with stores a flare’s start, peak, and end as bare hhmm times next to a date column. parsed on its own, “0331” isn’t 3:31 am. it’s a moment in 1970. the package always combines each time with its date, and catalogs with full timestamps are kept as they are, so a flare that starts before midnight and peaks after it stays on the right days.
the biggest change to matching was making the window rules explicit. by default, a flare counts if it peaks within [t, t + 24h) of the image time t. some published full-disk pipelines use the flare’s start time instead, and the legacy setup i validated against also counted the far edge of the window rather than t itself. both choices are now settings, event_time and interval_mode, and the defaults keep the original behavior so older code still produces the same labels.
to check that against something real, i wrote a validation script that builds a full-disk, m-class dataset from real hmi and goes data and compares it, timestamp by timestamp, with the output of a legacy labeling script. in that run, on 63,649 images and 14,401 flare events, the two agreed on 99.99% of labels. the remaining mismatches all traced back to one cause: the legacy window was one second short of 24 hours, so a flare starting at exactly t + 24h was left out there and counted here. flare-indexer keeps the exact 24-hour window, and the difference is documented instead of hidden.
sequences changed too. the original sequence mode groups the next few rows of the image index, whatever timestamps happen to be there, so unless you also set a cadence check, one missing image means a sequence silently covers a different stretch of time. the newer, time-based mode builds each sequence from an explicit grid instead: a start, a cadence, how much time one sequence covers, and how far the next one slides. if any image on the grid is missing, the whole sequence is skipped, never interpolated or filled in from a neighboring image. the grid also makes downsampling safe. from 30-minute images, you can ask for an hourly cadence and get only the on-the-hour images. a report shows how many sequences were kept, how many were skipped, and examples of the timestamps that were missing.
the last change was splitting labeling into stages. a strategy used to do two things at once: reduce a list of flares to one number, then turn that number into a label. now those are separate pieces, a reducer and a labeler, connected by a DatasetBuildingPipeline, and each stage can be run and inspected on its own. that matters in research, because the next question is usually a variation, like counting flares instead of taking the strongest one, or keeping the raw flux for regression instead of a yes-or-no label. a new reducer or labeler is a small class with one method, and the original strategies still work, because they’re now built from the same pieces.
this is roughly what a run looks like now, with the start-time window convention and hourly six-hour sequences:
from flare_indexer import (
DatasetBuilder, DatasetBuildingPipeline, MaxFluxReducer, BinaryThresholdLabeler,
)
pipeline = DatasetBuildingPipeline(
reducer=MaxFluxReducer(),
labeler=BinaryThresholdLabeler("M"),
)
builder = DatasetBuilder(
prediction_window=24,
strategy=pipeline,
event_time="start",
interval_mode="right_closed",
cadence="1h",
observation_window="6h",
sliding_window="3h",
)
dataset, report = builder.build("image_index.csv", "flare_catalog.csv", return_report=True)
looking back, most of this project was about making assumptions visible. which flare timestamp counts, where a window ends, what happens when an image or an active region is missing: each of those used to be a quiet decision inside a script. now each one is a named setting with a default, a documented behavior, and tests behind it. the package has 219 tests, and running them is how i know a change to one of those rules didn’t quietly change someone’s dataset.