Tutorial: Nextstrain Phylogenies for seasonal H1N1, H3N2 and B/Victoria Viruses

Part 2: Adding new sequencing runs and running the production pipeline

Elgin Akin

2026-08-10

From a new sequencing run to a reviewed build

flowchart LR
    A["New run FASTA + date<br/>→ lab SharePoint"] --> B["Copy down to<br/>source_prep/"]
    B --> C["python scripts/<br/>prepare_source.py"]
    C -->|READY| D["Copy 2 files<br/>→ source/"]
    G["GISAID vaccine.fasta"] --> D
    D --> E["snapshot_clean,<br/>dry run, run"]
    E --> F["Review, then<br/>upload by hand"]

The build reads exactly three files from source/:

source/JHH_sequences.fasta   ← built by prepare_source.py (never edit by hand)
source/JHH_metadata.txt      ← built by prepare_source.py (never open and save in Excel)
source/vaccine.fasta         ← downloaded from GISAID and reformatted by hand

You need Part 1 done first: the pekosz-nextstrain environment is built and active. The legacy manual method (cat + Excel) is kept in the Appendix for reference only.

Protect regulated data before you begin

Warning

The code is public; the data must never be. JHH run FASTAs, run_dates.csv, source/, source_prep/output/, fludb.db, results/ and snapshots/ are unpublished patient-sample data or licensed GISAID data. They stay on your computer and on the lab SharePoint.

  • Restricted data may only live in source/, source_prep/runs/, source_prep/output/, source_prep/run_dates.csv and tutorial/vaccine.fasta (plus the git-ignored build folders).
  • Keep the git hooks on (git config --get core.hooksPath prints .githooks). Never use git add -f or --no-verify.
  • The surveillance report in reports/ is confidential: named internal recipients only.
  • Only people with their own GISAID account may receive vaccine.fasta or anything made from it.
  • Using an AI coding agent? It must follow .agents/AGENTS.md: it never reads raw sequence files, never uploads and never pushes.

Get access first

  • Lab data (FluCrew): ask Dr. Andy Pekosz to add you to the FluCrew Team/SharePoint. It holds the lab master copy of runs/ and run_dates.csv.
  • Raw sequencing output (one folder per run, e.g. IV26Run8): the Mostafa Lab Influenza-Surveillance shared folder. Ask Dr. Heba Mostafa for access.
  • GISAID: your own individual account. Credentials are never shared.

Links and contacts are on the wiki: Step 1: Getting access.

Important

SharePoint is the master. Your source_prep/ folder is a working copy: change things on SharePoint first, then copy them down. If you change your local run_dates.csv, make the same change on SharePoint, or the next person will not have it.

The source_prep/ folder is where you work

nextstrain/
├── source_prep/                 ← the only place you work during preparation
│   ├── run_dates.csv            ← the ONLY file anyone types into (copied from SharePoint)
│   ├── runs/                    ← one FASTA per sequencing run (copied from SharePoint)
│   │   ├── ConsensusIV26Run7.fasta
│   │   └── ConsensusIV26Run8.fasta
│   └── output/                  ← written by the script; you copy 2 files to source/
│       ├── JHH_sequences.fasta
│       ├── JHH_metadata.txt
│       └── prep_report.txt      ← everything the script checked and found
├── scripts/prepare_source.py    ← the script (one command, no options needed)
└── source/                      ← the pipeline reads its inputs from here

bash scripts/build_env.sh created the empty runs/ and output/ folders. Git never tracks them.

Run names and run FASTA files

IV26Run8 = IV + 2-digit year (26 = 2026) + Run + run number in that year. A re-run may have one extra letter: IV25Run11b.

Rule Details
File name ConsensusIV26Run8.fasta (Mostafa Lab convention). The run name is read only from the file name.
One per run Two files for the same run stop the script. Copy the file itself, not the whole run folder.
Location Directly in runs/, not in a subfolder.
Content Plain-text FASTA. Never open and save it in Word, Excel or TextEdit.
Not read IV26_Run8.fasta, ConsensusIV26Run8.fasta.txt or .gz: rename or unzip to ConsensusIV##Run#.fasta.

FASTA headers connect sequences to specimens

Every header is > + specimen ID + _ + segment number:

>JH00001_4
ATGGAGAGAATAAAAGAGCTAAGAGACCTAATGTCGCAGTCCC...
  • Specimen IDs are JH + 2-digit year + number. No underscores inside the ID.
  • Controls are named after the run (e.g. IV25Run6NTC_3) and are left out automatically.
Segment Influenza A Influenza B
1 PB2 PB1
2 PB1 PB2
3–8 PA, HA, NP, NA, MP, NS PA, HA, NP, NA, MP, NS

flusort detects the type later in the pipeline, so the swap of segments 1 and 2 does not matter during preparation.

Part A: add the new run to SharePoint (once per run)

Done by whoever receives the new data, in about 2 minutes:

  1. Find the run’s folder in the Mostafa Lab shared folder, e.g. IV26Run8.
  2. Copy its consensus FASTA into runs/ on the lab SharePoint as ConsensusIV26Run8.fasta.
    • Exactly one FASTA per run. Unsure which file is the consensus? Ask the sequencing team. Don’t guess.
  3. Add one row to run_dates.csv on SharePoint, at the bottom:
sequencing_run,date,notes
IV26Run7,2026-06-08,
IV26Run8,2026-10-06,3 samples re-extracted      ← your new row

The date is the one the sequencing team gives you, written YYYY-MM-DD. Notes are optional. Saving from Excel: File > Save As > CSV UTF-8 (Comma delimited); never .xlsx.

Part B: build the two JHH files with one command

B1. Copy the latest files from SharePoint into source_prep/

  • All FASTA files from SharePoint runs/ → source_prep/runs/ (replace existing files).
  • SharePoint run_dates.csv → source_prep/run_dates.csv (replace the old one).

Tip

WSL: explorer.exe source_prep opens the folder in Explorer. Mac: in Finder press Cmd+Shift+G and paste the path to nextstrain/source_prep. If SharePoint files show a cloud icon, right-click → Always keep on this device first.

B2. Run the script

cd ~/nextstrain                      # or wherever your nextstrain folder is
conda activate pekosz-nextstrain
python scripts/prepare_source.py

It takes a few seconds and needs no options. It only reads runs/, run_dates.csv and source/, and writes into source_prep/output/.

Read the RESULT before you copy anything

The last lines say RESULT: READY or RESULT: STOPPED. Every line starts with a label:

Label Meaning → what you do
[OK] · [FIXED] All good, or harmless formatting fixed in the output (originals untouched) → nothing
[NOTE] Information, e.g. new or re-sequenced specimens → skim it
[WARNING] Handled, but a person should know → read it
[PROBLEM] Must be fixed; nothing is written until it is → follow the HOW TO FIX line
  • STOPPED: nothing was made. Fix each problem and run the command again until it says READY.
  • POSSIBLE CONTAMINATION (a negative control produced sequence): tell the sequencing team and Dr. Pekosz.

Full message reference: Step 1: Reading the result.

What the script decides for you

  • Controls (NTC, water…) are left out of both files.
  • A specimen sequenced in more than one run keeps one run: the one with the most segments; if equal, the most recent.
  • The same segment twice in one run file: identical copies merge; otherwise the longer one (fewest N) is kept, with a WARNING.
  • Short or low-coverage segments are kept; the pipeline filters them later and the script only reports them.
  • Nothing is ever changed in runs/, run_dates.csv or source/. It lists new, removed and moved specimens compared with source/.

JHH_metadata.txt is written for you, tab-separated, one row per specimen:

sequence_ID sample_ID sequencing_run date passage_history
JH00001 JH00001 IV26Run8 2026-10-06 vtm

The date comes from run_dates.csv. This replaces the old manual Excel and cat steps.

B3. Copy the two files into source/

Only after RESULT: READY, and after reading any [WARNING] lines, copy:

source_prep/output/JHH_sequences.fasta   →  source/JHH_sequences.fasta
source_prep/output/JHH_metadata.txt      →  source/JHH_metadata.txt
  • Replace the old files; don’t rename them.
  • Leave prep_report.txt where it is. It records what went into the build (run files, checksums, who ran it). Share it inside JHU only.
  • Stuck? RESULT: UNEXPECTED ERROR is a bug, not your fault: send everything printed (and prep_report.txt) to the maintainer.

Update vaccine.fasta only through authorized GISAID access

Usually done by the maintainer when WHO announces new vaccine strains. Using your individual GISAID account, follow Tutorial 3:

  1. Open Vaccine Reference Sequences.
  2. Create or update the workset seasonal influenza vaccine strains.
  3. Add all available segments for every required vaccine strain.
  4. Confirm the expected strains and segments, then download the FASTA.
  5. Reformat every header (next slide), then save as source/vaccine.fasta.

Important

Do not merge clinical headers into this file. Vaccine headers use a different, seven-field format required by fludb/scripts/upload_vaccine.py.

Vaccine headers must contain seven ordered fields

>Isolate name-Passage history | Isolate ID | Collection date | Passage history | Segment number | Type | Lineage

Illustrative example:

>A/Wisconsin/67/2022-E2/E3 | EPI_ISL_XXXXXXXX | 2022-03-15 | E2/E3 | 4 | A | H1N1
ATG...TAA
  • Passage history appears twice: appended to the isolate name in field 1 and again as field 4.
  • Type is A or B; lineage is H1N1, H3N2 or Victoria.

Check it (prints nothing when every header has seven fields):

grep '^>' source/vaccine.fasta | awk -F'|' 'NF != 7'

Record the download date, workset name, EPI_SET ID and your name in source/README.md (kept with the data, never committed).

Archive, dry-run, then run the production build

conda activate pekosz-nextstrain
snakemake --cores 8 snapshot_clean                # archive + clean the previous build
snakemake -n --cores 8                            # dry run: nothing executes
snakemake --profile profiles/default --cores 8    # full production run
  • snapshot_clean saves snapshots/{timestamp}.tar.gz, then deletes data/, results/, reports/, auspice/ and fludb.db. Build reports before cleaning.
  • No --configfile this time: the pipeline reads source/.
  • Always dry-run first and read what it plans to rerun.
  • Expect 10 to 20 minutes on a recent 8-core laptop. Each run downloads the current Nextclade HA/NA datasets.

Tip

Want every successful full build archived automatically? Add --configfile config/snapshot_clean.yaml.

Review every build, then upload by hand

Important

Uploads are always manual: never part of a Snakemake rule, a scheduled job, or an AI agent’s work.

  1. Open the builds locally: auspice view --datasetDir auspice/h3n2 (or auspice.us) and reports/surveillance_report.html.
  2. Work through the review checklist: run finished SUCCESS, plausible tip counts, newest run present (color by sequencing_run), vaccine strains present, sensible clades and dates, no patient-level fields.
  3. Upload to the private group first, then check online:
nextstrain login                                     # your own account
python scripts/nextstrain_upload_private_genomes.py  # read it before running
nextstrain remote list nextstrain.org/groups/PekoszLab
  1. Public release to PekoszLab-Public only with PI approval and the GISAID acknowledgement in config/description.md.

Before a production run: checklist

nextstrain/
└── source/
    ├── JHH_sequences.fasta
    ├── JHH_metadata.txt
    └── vaccine.fasta

Additional Resources

Back to the post

Appendix: preparing the JHH inputs by hand (legacy)

Before scripts/prepare_source.py, the two JHH files were built by concatenating run FASTAs with cat and curating the metadata in Excel. This appendix keeps that method for reference, and for the rare case where the script cannot be used.

Warning

Use prepare_source.py whenever you can. The manual method does none of its checks:

  • negative controls (NTC, water…) are not removed, and contamination is not flagged;
  • a specimen sequenced in more than one run keeps every copy;
  • run dates, headers and bases are not validated; and
  • there is no prep_report.txt recording which run files went into the build.

When the old hand-made files were rebuilt with the script (October 2026), two NTCs with full B/Victoria genomes had been included in the B/Victoria builds, one specimen was listed twice in the metadata, and hundreds of metadata rows had no sequences.

The steps: (1) concatenate the run FASTAs → (2) review the FASTA → (3) extract the specimen IDs → (4) build the metadata in Excel → (5) export, check and copy into source/.

Appendix 1–2: concatenate and review the FASTA

Work in an approved, access-controlled folder (e.g. source_prep/), never in a tracked folder. Rebuild from the complete, ordered list of original run files every time:

cat ConsensusIV26Run6.fasta ConsensusIV26Run7.fasta ConsensusIV26Run8.fasta > sequences.fasta

Do not use >> to append a new run to an old concatenated file. Record the run file names you used.

Open sequences.fasta in a plain-text editor (never Word or Excel) and confirm that:

  • the file begins with a header, and every header is >JH#####_N with N from 1 to 8;
  • sequence lines sit directly below their headers, and every intended run is included once.

Then run these checks. Both should print nothing:

grep '^>' sequences.fasta | sed -E 's/[[:space:]].*//' | sort | uniq -d                       # duplicate headers
grep -iE '^>.*(NTC|NEG|BLANK|WATER|H2O|CTRL|CONTROL|PTC|POS)' sequences.fasta                  # controls

A duplicate is usually a specimen re-sequenced in another run: keep one run’s copy. Any control that appears must be listed in config/exclude.tsv (ask the maintainer) so the builds drop it.

Appendix 3: extract the specimen IDs

The old helper scripts/jhh_unique_strains.py was removed from the repository when prepare_source.py replaced it. This one-line command does exactly what it did (the first word of each header, minus its _segment suffix, de-duplicated and sorted) with standard macOS/WSL tools:

{ echo "sequence_ID"; grep '^>' sequences.fasta \
    | sed -E 's/^>//; s/[[:space:]].*//; s/_[0-9]+$//' \
    | LC_ALL=C sort -u; } > unique_sequence_IDs.tsv

tail -n +2 unique_sequence_IDs.tsv | wc -l      # the number of metadata rows to expect

unique_sequence_IDs.tsv is a one-column TSV headed sequence_ID, ready to open in Excel.

Note

Need the original script? It is still in the git history: git show 36c1e158^:scripts/jhh_unique_strains.py > jhh_unique_strains.py (run it with python jhh_unique_strains.py --input sequences.fasta --output unique_sequence_IDs.tsv).

Appendix 4: build the metadata in Excel

Open unique_sequence_IDs.tsv in Excel and save a working .xlsx copy in the approved location. Add four columns so the five appear in exactly this order:

Column What to enter
sequence_ID Already supplied by step 3
sample_ID Same as sequence_ID (it becomes the strain name in every build, so it must be unique)
sequencing_run Run the specimen came from, e.g. IV26Run8
date Sequencing run date, YYYY-MM-DD
passage_history vtm (viral transport medium)
  • Before typing, format date as Custom yyyy-mm-dd, and every other column as Text so Excel does not reinterpret IDs.
  • Fill every row by hand from the run sheet; never guess a value from neighbouring rows.
  • Filter each column for blanks, highlight duplicate sequence_IDs (Conditional Formatting → Duplicate Values), and confirm the row count matches step 3.
  • Have a second reviewer check the workbook when possible.

Appendix 5: export, check and copy into source/

  1. With the metadata sheet active, File > Save As > Tab Delimited Text (.txt), named metadata.txt. Keep the reviewed .xlsx as the master copy.
  2. Reopen metadata.txt in a plain-text editor: five tab-separated columns, complete rows, YYYY-MM-DD dates.
  3. Check the columns from the terminal. It should print a single line ending in 5:
awk -F'\t' '{print NF}' metadata.txt | sort | uniq -c
  1. Copy both files into source/ under the names the pipeline reads:
sequences.fasta   →  source/JHH_sequences.fasta
metadata.txt      →  source/JHH_metadata.txt
  1. Continue with the vaccine file, snapshot_clean, the dry run and the run, exactly as in the main deck.

Important

These files hold the same unpublished JHH data as the script’s output: keep them in approved storage and out of version control.