Skip to content

Cryo-EM

Domain: Structural biology — EMPIAR-10017 β-galactosidase micrographs via CryoPPP

Source: use_cases/cryoem/

Demo bundle: Download .tar.gz

DSAgt-assisted curation of cryo-EM data from the EMPIAR public archive (EMPIAR-10017 β-galactosidase micrographs via CryoPPP) — register curation tools, ingest cryo-EM quality knowledge, and build a micrograph-preprocessing pipeline.

Estimated time: ~40 minutes — this is the broadest demo (7 stages) and the least quick. It pulls a ~2 GB EMPIAR-10017 download, an arXiv PDF, and a full CryoPPP repo clone, then KB-ingests the whole repo (minutes on the local embedder) before any pipeline work. Best treated as an advanced, bring-time walkthrough rather than a 10-minute demo.

This guide documents a comprehensive DSAgt demonstration using cryo-electron microscopy (cryo-EM) data. It exercises knowledge ingestion, KB-guided pipeline design, code registration from third-party scripts, cross-collection knowledge synthesis, and multi-stage pipeline execution with domain-specific evaluation.

Prerequisites

  • DSAgt installed (uv sync --all-groups)
  • An agent platform installed and already authenticated (e.g., claude for Claude Code) — BYOA: dsagt writes no credentials. The default local embedder needs no API key.
  • ~2 GB disk space for the cryo-EM test data
  • Git installed

Setup

1. Download cryo-EM dataset

Download the EMPIAR-10017 (β-galactosidase) subset from the CryoPPP dataset — 84 micrographs.

mkdir -p demo_data/cryoem
cd demo_data/cryoem
curl -L https://calla.rnet.missouri.edu/cryoppp/10017.tar.gz -o 10017.tar.gz
tar xzf 10017.tar.gz
cd ../..

2. Download the CryoPPP paper

mkdir -p demo_data/cryoem/papers
curl -L https://arxiv.org/pdf/2304.02011 -o demo_data/cryoem/papers/cryoppp_paper.pdf

3. Clone the CryoPPP repository

mkdir -p demo_repos
git clone https://github.com/BioinfoMachineLearning/cryoppp.git demo_repos/cryoppp

4. Initialize a DSAgt project

dsagt init cryoem-pipeline --agent claude

(The default local embedder needs no key. To use a hosted embedder instead, set embedding.backend: api in ~/dsagt-projects/cryoem-pipeline/.dsagt/config.yaml and export EMBEDDING_API_KEY in your shell — never written to disk.)

5. Start the session

dsagt start cryoem-pipeline

Execution

1. Create a cryo-EM knowledge collection

Ingest the folder demo_repos/cryoppp/ into the knowledge base as a collection called "cryoppp".

Wait for the ingest job to complete, then:

Append the file demo_data/cryoem/papers/cryoppp_paper.pdf to the cryoppp collection.

Verify:

List all knowledge base collections.

Should show cryoppp.

2. Query the knowledge base for pipeline design

Search the cryoppp collection for guidance on creating an AI-ready data processing pipeline for cryo-EM micrographs. What quality parameters should I filter on, and what thresholds are recommended?

The agent should return chunks describing quality metrics: CTF resolution, defocus ranges, ice thickness thresholds, and motion statistics.

3. Register CryoPPP processing codes

Look at the scripts in demo_repos/cryoppp/ and register any data-processing or evaluation codes you find. Run --help on each script to discover its interface.

Verify:

Search the registry for cryo-EM codes.

4. Create a quality scoring code

Write a Python script that scores cryo-EM micrographs based on:
- CTF fit resolution (CTFMaxRes)
- Defocus range
- Ice thickness
- Motion statistics

Use the CryoCRAB 0-7 scoring scheme described in the cryoppp collection. The script should read a metadata CSV and output a scored CSV with quality_score and quality_tier (high/medium/low) columns. Save the script under codes/<name>/scripts/ and register it as a code.

The agent should search the knowledge base, write the script, and register it via save_code_spec.

5. Run the pipeline

Run the pipeline on the EMPIAR-10017 dataset in demo_data/cryoem/10017/:
1. Scan the directory to understand what's there
2. Profile the micrograph metadata
3. Run the quality scoring tool on the metadata
4. Summarize: how many micrographs fall into each quality tier?

6. Generate a datacard

Search for a skill that can generate a datacard for the processed cryo-EM data, then use it.

7. Reconstruct the pipeline

Reconstruct the pipeline from the execution records as a bash script.

Post-Conditions

  1. Knowledge base contains cryoppp collection with repo code, docs, and appended paper.
  2. Code registry includes the CryoPPP processing codes and the quality-scoring code.
  3. Quality-scored CSV exists with tier distribution.
  4. A datacard exists for the processed dataset.
  5. A reconstructed pipeline script is available.
  6. Code execution records in trace_archive/ document the full provenance chain.
  7. MLflow traces (in the serverless mlflow.db store) capture token usage, latency, and full request/response history.

What This Tests

DSAgt Capability Steps
Knowledge ingestion (folder) 1
Knowledge append (single file) 1
Semantic search 2
Code discovery via registry 3
Code registration 3, 4
KB-guided code generation 4
Code execution with provenance 5
Skill discovery and use 6
Pipeline reconstruction 7

Cleanup

dsagt rm cryoem-pipeline -y          # unregisters the project and removes its dir
rm -rf demo_data/cryoem demo_repos/cryoppp