OncoKB scraper microservice: guide

Research use only. This service reads the public oncokb.org website with a registered (free, academic) web account and returns OncoKB's evidence as JSON. It is not the licensed OncoKB API and its output is not for clinical reporting without review. Every response carries "research_only": true.

• Base URL: https://3-110-133-194.sslip.io (Lightsail Ubuntu-1, Mumbai)
• Interactive API reference (try calls in the browser): /docs. This page: /
• Every call needs the header X-API-Key: <key> (ask the service owner)

1. Which call do I use?

File uploads return a job. Poll GET /api/v1/jobs/{id} until status is done (or failed), then fetch GET /api/v1/jobs/{id}/report (JSON) or GET /api/v1/jobs/{id}/report.tsv (spreadsheet). Delete it when you're done: DELETE /api/v1/jobs/{id} (finished jobs are removed after 30 days anyway).

Always pass tumor_type when the diagnosis is known (OncoTree code or name: AML, MDS, CLLSLL, NSCLC, Melanoma...). Without it the answer is pan-cancer: oncogenicity is the same, but levels are listed for every tumour type and highest_*_level is not tumour-specific.

With a tumour type, only that tumour's levels come back: JAK2 V617F with AML has none (its Dx2 levels are for MPN, ET and PMF), with MPN it has Dx2. A parent code does not roll up its children (MNM, myeloid neoplasm, returns no levels for JAK2 V617F). So pass the actual diagnosis; when it is not settled yet (a myeloid panel ordered to work up a cytopenia), leave tumor_type out and read levels_any_tumor / level_associated_cancer_types. On the 84 Thermo files: 185 rows carry a level in some tumour type, 101 with tumor_type=AML; oncogenicity is identical either way.

2. How the assays differ, and why each needs its own path

2.1 Thermo Ion Reporter TSV (/api/v1/oncomine)

Upload one or more .tsv exports (multipart field files). Two kinds are recognised automatically:

• Selected export (what the lab reviewed and kept): taken as it is.
• Full / unselected export (> 300 rows or any ABSENT call): filtered the way Ion Reporter's default Oncomine filter chain does (thermo_like_v1):
• SNV / indel: call PRESENT, exonic or splice site, protein-changing, population AF <= 1 % (waived when Ion Reporter assigns an Oncomine Variant Class such as Hotspot or Truncating: ASXL1 G646Wfs*12, for example, sits in population databases as a homopolymer artefact), VAF >= 2 % (>= 1 % for Oncomine-classed calls), depth >= 15.
• Fusions, MET exon 14 skipping, EGFR vIII, KMT2A-PTD (RNA) and FLT3-ITD: kept when called PRESENT.
• CNV: Amplification when copy number >= 4 (flag low_level_amplification below 6). Deletion is sent to OncoKB only at copy number <= 0.5, because OncoKB "Deletion" means a deep (two-copy) loss and Ion Reporter's class also covers one-copy loss; one-copy losses are listed but not queried.
• TMB / MSI (CGP header): queried as TMB-High / MSI-High only when Ion Reporter calls them High / MSI-H with QC PASS.
• Checked against the lab: on 57 lab-selected exports the filter keeps 320 / 320 of the lab's variants.
• Transcript fix: Oncomine sometimes numbers a protein change on a different transcript than OncoKB (NF1 NM_001042492; MYD88 L252P is OncoKB's L265P; WT1, CSF3R). When OncoKB rejects the protein change, the variant is re-asked by its hg19 genomic position and OncoKB maps it itself (warning oncomine_transcript_differs:A->B).
• Flags per variant: possible_germline (VAF 40-60 % or >= 90 %), in_population_databases, low_level_amplification, lab_selected_outside_filter:....
• Patient names in the file name or ##Sample Name header are never stored: samples are named by run date + position (011026_3) or sample_N.

2.2 Illumina VCF (/api/v1/vcf)

Upload one VCF (.vcf or .vcf.gz, GRCh38; multipart field file). Form fields: mode (germline | somatic), tumor_type, sample (required when the VCF has several sample columns, e.g. tumour/normal), max_pop_af (default 0.01), include_synonymous (default false).

1. PASS calls in the coding sequence (+/-2 bp splice) of the 1,033 OncoKB-annotated genes are kept: about 1,200 of 5.2 million in a 30x genome. HLA and KIR genes are skipped (their normal allele differences look like truncations).
2. Local consequence on the NCBI MANE Select v1.5 transcript (missense / nonsense / frameshift / in-frame / splice / synonymous), computed on the server; nothing about the call leaves the machine at this step.
3. gnomAD v4.1 population filter (exomes + genomes, ~807,000 people), using the filtering allele frequency (95 % CI, highest ancestry group, as ACMG BA1/BS1 recommend): common_ba1 (>= 5 %) and common (>= max_pop_af) are listed with their frequency but not sent to OncoKB; rare and absent go on. Sites that failed gnomAD's own QC are flagged, never used to drop a call. The gnomAD subset (1.5 million sites, 24 MB) lives on the server; only public gene intervals were downloaded to build it, patient positions are never looked up anywhere.
4. OncoKB: SNVs by protein change; if OncoKB's transcript differs, retried by genomic coordinates. Indels and splice sites by genomic coordinates.

mode=germline: calls with the DRAGEN MOSAIC flag or VAF < 30 % are marked origin: mosaic and listed first (possible somatic / clonal haematopoiesis: the only germline-file calls where OncoKB's somatic annotation applies directly). mode=somatic: every call is origin: somatic.

Germline caveat. OncoKB describes what a variant does in a tumour. For a germline genome its answer means "would be oncogenic if somatic", not "inherited cancer risk". Germline classification belongs to ClinVar / ACMG. In our test genome the gnomAD filter removed 365 of 390 coding calls; all six earlier "Likely Oncogenic" hits (ATM H1380Y, XRCC3 T241M, KDR Q472H, EGFR R521K, LATS2, RAD17) were common polymorphisms.

2.3 What is not covered (yet)

• Illumina TSO500 / other vendors' TSV formats: send a VCF (mode=somatic) or single variants instead.
• hg19 VCFs: the VCF path is GRCh38 only (single-variant genomic_coordinates accept GRCh37 and GRCh38).
• Structural variants / CNV from a VCF (DRAGEN SV/CNV callers): not read; send them as single CNV/fusion calls.
• HRD, LOH, mutational signatures: OncoKB has no web page for them (UNSUPPORTED_BIOMARKER).

3. Calls and examples

Set B=https://3-110-133-194.sslip.io and K=<your key>.

Single variant

curl -s -X POST $B/api/v1/annotate -H "X-API-Key: $K" -H 'Content-Type: application/json' \
  -d '{"gene":"IDH1","alteration":"R132S","tumor_type":"AML"}'

Accepted alteration forms: R132S, p.Arg132Ser, c.394C>A:p.Arg132Ser, W288Cfs*12, ITD, PTD, Exon 19 in-frame deletions. Fusion: {"gene":"BCR","fusion_partner":"ABL1"} or {"gene":"BCR::ABL1"}. CNV: {"gene":"ERBB2","copy_number_type":"Amplification"}. Biomarker: {"gene":"Other Biomarkers","alteration":"TMB-High"}. Genomic: {"gene":"BRAF","genomic_coordinates":{"chromosome":"7","position":140753336,"reference":"A","alternate":"T","genome_build":"GRCh38"},"tumor_type":"MEL"}.

Response (abridged):

{"status":"annotated","research_only":true,
 "query":{"gene":"IDH1","alteration":"R132S","oncotree_code":"AML","oncotree_name":"Acute Myeloid Leukemia"},
 "gene":{"hugo_symbol":"IDH1","oncogene":true,"tumor_suppressor":false,"gene_summary":"..."},
 "variant":{"canonical_alteration":"R132S","hotspot":true,"curated_in_oncokb":true,"match":{...}},
 "oncogenicity":{"status":"Oncogenic"},
 "mutation_effect":{"known_effect":"Gain-of-function","description":"...","pmids":["..."]},
 "evidence":{"highest_sensitive_level":"Level 1",
             "treatments":[{"level":"Level 1","drugs":["Ivosidenib"],"matched_alterations":["R132"],"match_type":"positional",
                            "level_associated_cancer_types":["Acute Myeloid Leukemia"],"in_query_tumor_type":true,"pmids":["..."]}],
             "diagnostic":[],"prognostic":[],"fda":[],"other_tumor_type_levels":[]},
 "provenance":{"page_url":"https://www.oncokb.org/gene/IDH1/somatic/R132S/AML","oncokb_data_version":"v7.6","cached":false},
 "warnings":[],"errors":[]}

status values: annotated (OncoKB has curated it) · not_curated (gene is in OncoKB, this variant is not: "Unknown Oncogenic Effect") · not_found (gene not in OncoKB, or OncoKB doesn't recognise the alteration) · invalid (input rejected, HTTP 422: see errors[].code) · error (could not read oncokb.org, HTTP 503, retryable: true).

Thermo TSV

curl -s -X POST $B/api/v1/oncomine -H "X-API-Key: $K" \
  -F "files=@011026_1_selected_variants.tsv" -F "files=@011026_2_selected_variants.tsv" -F "tumor_type=AML"
# -> {"job_id":"8f2c...","poll":"/api/v1/jobs/8f2c...","report":".../report","tsv":".../report.tsv"}
curl -s $B/api/v1/jobs/8f2c... -H "X-API-Key: $K"            # status, progress, log
curl -s $B/api/v1/jobs/8f2c.../report -H "X-API-Key: $K"     # JSON: samples[] with variants[] and biomarkers[]
curl -s $B/api/v1/jobs/8f2c.../report.tsv -H "X-API-Key: $K" -o oncomine.tsv

Illumina VCF

curl -s -X POST $B/api/v1/vcf -H "X-API-Key: $K" -F "file=@sample.hard-filtered.vcf.gz" -F "mode=germline"
# a 30x genome (~475 MB) uploads in a few minutes, scans in ~15 s on the server; OncoKB time depends on the rare calls left

Report: calls[] with origin, gene, chrom/pos/ref/alt, consequence, protein_mane, genotype, vaf, gnomad_class, gnomad_filtering_af, gnomad_grpmax, status, oncogenicity, levels_any_tumor... Add ?full=true to get the complete OncoKB answer per call.

Lists

POST /api/v1/annotate/batch with {"items":[{...},{...}]} (<= 100, answered inline) or POST /api/v1/jobs with the same body (<= 20,000), then GET /api/v1/jobs/{id}?offset=0&limit=500 to page through results.

4. What we built, and why

Starting point

The admin's first scraper (2.0.0) read oncokb.org pages but had concrete defects: the wrong fields were returned for some page layouts, citations and hotspot flags were missing or wrong, fusions and CNVs were not handled, pan-cancer pages came back empty, data behind OncoKB's account wall was read as if it were an answer, and there was no caching, concurrency control or auth. The decision for this phase was scraper only (research use, free academic web account), not the licensed API.

Design decisions

1. Read what the page shows, cross-check it with the page's own data. Each page load also captures the JSON the site itself fetches (variantAnnotation, genes/lookup). Tiles (Oncogenicity, levels) are checked against table rows; a mismatch is a warning, never silently resolved.
2. Normalise input before touching the site. Gene aliases, 3-letter to 1-letter amino acids, p./c.:p. forms, fusion order (5'::3'), CNV words, OncoTree codes and names (closed list, with suggestions for typos). Inputs that can't be right (p.=, ?, unknown letters) are rejected with a code instead of loading a page.
3. Never read past the account wall. If oncokb.org asks for an account, the answer is LOGIN_REQUIRED, not data. A stale session makes the site fall back to anonymous mode silently (BRAF then shows "no information for this gene"), so any HTTP 401 from the site during a load marks the result SESSION_INVALID; the service logs in again and reloads. Errors are never cached.
4. Match type is explicit. A treatment row can apply to the exact variant, to a position (R132), or to a generic class (Oncogenic Mutations). Every row says which (exact / positional / generic), so "Level 1" via a generic row is visible as such (this is how ATM H1380Y looked Level 1).
5. Tumour type matters. With a tumour type the page lists that tumour's levels, and the OncoTree lineage decides whether a level written for a parent ("Myeloid Neoplasm", "All Solid Tumors") covers it. Without one, the pan-cancer page's tabbed tables (Therapeutic / Diagnostic / Prognostic / FDA) are read tab by tab.
6. Ask OncoKB only what it can answer. The OncoKB Cancer Gene List (1,247 genes, 1,033 annotated) answers "gene not in OncoKB" without a page load.
7. Lab files are the real input. Labs send Ion Reporter exports and DRAGEN VCFs, not clean variant lists, so adapters do what a scientist would do first: Thermo's filter chain for full exports, MANE consequence + gnomAD for VCFs, transcript reconciliation by genomic position.
8. Patient data stays here. Only gene, alteration, coordinates and tumour code go to oncokb.org. Uploads are deleted when the job ends; reports carry run-date sample IDs, never names; jobs expire after 30 days.

Speed and the always-on browser

One Chromium stays open with a pool of warm tabs (a reused tab loads a page in ~1 s against ~5 s for a new one), plus a keeper tab opened on demand for token checks and logins. Images and fonts are not downloaded, and a page counts as rendered as soon as its variantAnnotation data has arrived and been drawn. A watchdog relaunches the browser if it dies and checks the session token every 30 min (as the site does, via /api/account/tokens), logging in again with the configured account when it is invalid or about to expire; when many tabs hit an expired session at once, one login runs and the rest reuse it.

Memory

Each oncokb.org tab is a ~350 MB Chromium renderer (the site is a large single-page app), on top of ~650 MB fixed (Python, Playwright driver, browser and GPU processes). So: idle tabs are sent to about:blank between loads, the keeper tab is opened only for token checks and logins, and the watchdog drains the tabs and restarts the whole browser when service memory reaches 80 % of the systemd MemoryHigh (or ONCOKB_MEM_RESTART_MB), and after every ONCOKB_BROWSER_RESTART_AFTER (2000) page loads. /health shows memory_mb and memory_restart_mb. The first deploy ran 3 tabs under a 1.2 GB cap; the tabs alone exceeded it, the kernel throttled the service and page loads timed out for hours. That is why the numbers below are what they are. Pacing adapts: HTTP 429/5xx doubles the gap between page loads, success shrinks it back. Measured: 30 pages/min before, 89 pages/min with 4 tabs (default), 141/min with 8. Results are cached for 7 days (identical query = ~50 ms).

Components

Results on the lab's data (Oct 2026)

• 84 Ion Reporter files (78 Myeloid, 2 CGP, 4 Precision): 542 variants kept; 185 variants in 74 samples carry OncoKB levels (JAK2 V617F, NPM1, FLT3-ITD, IDH2 R172K, DNMT3A R882C...); 18 transcript differences repaired.
• One DRAGEN 30x germline genome: 5.2 M calls, 390 coding calls in OncoKB genes, 365 removed as common, 25 rare calls queried, none with a known oncogenic effect.

5. Operations

The OncoKB web session token currently expires 2026-11-14. oncokb.org returns the same token on re-login, so if /health shows session_token.renewal, renew the token on the oncokb.org account page; the service logs in again by itself once the old one stops working.

Tuning (in /etc/oncokb/env): ONCOKB_CONCURRENCY (tabs, default 4; 2 on this shared server, ~350 MB each), ONCOKB_RECYCLE_AFTER (100 here), ONCOKB_MEM_RESTART_MB (default 80 % of MemoryHigh), ONCOKB_MIN_INTERVAL_S (0.5), ONCOKB_CACHE_TTL_S (7 days), ONCOKB_JOB_RETENTION_DAYS (30), ONCOKB_MAX_UPLOAD_MB (1024).