# cCRE→gene regulatory views — Gosling Designer (visualizationhub.org) bundle

Six Gosling views from the `reanalysis0612tecounts` cCRE→gene map, ported from the marimo z-series
forks to **plain Gosling grammar JSON**. Bring-your-own-data: everything in `data/` is a verbatim copy
of an O-script output. No Python, no HiGlass, no marimo — upload the files, paste a spec, rewire one
URL.

All six specs are **validated against the official Gosling JSON schema** (`GoslingSpec`, gosling 0.3.0).

---

## 1 · Three-step workflow

1. **Upload** the files from `data/` (and `data_derived/`) into your Gosling Designer workspace.
2. **Open a spec** from `specs/` and paste it into the Designer code editor.
3. **Rewire the URL.** Every `data.url` in every spec begins with the literal placeholder:

   ```
   REPLACE_WITH_WORKSPACE_URL/<filename>
   ```

   Find-and-replace `REPLACE_WITH_WORKSPACE_URL` with the base URL your workspace assigns to the
   uploaded files. That is the **only** edit required to make a spec live.

Each spec is self-contained otherwise: the hg38 ideogram (24 rows), the per-program gene list, the
colour scales and the locus windows are all inlined as constants.

---

## 2 · What is in `data/` — all verbatim O-script outputs

| File | Rows | Size | Feeds |
|---|---:|---:|---|
| `O1g_arcs.tsv` | 823 | 98 KB | **locus, program ×2, atlas** — the master arc table |
| `O1e_credible_variants.tsv` | 14,789 | 1.7 MB | **locus** lollipops (fine-mapped eQTL PIP) |
| `O1d_crispr_pairs.tsv` | 9,223 | 1.2 MB | **crispr** arcs + effect-size skyline |
| `O1g_elements.bed` | 822 | 29 KB | *alternate* cCRE-rect source (BED, `score = 1000·ρ`) |
| `O1g_eqtl_pip.bed` | 10,005 | 340 KB | *alternate* lollipop source (BED, `score = 1000·PIP`) |
| `O1g_arcs.bedpe` | 822 | 46 KB | *alternate* arc source (BEDPE, `score = 1000·ρ`) |

The three `*.bed` / `*.bedpe` files are not referenced by any spec — they are included because Gosling
reads BED/BEDPE natively via the `csv` type, so they are drop-in alternates if you prefer integer
scores over the float columns in `O1g_arcs.tsv`.

### Headered versions (`*.headed.tsv`)

BED/BEDPE carry no header line, but Gosling's `csv` loader addresses columns **by name**, so these
files cannot be used as-is. Each therefore ships with a headered twin (originals kept untouched):

| File | Rows | Header |
|---|---:|---|
| `O1g_elements.headed.tsv` | 823 | `chrom, start, end, gene, score_rho1000, strand` |
| `O1g_eqtl_pip.headed.tsv` | 10,006 | `chrom, start, end, name, score_pip1000, strand` |
| `O1g_arcs_bedpe.headed.tsv` | 823 | `chrom1, start1, end1, chrom2, start2, end2, gene, score_rho1000` |

Named `.headed.tsv` rather than `.bed` deliberately: a header line makes them **invalid strict BED**
(bedtools/tabix would choke), so the extension signals they are for the name-addressed `csv` path only.
Scores are integers — `score_rho1000 = 1000·ρ`, `score_pip1000 = 1000·PIP` — so if you drive a colour
scale from them the domain is `[280, 670]`, not `[0.28, 0.67]`.

The four files the specs actually use (`O1g_arcs.tsv`, `O1e_credible_variants.tsv`,
`O1d_crispr_pairs.tsv`, `eqtl_ld_variants.csv`) **already carry headers** — nothing to do there.

### Tabix indexes (`*.tbi`)

| Index | Indexes | Built with |
|---|---|---|
| `O1g_elements.bed.gz.tbi` | `O1g_elements.bed` | `tabix -p bed` |
| `O1g_eqtl_pip.bed.gz.tbi` | `O1g_eqtl_pip.bed` | `tabix -p bed` |
| `O1g_arcs.sorted.bedpe.gz.tbi` | coordinate-sorted `O1g_arcs.bedpe` | `tabix -0 -s1 -b2 -e3` |

> ### ⚠️ These indexes are currently NON-FUNCTIONAL
> A `.tbi` encodes byte offsets into **bgzip blocks**, not into a plain `.bed`. The matching `.bed.gz`
> files were removed by request, so nothing can resolve these indexes as they stand. To make them work,
> regenerate the pair — it takes one line each:
>
> ```bash
> bgzip -c O1g_elements.bed  > O1g_elements.bed.gz  && tabix -p bed -f O1g_elements.bed.gz
> bgzip -c O1g_eqtl_pip.bed  > O1g_eqtl_pip.bed.gz  && tabix -p bed -f O1g_eqtl_pip.bed.gz
> sort -k1,1 -k2,2n O1g_arcs.bedpe | bgzip -c > O1g_arcs.sorted.bedpe.gz
> tabix -0 -s1 -b2 -e3 -f O1g_arcs.sorted.bedpe.gz
> ```
>
> Then query: `tabix O1g_eqtl_pip.bed.gz chr19:35200000-35420000`

**BEDPE caveat.** `O1g_arcs.bedpe` ships ordered by gene name, which tabix cannot index, so its index
was built on a **coordinate-sorted derivative** (same 823 rows, reordered). tabix has no bedpe preset,
so it indexes the **first anchor only** — a region query returns pairs whose *first* anchor falls in
the window. For true two-anchor indexing use `pairix`.

**What the index does *not* do here:** Gosling loads BED/BEDPE through the `csv` data type, which
fetches the **whole file** and ignores a tabix index. None of the six specs in `specs/` use these —
they are for tools that *do* consume tabix: IGV, samtools/bcftools, HiGlass, and Gosling's indexed
`vcf` / `gff` / `bam` readers (which accept an `indexUrl`). At these file sizes (≤ 340 KB) the plain
`csv` path is fast anyway; indexes matter only if you scale these tracks up.

### `data_derived/` — the single derived file

| File | Rows | Size | Why it exists |
|---|---:|---:|---|
| `eqtl_ld_variants.csv` | 14,789 | 2.1 MB | **eqtl_ld** view |

**Gosling has no join.** The eQTL-LD view needs per-variant geometry (`var_pos`, `var_pip`, `in_ccre`
— from `O1e_credible_variants.tsv`) *and* the real 1000G r² to the lead (`r2_cohort`, `r2_gold_eur` —
from `O1f_ld_pairs.tsv`), which live in two files keyed on `(link_key, varID == member_var)`. This file
is that join, plus the precomputed `cat` colour bin. Reconciles to **831 link_keys / 622 genes**;
11,546 of 14,789 variants carry a real r² (the rest had no 1000G match, binned as *not in 1000G*).

Everything else that used to be pandas now lives **inside the spec** as `dataTransform`.

---

## 3 · The six views

| Spec | View | Data | Default subject |
|---|---|---|---|
| `spec_locus.json` | One gene: ρ arcs + ρ cCRE rects + gene anchor + PIP lollipops | `O1g_arcs.tsv`, `O1e_credible_variants.tsv` | **MLPH** (8 cCREs, 232 variants) |
| `spec_program_linear.json` | One NMF program, whole genome: ideogram + arcs + ρ skyline | `O1g_arcs.tsv` | **NMF16** (50 genes, 65 arcs) |
| `spec_program_circular.json` | Same program as ρ chords on a chromosome ring | `O1g_arcs.tsv` | **NMF16** |
| `spec_atlas.json` | Every arc — Circos: ideogram ring + cCRE ring + chords by chromosome | `O1g_arcs.tsv` | all **823** arcs |
| `spec_eqtl_ld.json` | One credible set, variants coloured by real 1000G r² to the lead | `eqtl_ld_variants.csv` | **LDLRAP1** (174 variants) |
| `spec_crispr.json` | EPCrisprBenchmark pairs by confusion category + CRISPRi effect skyline | `O1d_crispr_pairs.tsv` | all **9,223** pairs (K562) |

### Retargeting a view (no rebuild needed)

Every subject is selected by a `filter` transform inside the spec — edit the list and the view moves.

```jsonc
// spec_locus.json / spec_eqtl_ld.json — change the gene
"dataTransform": [{ "type": "filter", "field": "gene", "oneOf": ["MLPH"] }]

// spec_program_*.json — change the program by swapping the gene list
"dataTransform": [{ "type": "filter", "field": "gene", "oneOf": ["ACTN1", "ADAM15", ...] }]
```

Gene lists for all **24 programs** (NMF16 50 · NMF9 47 · NMF6 46 · NMF19 31 · NMF17 30 · NMF18 30 ·
NMF23 30 · NMF21 29 · NMF4 28 · NMF12 27 · NMF22 27 · NMF13 26 · …) are in
`build/program_gene_lists.json` — paste any one into the `oneOf`.

If you change the gene in `spec_locus.json`, also update the `x.domain.interval` (the locus window) and
`chromosome`, since those are pinned per gene.

**CRISPR legibility note:** the benchmark is dominated by true negatives (TN 8,788 · FN 380 · TP 34 ·
FP 21). To show only the informative pairs, add to the arc and skyline tracks:

```json
{ "type": "filter", "field": "category", "oneOf": ["TP", "FP", "FN"] }
```

---

## 4 · Encoding invariants — do not "fix" these

Carried verbatim from `reference/z_gb_specs.py`; each was learned the hard way against gosling 0.3.0.

1. **Links are coloured on `stroke`, not `color`.** A thin/degenerate `withinLink` renders only its
   stroke outline (the fill area → 0), and an unset stroke defaults to **black** — which is why an
   earlier version of the circular chords came out uniformly black regardless of ρ.
2. **ρ colour pins the `domain` only; `range` is deliberately omitted.** Gosling renders a
   `type: "quantitative"` colour channel with its own perceptually-uniform (viridis) scheme and
   silently ignores a custom two/three-stop range. `RHO_DOMAIN = [0.28, 0.67]` matches the observed
   spread (actual: 0.287–0.665), so the ramp resolves the data instead of crushing it.
3. **At whole-genome scale a cis arc is sub-pixel** (median cCRE→TSS distance ≈ 0 bp, max ≈ 200 kb), so
   it collapses onto the baseline when fully zoomed out. That is why the whole-genome views pair the
   arc track with a **skyline** (one point per link) that stays legible at any zoom — the arcs resolve
   as you zoom into an arm. All tracks share one `linkingId`, so they pan/zoom together.
4. **Lollipops are two overlaid marks** — a `bar` STEM (0 → PIP) plus a `point` HEAD — on a left PIP
   axis `[0, 1]`. A bare point mark has no readable height scale.
5. **Nominal vs quantitative colour**: a `type: "nominal"` channel *does* honour custom
   `domain`/`range` (used for `evid`, `cat`, `category`, chromosome bands); a quantitative one does not
   (see #2).

### Constants inlined in the specs

- `RHO_DOMAIN = [0.28, 0.67]` (viridis, domain-only)
- hg38 chromosome sizes — 24 rows, inlined per view as the ideogram; `assembly: "hg38"`
- chromosome bands: `#3B4A5A` / `#8CA0B3` (alternating, so the ideogram reads as blocks not a rainbow)
- palette: NAVY `#10243F` · TEAL `#107A73` · GOLD `#B87A0E` · GREEN `#1E7D46` · RED `#AF3A2E` ·
  SLATE `#4A4E69` · MUTE `#6B7785` · GREY `#C9D3DD` · INK `#1B222B`

**TSS caveat (inherited):** the source `gene_coords` has no strand, so the TSS anchor is the gene
*start*; arcs and the caret do not imply transcriptional direction.

---

## 5 · `tilesets_higlass_only/` — included on request, but **not usable here**

| File | Size |
|---|---:|
| `O1g_eqtl_pip.hitile` | 8.3 MB |
| `O1g_eqtl_pip.beddb` | 1.6 MB |
| `O1g_arcs.bed2ddb` | 196 KB |
| `O1g_elements.beddb` | 168 KB |

Per the [Gosling data docs](https://gosling-lang.org/docs/data/), Gosling splits sources into two
classes:

- **Static files, no server** — `csv`/TSV/**BED**/**BEDPE**/GFF, `json`, `bigwig`, `bam`, `vcf`.
- **Tilesets, HiGlass server required** — `beddb`, `vector`, `multivec`, 2D `matrix`/`bed2ddb`.

So **`.beddb` / `.bed2ddb` / `.hitile` cannot be loaded by Gosling from a plain URL** — they need a
running HiGlass tile server, which is exactly what this bundle avoids. They are shipped only as a
fallback if you later stand one up. **The pure-Gosling path uses `data/` and nothing in this folder.**
Note that Gosling reads BEDPE directly as a `csv` source for `withinLink` arcs, so the 2D tileset buys
you nothing here.

---

## 6 · Provenance

`build/` holds the one-time scripts that produced this bundle. **They never run on the platform** —
they are included so every file is traceable.

| Script | Produced |
|---|---|
| `build/make_bundle.py` | `data_derived/eqtl_ld_variants.csv` (the join) and `build/program_gene_lists.json` |
| `build/gen_specs.py` | all six `specs/*.json` |

`reference/z_gb_specs.py` and `reference/z_ev_specs.py` are copies of the authoritative marimo builders
the specs were ported from — the source of truth for every encoding above.

### Source paths (originals, never modified)

```
/net/users/voy/nmf/data_itcr/cnTFeffects/reanalysis0612tecounts/
├── results/O1_ccre_gene_map/browser/   O1g_arcs.tsv, O1g_elements.bed, O1g_eqtl_pip.bed,
│                                       O1g_arcs.bedpe, *.beddb, *.bed2ddb, *.hitile
├── results/O1_ccre_gene_map/tables/    O1e_credible_variants.tsv, O1d_crispr_pairs.tsv,
│                                       O1_program_gene_map.parquet
└── genotype_ld/results/O1f_eqtl_ld_1kg/tables/   O1f_ld_pairs.tsv
```

`O1_program_gene_map.parquet` is **not** shipped — Gosling cannot read parquet, so program membership
was resolved at build time into the inlined `oneOf` gene lists (`build/program_gene_lists.json`).

Assembly: **hg38** throughout. Underlying counts: 823 arcs · 615 genes · 831 eQTL-supported links ·
10,006 PIP points · 24 NMF programs.