⚠️ DEPRECATED — 2026-05-31. No longer maintained as a standalone project. All content and changes now live in the TruthSeeker iOS app (~/truth-seeker). Do not add features or content here — make changes in the app only.

newsfeed

Personal RSS + Twitter aggregation pipeline that produces two dark-mode HTML digests (a curated feed and a discovery feed), served on-device over Tailscale Funnel. Launchd-scheduled, swipe-feedback driven, with on-demand ElevenLabs TTS for individual items.

Status

Running in production. Phases 1–3 of the 2026-04-19 refactor shipped (ElevenLabs TTS + budget, curated/experimental feed split, Twitter as a first-class source). Phase 4 added the podcast tab. Phase 5 (2026-04-26) added per-card topic chips, a listen button on every compact card with single-instance audio, and Claude Haiku body cleaning that strips boilerplate from cached .txt files into a sibling .clean.txt. See History for completed fixes and Open questions for outstanding work. The plan that produced Phase 1–3 is at /Users/mark/.claude/plans/2026-04-19-newsfeed-refactor.md; the Phase 5 plan is at /Users/mark/.claude/plans/2026-04-26-newsfeed-quality-and-audio.md.

Scope

newsfeed answers one question: "what's worth my attention right now?" Twice a day, it fetches ~100 feeds (RSS + Twitter + podcast RSS), deduplicates, re-ranks with Claude, scores by source quality + recency + swipe history, and publishes three HTML pages optimized for iPhone dark-mode reading:

• newsfeed.html — curated. Only sources on the whitelist (whitelist.yaml). Mark's trusted daily read. A push to the Truth Seeker app's News tab with the top N items is sent at publish time.
• newsfeed-experimental.html — discovery. Same pipeline, no whitelist filter. Pull-only — never pushed.
• newsfeed-podcast.html — a dedicated podcast surface (its own render_podcast_html → templates/podcast.html.j2, not the shared digest.html.j2). A Latest | Shows segmented control toggles a newest-first episode tile grid and a by-show grid (one tile per subscription, including shows with no recent episode). Tiles play in-page through a single persistent bottom mini-player (one <audio>, big play / skip-15 / skip-30 / speed buttons) instead of navigating out; the episode's canonical URL survives only as a secondary ↗ link. Skips Claude rerank + Haiku summarization (title-only signal isn't worth the cost). Pull-only, no push, no TTS button.

All three pages share a DB, dedup, and upsert passes. Tab-nav in the header switches between them without reloading JS. A single swipe on the curated/experimental pages feeds back into the ranking of both on future runs; podcast cards have no swipe controls.

Out of scope: - Anything interactive beyond swipe feedback + TTS play. - Multi-user — everything is keyed to Mark's phone and whitelist. - Full-text archive. Items beyond ~180 days are pruned weekly. - Commenting, bookmarking, or cross-device sync.

Architecture

Pipeline (one "run")

fetch            ← RSS + Twitter + gnews queries (ThreadPoolExecutor, 4 workers)
  ↓
canonicalize     ← rapidfuzz URL normalize + shortlink HEAD resolve
  ↓
gnews-resolve    ← rewrite news.google.com wrapper URLs to real source
  ↓
upsert + tag     ← db.upsert_item; whitelist.classify_item sets feed=…
  ↓
fuzzy dedup      ← rapidfuzz on titles; kind-guarded (tweet≠article)
  ↓
semantic dedup   ← Claude Haiku over cluster leaders
  ↓
filter           ← recency cutoff, swipe-block, delivered-history, arxiv gate
  ↓
rerank           ← Claude Opus 4.6 (1M) over candidate pool
  ↓
score + allocate ← local score (rerank rank + recency + quality + bucket)
                    then per-feed allocate_digest() with bucket quotas
  ↓
enrich           ← og:image thumbnails, Haiku 1-sentence summaries
  ↓
render × 2       ← Jinja2 renders curated + experimental HTML
  ↓
publish          ← atomic write to ~/www/newsfeed{,-experimental}.html
  ↓
teaser           ← push (curated slate only)

Components

DB schema (SQLite, WAL)

One file: ~/Library/Application Support/newsfeed/newsfeed.db. Migrations tracked via PRAGMA user_version; current version = 9.

Interface

CLI (newsfeed.py)

All subcommands read config from ~/.config/newsfeed/ unless NEWSFEED_CONFIG_DIR=… is set. Default subcommand is run.

newsfeed.py run [--skip-deliver] [--prefetch-articles] [--rewrite-articles]
    Full pipeline. Launchd entry point. Acquires a flock — concurrent runs
    are rejected. --skip-deliver writes HTML previews to /tmp instead of
    ~/www/ and suppresses push. --prefetch-articles pre-warms article
    text + markdown caches for top items (off by default; cache misses are
    serviced on-demand via webpage-server's /article/<id>).
    --rewrite-articles runs Claude Haiku over each cached .txt to produce
    a sibling .clean.txt with ads/boilerplate stripped (off by default;
    lazy mode kicks in on the first reader-view click for any item that
    lacks a .clean.txt).

newsfeed.py fetch [SOURCE_NAME]
    Fetch one source (or all) and dump each Item as JSONL to stdout.
    Useful for debugging a single feed without touching the DB.

newsfeed.py render RUN_ID [--feed {whitelisted,experimental,podcast}]
    Re-render a prior run's HTML from DB state. Useful when a template
    bug shipped — fix the template, then re-render without re-fetching.

newsfeed.py podcasts-refresh
    Podcast-only refresh — fetch ONLY the podcast feeds and republish
    newsfeed-podcast.html from the DB. No LLM, no push, no news/papers.
    Cheap enough for a 5-minute launchd cadence (see "Podcasts" below);
    keeps the podcast tab current between the twice-daily full runs.

newsfeed.py send RUN_ID
    Re-push a prior run's teaser to the Truth Seeker app. No-op under NEWSFEED_DRY_RUN.

newsfeed.py backfill --hours N
    Re-score items from the last N hours without re-fetching. Prints the
    top 20 titles. Primarily for ranker / scoring experimentation.

newsfeed.py healthcheck
    Print status: source count, bucket count, items in DB, last run
    stats, claude-CLI version, config + db paths.

newsfeed.py server
    Minimal static server for ~/www on :8899. Fallback only — in
    production, webpage-server.py serves the digest and API.

Environment variables:

HTTP endpoints (served by webpage-server.py on :8899)

newsfeed does not run its own production HTTP server. The sibling webpage-server/ project serves ~/www/ and exposes JSON endpoints used by the digest frontend. They are in-scope for newsfeed because the Jinja template + the digest JS depend on them.

Streaming reader + audio (added 2026-05-05)

Reader mode uses one Haiku rewrite stream as the source of truth, fed in parallel into ElevenLabs' stream-input WebSocket so audio chunks start arriving ~500ms after the user taps play (vs 10–30s for the legacy blocking synth). Cache lives at ~/Library/Application Support/newsfeed/article-streams/{id}/:

text.txt              # Haiku-cleaned plain text (source of truth)
audio/0001.mp3 …      # ElevenLabs audio chunks in arrival order
audio/index.json      # {chunks: [{seq,file,bytes,start_ms,…}, …]}
meta.json             # {voice_id, model_id, audio_format, total_chars,
                      #  created_at}

First click pays the Haiku + ElevenLabs cost (~$0.06 input + ~$0.30 audio for a 2K-char article). Replay is free — both text.txt and audio/index.json are served straight to the browser without invoking either provider.

Frontend pieces (in templates/article.html.j2):

• marked.min.js (vendored at ~/www/vendor/marked.min.js) renders the streaming Markdown buffer once per requestAnimationFrame.
• A MediaSource attached to the existing <audio> element consumes base64'd MP3 chunks via appendBuffer, giving gapless playback.
• Each rendered paragraph is post-processed into <span class="sent" data-idx="N"> spans; audio.timeupdate toggles .sent.active to highlight the currently-playing sentence; tapping a sentence seeks the audio there.

Cache eviction: newsfeed.py weekly prune deletes per-item subdirs older than 60 days via article_stream.prune_streams.

The feedback token is written by deliver.ensure_feedback_token() to ~/Library/Application Support/newsfeed/feedback_token.txt, read by webpage-server at request time, and injected into the HTML via <meta name="newsfeed-token">. It is STABLE — minted only when the file is absent or empty, never rotated per run. Pages baking the token stay loaded in the Truth Seeker app for hours; when podcasts-refresh rotated it every 5 minutes, every search/subscribe/feedback call from an already-open page 401'd (2026-07-10). Guard test: tests/test_feedback_token.py.

whitelist.yaml schema

All sections optional; empty list or missing section are both fine. Missing file → curated feed is empty (warn, don't crash).

websites:
  - bloomberg.com        # exact hostname, no scheme, no leading www.
  - stratechery.com
twitter_users:           # @handle without the @; case-insensitive
  - dhh
  - patio11
rss_feeds:               # either a sources.toml `name` OR the feed URL
  - anthropic-blog
  - https://example.com/feed.xml
domains:                 # suffix match; '*.substack.com' or bare 'substack.com'
  - nytimes.com
  - "*.substack.com"

Match order in classify_item (first match wins):

1. Twitter author handle in twitter_users (only for source_kind == 'twitter').
2. Source name in rss_feeds, OR feed URL in rss_feeds.
3. Canonical hostname in websites.
4. Canonical hostname matches any suffix in domains.
5. Else experimental.

Data sources

Sources are configured in ~/.config/newsfeed/sources.toml (template: examples/sources.toml) and classified into 23 buckets with quotas normalized to 1.0 by the loader.

RSS (direct)

~80 curated feeds across ai-release, ai-research, red-teaming, alignment, tooling, long-context, new-architectures, open-weights, tech, national, world, science, finance, local, and weird. See sources.toml for the full list. Each source has a quality_seed (0.6–0.98) that warms the EMA in source_stats.

Google News RSS queries

[queries.<bucket>] tables in sources.toml expand at load time into virtual sources named gnews-<bucket>-<N>. Each query becomes https://news.google.com/rss/search?q=…&hl=en-US&gl=US. Wrapper URLs (news.google.com/articles/…) are resolved to the real source URL via gnews_resolve.py before dedup. Title-based fallback (_source_from_title) handles cases where resolution fails.

arXiv

arxiv-cs-lg, cs-ai, cs-cr, cs-cl, stat-ml, cs-ne, q-bio-nc, cs-dc, cs-hc, cs-se feeds via rss.arxiv.org. Papers are gated by [arxiv_gate] in config.toml: min_title_chars=40, require_abstract=true. The rerank prompt has a dedicated arXiv rule — low-signal titles are ranked low by design.

Twitter (via twitterapi.io)

One [[source]] with kind = "twitter" routes the TwitterFetcher into the pipeline. The fetcher pulls handles from whitelist.yaml → twitter_users (not from sources.toml — single source of truth). Replies and retweets are skipped by default; set extras = {include_replies = true, include_retweets = true} to keep them. Per-handle since_id state lives at ~/Library/Application Support/newsfeed/state/twitter/<handle>.since. Auth: ~/.config/newsfeed/twitterapi-io-key (chmod 600, one line). Provider is swappable via _TwitterClient indirection; RSSHub or X API v2 can drop in without touching _shape_tweet or TwitterFetcher.

Podcasts

Subscribed shows are configured in ~/.config/newsfeed/podcasts.yaml (template: examples/podcasts.yaml). Each entry is {name, category, feed_url, per_run_cap}; feed_url is blank in the template so the user can bulk-populate it by running ./scripts/resolve_podcast_feeds.py, which queries the iTunes Search API (https://itunes.apple.com/search?media=podcast&term=…) and writes the resolved feed URL back. podcast_sources.py then folds each subscription into cfg.sources as a SourceConfig with kind='podcast' and name='podcast-<slug>'. The PodcastFetcher reuses feedparser's iTunes-namespace support (enclosures, itunes:duration, itunes:image) — episodes outside the 72h window are dropped. Podcast items always route to feed='podcast' via a short-circuit in whitelist.classify_item, so they never land in the curated or experimental slates.

Per-episode dedup key (gotcha). upsert_item dedups on canonical_url. Some hosts (Captivate, e.g. Flirting with Models) put the show HOMEPAGE in every episode's <item><link>, so a bare link collapses every episode onto one DB row and only the first-seen episode is ever shown — a new drop silently never appears. PodcastFetcher therefore sets canonical_url = f"{link}#{entry_id}" (the stable per-episode id as a fragment): unique per episode, idempotent across refetches, and the ↗ link still navigates since browsers ignore the fragment. Guard test: tests/test_podcast_fetch_state.py. To confirm every show still serves itself (not just that it fetched), run ./audit_podcasts.py — see Operations.

The podcast slate renders through its own path: render.render_podcast_html flattens each episode (_podcast_episode_view) and pairs the run's episodes with the full subscription list from load_podcasts / to_source_configs (matched on source == 'podcast-<slug>') so the Shows grid lists every subscribed show, even ones with zero recent episodes. Show art is read from a network-free cache at ~/Library/Application Support/newsfeed/podcast-art.json (absent → tile shows the show's initial on an accent swatch). The page bakes the feedback token + run id into <meta> tags for the Phase-1d search/subscribe endpoints. Guard tests: tests/test_podcast_render.py.

Round-2 surface (templates/podcast.html.j2). Each episode tile carries the browsing data inline (data-*), and four interactions ride on it:

• Episode-description sheet. Tapping a tile (onclick="openDetail") opens an in-page slide-up overlay (#detail-sheet) with the show-notes (data-desc) and a Play button that plays through the same path as the corner play-badge — no navigation out, no new route. The corner play-badge stays a quick-play (stopPropagation so it doesn't also open the sheet). The description is body_text (the fuller HTML-stripped show-notes the fetcher persists, up to 4000 chars) falling back to snippet/summary for episodes that predate body_text. Because upsert_item keeps existing soft fields on collision, cmd_podcasts_refresh force-writes the freshly fetched body_text onto the row each refresh — that backfills the sheet for pre-existing episodes.
• Unfollow. The show-detail header carries an Unfollow control wired to POST /api/podcasts/unsubscribe (token-gated; atomic rewrite of the live podcasts.yaml, never truncated).
• Search dismiss-on-blur + Liquid Glass translucent chrome (search / segmented control / detail sheet / mini-player), behind @supports (backdrop-filter).

This template is NOT autoescaped — render._env() uses select_autoescape(["html","xml"]), which matches .html/.xml but not .j2, so every feed-controlled interpolation (episode title, show name, show-notes, art/canonical URLs) is escaped explicitly with the | e filter. A raw </" from a podcast feed would otherwise break the tag or inject. Guard tests: tests/test_podcast_detail.py (the four interactions + the title/desc escaping).

Refresh cadence (every 5 min). The twice-daily run republishes the podcast tab only at 8:00 and 18:00, so new episodes could sit unseen for hours. A separate lightweight job keeps it current: newsfeed.py podcasts-refresh fetches ONLY the podcast feeds (conditional GET — unchanged shows return HTTP 304 and cost almost nothing), upserts new episodes, and re-renders newsfeed-podcast.html from the DB (every feed='podcast' episode within a 7-day window, newest first) with no LLM rerank/summary and no push. It runs every 5 minutes via the com.mark.newsfeed-podcasts launchd job (examples/com.mark.newsfeed-podcasts.plist, StartInterval=300, RunAtLoad), installed by install.sh with the modern launchctl bootstrap gui/$UID syntax. Rendering from the DB rather than the current fetch is deliberate: a 5-minute poll mostly sees 304s, so a fetch-only render would blank every quiet show each cycle — the DB holds the accumulated recent episodes and the fetch just tops it up. The 7-day display window (vs the fetcher's 72h fetch window) keeps weekly shows visible between drops. Guard test: tests/test_podcasts_refresh.py (the 304-survival contract + window cutoff); healthcheck reports the last refresh on its own line.

Interests (Mark's taste profile)

~/.config/newsfeed/interests.md is appended to the rerank prompt. Free-form markdown. Describes focus areas, hard exclusions, and per-bucket taste.

Dependencies

Python (PEP 723 inline, resolved by uv run)

Declared in newsfeed.py's script header. ./newsfeed.py run resolves everything via uv automatically — no global pip install needed.

feedparser>=6.0.10    rapidfuzz>=3.5      jinja2>=3.1
tldextract>=5.1       httpx>=0.27         beautifulsoup4>=4.12
Pillow>=10.0          trafilatura>=2.0    markdown>=3.4
lxml-html-clean>=0.4  pdfplumber>=0.11    elevenlabs>=1.0.0
PyYAML>=6.0           tomli>=2.0 (py<3.11)

External CLIs

• claude — only probed by the status command's health check (claude --version). The five LLM stages no longer shell out; they drop events for dedicated poll sessions (see External services).
• tailscale — used by deliver._funnel_url() to discover the public https://<machine>.tail<tailnet>.ts.net/ base URL. LAN fallback kicks in if absent.

External services

• ElevenLabs — TTS synthesis. Key at ~/.elevenlabs-key (primary) or [elevenlabs] api_key in ~/.bb-config (fallback).
• twitterapi.io — tweet fetching. Key at ~/.config/newsfeed/twitterapi-io-key.
• Claude, via newsfeed-* poll sessions — rerank, semantic dedup, summarize, article rewrite, and article stream each drop hash-keyed events for a long-lived poll session (newsfeed-rerank, newsfeed-semantic-dedup, newsfeed-summarize, newsfeed-rewrite, newsfeed-stream): newsfeed-rerank rides the poll Opus default, the other four are pinned to Haiku by poll-bringup. Auth is the poll session's Claude Code subscription; newsfeed does not manage an Anthropic key directly.

Related projects

• ~/webpage-server — required in prod. Serves ~/www/ on :8899 and hosts the /api/feedback, /api/tts, /api/tts/status, /api/items endpoints. Must be running for the digest frontend to function.

Configuration

All configuration lives under these paths. install.sh seeds the ~/.config/newsfeed/ templates from examples/ on first run and leaves existing files alone.

Example — minimal config.toml

phone = "+17076556006"
digest_size_html     = 1000
digest_size_imessage = 6
recency_cutoff_hours = 24

rerank_model        = "claude-opus-4-6[1m]"
haiku_model         = "claude-haiku-4-5-20251001"

[delivery]
public_url_base = "auto"     # "auto" | "lan" | explicit URL
webpage_port    = 8899

[tts]
provider                  = "elevenlabs"
voice_id                  = "<paste from list_voices()>"
model                     = "eleven_flash_v2_5"
fallback_model            = "eleven_turbo_v2_5"
daily_char_budget         = 50000
daily_char_budget_testing = 5000
testing                   = false
# Streaming TTS (additive; falls back to one-shot synth on failure).
streaming_enabled          = true
streaming_model            = "eleven_turbo_v2_5"
streaming_output_format    = "mp3_44100_128"
streaming_voice_id         = "<defaults to voice_id above>"
streaming_chunk_schedule   = [120, 160, 250, 290]
streaming_idle_keepalive_s = 15      # < 20s ElevenLabs WS inactivity timeout
streaming_max_input_chars  = 16000
streaming_replay_speed     = "fast"  # "fast" | "normal" (cache replay)

[bucket_quotas]
"ai-release" = 0.08
national     = 0.06
# … normalized to 1.0 by the loader

See examples/config.toml for the full authoritative template.

Deployment

cd ~/newsfeed
./install.sh

install.sh is idempotent. It:

1. mkdir -p ~/.config/newsfeed and copies each of config.toml, sources.toml, interests.md only if the destination does not already exist (hand-edits are never clobbered).
2. mkdir -p the state dir, log dir, and ~/www/newsfeed/ archive dir.
3. Copies examples/com.mark.newsfeed.plist to ~/Library/LaunchAgents/com.mark.newsfeed.plist. Plist is copied not symlinked — launchd distrusts symlinked plists.
4. Unloads + loads the launchd job. Schedule: 8:00 and 18:00 local daily (StartCalendarInterval). RunAtLoad=false, KeepAlive=false.

Secrets (manual, one-time)

# ElevenLabs
echo "sk_…" > ~/.elevenlabs-key && chmod 600 ~/.elevenlabs-key
# twitterapi.io
echo "tapi_…" > ~/.config/newsfeed/twitterapi-io-key && chmod 600 ~/.config/newsfeed/twitterapi-io-key
# Seed whitelist (otherwise curated feed is empty)
"$EDITOR" ~/Library/Application\ Support/newsfeed/whitelist.yaml

Neither key goes in git — .gitignore at the monorepo root blocks .bb-config, api_key, and token.json patterns.

Uninstall

launchctl unload ~/Library/LaunchAgents/com.mark.newsfeed.plist
rm ~/Library/LaunchAgents/com.mark.newsfeed.plist
# Data survives — remove manually if desired:
# rm -rf ~/Library/Application\ Support/newsfeed
# rm -rf ~/www/newsfeed ~/www/newsfeed-experimental ~/www/audio

Operations

Run once, immediately

launchctl start com.mark.newsfeed          # via launchd
./newsfeed.py run                           # direct
NEWSFEED_DRY_RUN=1 NEWSFEED_VERBOSE=1 ./newsfeed.py run   # smoke test

Logs

tail -f ~/Library/Logs/newsfeed/newsfeed-$(date +%Y%m%d).log
tail -f ~/Library/Logs/newsfeed/launchd.log

Healthcheck

./newsfeed.py healthcheck
# sources configured / buckets configured / items in db / last run stats
# / claude-cli status / rerank model / phone

Audit podcast feeds (does every show still serve itself?)

./audit_podcasts.py
# one row per configured show: FLAG | feed's channel title | newest episode + age
# exits nonzero if anything is flagged

Guards the two ways the Podcasts tab has gone wrong: a feed slug silently reassigned to a DIFFERENT show (art19 handed "Machine Learning Street Talk" to "Machine Learning Podcast"), and a show whose newest episode is staler than you'd expect. MISMATCH = the feed's <channel> title no longer matches the configured name (wrong show, or a rebrand to reconcile); UNVERIFIED = title wouldn't parse; FETCH FAIL = unreachable. Logic is unit-tested in tests/test_podcast_audit.py. Run it whenever a show looks off.

Re-render a run without re-fetching

sqlite3 ~/Library/Application\ Support/newsfeed/newsfeed.db \
  "SELECT id, started_at, slot FROM runs ORDER BY id DESC LIMIT 5;"
./newsfeed.py render 27 --feed whitelisted
./newsfeed.py render 27 --feed experimental

Re-push a prior run's teaser

./newsfeed.py send 27

Re-delivers the run's curated teaser as a push to the Truth Seeker app's News tab.

Re-score recent items (no fetch)

./newsfeed.py backfill --hours 24

TTS budget + status

curl http://localhost:8899/api/tts/status
# {"enabled": true, "budget": 50000, "used_today": 1234, "remaining": 48766,
#  "rewrite_chars_today": 89400, "rewrite_count_today": 12}

sqlite3 ~/Library/Application\ Support/newsfeed/newsfeed.db \
  "SELECT date(ts, 'localtime') AS d, SUM(char_count), SUM(cost_cents)
   FROM tts_log GROUP BY d ORDER BY d DESC LIMIT 7;"

Article rewrite stats (Haiku body cleaning)

sqlite3 ~/Library/Application\ Support/newsfeed/newsfeed.db \
  "SELECT date(ts, 'localtime') AS d, COUNT(*) AS n,
          SUM(input_chars) AS in_c, SUM(output_chars) AS out_c,
          AVG(duration_ms) AS avg_ms, SUM(ok) AS ok_n
   FROM article_rewrite_log GROUP BY d ORDER BY d DESC LIMIT 7;"

# Failure breakdown (validation rules + claude_cli_error)
sqlite3 ~/Library/Application\ Support/newsfeed/newsfeed.db \
  "SELECT reason, COUNT(*) FROM article_rewrite_log
   WHERE ok=0 AND date(ts) >= date('now','-7 days')
   GROUP BY reason ORDER BY 2 DESC;"

Enable TTS on webpage-server

TTS is default-off. Set TTS_ENABLED=1 in the webpage-server process environment (not the newsfeed pipeline process — the pipeline never synthesizes).

TTS_ENABLED=1 ~/bin/webpage-server.py   # or update its plist EnvironmentVariables

Rollback a bad schema migration

DB migrations are additive (ALTER TABLE … ADD COLUMN) and safe to keep after a code rollback. Concretely: - Rolling back Phase 1 leaves tts_log orphaned (harmless). - Rolling back Phase 2 leaves items.feed orphaned (default 'experimental'). - Rolling back Phase 3 leaves items.kind orphaned (default 'article'). - Rolling back Phase 5 (Haiku rewrite, schema v9) leaves article_rewrite_log and the <sha1>.clean.txt cache files orphaned. /api/tts/status will report rewrite_chars_today=0 because the webpage-server's getter is wrapped in try/except for older DBs.

No manual down-migration needed; revert code and re-deploy.

Pruning (automatic)

Every Monday morning run triggers: - db.prune(days=180) — drop body_text on items older than 180 days. - deliver.prune_archive(keep=60) — both archive dirs. - thumbnails.prune_thumbs(keep_days=30). - tts.prune_audio(keep_days=60). - Per-source update_source_quality_ema.

Testing

Current

• Playwright WebKit viewport screenshots (desktop + mobile dimensions).
• curl-based API endpoint testing.
• Manual verification on iPhone via the Tailscale Funnel URL.

Needed

• iOS Simulator testing via Xcode (xcrun simctl) — Playwright WebKit on macOS is close to iOS Safari but differs on touch events, safe-area insets, autoplay policy, and memory pressure.
• Automated TTS quality checks (extracted article length > threshold).
• Thumbnail-coverage regression tracking (% per run).
• Infinite-scroll load testing (scroll through all ~1000 items).
• Audio playback testing on iOS (autoplay policy, format compatibility).

Open questions

O1 — Missing thumbnails (HIGH)

• Impact: ~75% of items have no thumbnail (719/961 in run 27).
• Root cause: All gnews-* sources return 0 thumbnails. Google News RSS doesn't include <media:thumbnail> or <enclosure>. Non-gnews sources (HN, MarketWatch, Seeking Alpha, FT) have 60–100% coverage.
• Proposed fix: Background og:image pass after ingestion for picked items. Rate-limit fetches, cache results, fall back to favicon or domain icon, graceful degradation when sites block scraping.

O2 — TTS transcribes wrong content (HIGH)

• Impact: Arxiv abstract pages, paywall stubs, and redirect/index pages get transcribed instead of the real article.
• Specific cases:
• Arxiv: trafilatura extracts the abstract page, not the paper. Fix: follow the /pdf/<id> link or use the arxiv API.
• Paywalled sites: trafilatura gets the paywall nag. Fix: DDG cache or Google cache as fallback.
• Archive/index pages: URLs point to topic or RSS landing pages, not individual articles.
• Current mitigation: URL-type dispatch in tts.py (arxiv API, PDF extraction, GitHub README extraction).
• Proposed fix: Content-type aware TTS pipeline — detect URL type, apply URL-specific extraction, validate quality (min length, not boilerplate), fall back to DDG snippet on failure.

O3 — TTS cold-start 10–30s (MEDIUM)

• Impact: First click takes 10–30s (article fetch + ElevenLabs latency). Intermittent failures without client retry.
• Root cause: fetch_full_article() has a 10s timeout — slow sites eat the budget. ElevenLabs adds ~5–15s. No client-side retry on 5xx.
• Current mitigation: --prefetch-articles pre-warms the article cache for top items.
• Proposed fixes: UI loading/progress indication, client-side exponential-backoff retry on 5xx, post-ingestion article pre-fetch for top items, shorter TTS model option for long articles.

O4 — iOS Safari testing (MEDIUM)

• Impact: Playwright WebKit on macOS differs from iOS WebKit on touch events, safe-area insets, autoplay policy, memory pressure, and IntersectionObserver + rubber-band scrolling.
• Proposed fix: Install Xcode iOS Simulator. xcode-select --install, download iOS simulator runtime, boot iPhone 16 Pro via xcrun simctl, open the digest URL in mobile Safari, automate via xcrun simctl openurl.

O5 — Fallback graphic for non-image items (LOW)

• Impact: Items with no thumbnail show empty space / broken layout.
• Proposed fix: Domain favicon fallback (https://www.google.com/s2/favicons?domain=DOMAIN&sz=128), colored placeholder with topic label, or collapse the image area entirely.

O6 — Full article pre-fetch pipeline (LOW)

• Impact: TTS cold-start is slow because article text is fetched on-demand.
• Proposed fix: Post-ingestion step fetches full article text for top-scored items and stores it in body_text or a new full_text column. TTS then only needs to call ElevenLabs. Run as a background job rather than gated behind --prefetch-articles.

Carry-overs from Phase 1–3 refactor

• Voice ID placeholder REPLACE_ME_AT_FIRST_RUN still in use until Mark runs python -c "from tts import list_voices; list_voices()" and edits config.toml.
• twitterapi.io is the current Twitter provider. Fallback path to RSSHub is documented in the refactor plan but not implemented.
• ElevenLabs cost-per-char constants in tts.py are best-guess and should be re-verified against current pricing.

History

2026-07-26 — Coral purge

Truth Seeker's News and Podcasts tabs are WKWebViews onto newsfeed.html and newsfeed-podcast.html, and its design system bans coral outright. Coral was the universal accent here, so the purge lands in this repo.

• vendor/theme.css is now in the repo. It had only ever existed at ~/www/vendor/theme.css — unversioned, untested, and the actual home of the coral --accent. install.sh now copies it into place, and a test asserts the deployed copy has not drifted from the source.
• Palette. --accent → #339cff (Truth Seeker's vendored blue), --link folds into --accent, --pnl-down → a true red #e02e2a, and a new --mark-rgb: 181 137 0 drives ::selection, so yellow is the one highlight colour. --source-red is retired from digest.html.j2 and podcast.html.j2; its jobs now name themselves — source lines and show names take --fg-soft, the CURATED badge and playback controls take --accent, unfollow takes --danger.
• BUCKET_ACCENTS re-hued. Nine AI-family entries sat in the coral band; each rotates hue only, keeping its saturation and lightness exactly, so the family reads gold → chartreuse → moss instead of amber → coral. No moved entry lost contrast against #18160f, and the tightest pair in the palette went from ΔE 5.9 to 7.0.
• webpage-server.py's mirrored copy was 23/23 out of sync and would have kept serving coral to the digest's infinite scroll. Resynced, and tests/test_no_coral.py now fails if the two ever diverge again.
• The guard is a band, not a blocklist. is_coral() rejects any hue in 335°–26° at saturation ≥ 0.18, so coral cannot return under a new literal. An earlier draft also required a minimum lightness; a vermillion walked through it, so lightness is deliberately not part of the predicate.

Known gap, deliberately not addressed: both templates override the theme's warm --bg with #000 (from a9e4e49, "black background + drop per-article category banners"). Mark set that on purpose — the warm #18160f ground read as "booger brown" on this page. It is his call, already made, and it is not a coral question.

Follow-up the same day, after checking the deployed files rather than the source:

• The cache-buster was never bumped, so none of the above reached a client that had loaded the page before. All three templates still linked /vendor/theme.css?v=2026-05-07; the query string is the only signal a browser or WKWebView gets that the file changed. The version is now derived — render.theme_version() hashes vendor/theme.css and _env() hands it to every template as a Jinja global, so it moves whenever the stylesheet does and there is nothing left to forget. A test asserts each rendered page carries the current hash; a second forbids a hand-typed ?v= in any template, since three literals agreeing with each other is exactly what passed while all three were stale.
• test_no_coral.py no longer strips comments before scanning. The acceptance check is a plain grep over shipped assets, and vendor/theme.css ships comments and all, so two lines documenting the retirement by quoting the retired hex would have read as a violation. The rationale stays; it now says "the retired coral accent" instead of naming it. The blocklist itself is built from rgb triples via _spellings(), which also covers the 3-digit form and keeps the banned literals from being spelled anywhere in the repo.
• 1,800 archived snapshots under ~/www/<feed>/ still carried coral. publish_html drops a timestamped copy of every render there and they are served, but they cannot be regenerated — newsfeed.py render stamps with now, and rebuilding one from today's DB would invent a history the page never had. They were moved (not deleted) to ~/Library/Application Support/newsfeed/archive-pre-decoral/. Everything written since is coral-free, and the deployed-asset test now scans the archive directories so it stays that way.
• newsfeed-experimental.html was live, coral, and on the stale stylesheet version too. Re-rendered; it is covered by the same tests now.

2026-04-25 — Quality + design overhaul (4 phases)

Plan: /Users/mark/.claude/plans/2026-04-25-newsfeed-quality-and-design.md.

• Phase 1: Whitelist + classify_item correctness. googlenewsdecoder added to PEP 723 deps so gnews_resolve.resolve() actually runs on cron-spawned uv run --script venvs (was silently failing → ~21k misclassified items in experimental). whitelist.yaml extended with Reuters, AP, Bloomberg, Economist, LessWrong, ACX/SSC, Marginal Revolution, Stratechery, The Information, Commonwealth Beacon, Longreads, Harper's, The Atlantic, Noahpinion. classify_item falls back to title-suffix host extraction when canonical host is a Google wrapper. cmd_healthcheck prints gnews resolver: OK / FAIL and cache size. Backfill script at scripts/backfill_gnews_classify.py (cache-only path is safe to re-run; --cache-only flag bypasses fresh resolves).
• Phase 2: Dedup hardening. _normalize_title_for_dedup() strips trailing " - " / " | " / " — " boilerplate before fuzzy comparison so wire stories collapse across publishers. cluster_titles() compares against the top-3 cluster members (not just the leader) for transitive matches. dedup_by_canonical_url() runs as a pre-pass before fuzzy clustering. db.recently_delivered_urls() unions canonical_url + meta_json resolved_url so Phase-1 backfilled items don't re-send.
• Phase 3: Less-clumpy ranking. interleave_for_delight() replaces round-robin-by-bucket. Multi-axis fitness scoring with penalties for same-source within same_source_min_gap (default 4), same-bucket beyond same_bucket_run_max (default 2), same-kind 3+ runs, and thumbless adjacency. Position-0 lock pins the rank-0 item. Wildcard slot every wildcard_every_n (default 7) pulls a tail item forward. Deterministic — random.Random(f"newsfeed-{run_id}") only used for tie-breaking. Backward-compat alias interleave_by_bucket(items, max_run=2) delegates to the new function. Tunables in ~/.config/newsfeed/config.toml [ranking].
• Phase 4: Editorial-broadsheet redesign. templates/digest.html.j2 rewritten end-to-end. Paired serif/sans typography (display serif "New York", "Iowan Old Style"... for headlines; SF Pro for body and meta). Per-bucket accent palette in render.py:BUCKET_ACCENTS (23 entries grouped by topic family — AI = gold→chartreuse→moss since the 2026-07-26 coral purge below; News = cool blue/slate; Tech/science = violet; Boston-local + weird = green/mustard; Finance = gold), set per card via inline style="--accent: ...;". Bucket label promoted from <span class="chip"> inside meta row to dedicated <div class="bucket-label"> above headline. Thumbnail moved to right column (140×120 desktop, 96×120 mobile). Hairline separators every 4 cards (not every card). Hover: subtle background flush + headline underline. Listen + restart buttons moved to bottom-right of thumbnail. webpage-server.py mirrors BUCKET_ACCENTS and emits bucket_accent on /api/items so JS infinite-scroll cards match SSR ones.

2026-04-19 — Phases 1–3 of the newsfeed refactor

Plan: /Users/mark/.claude/plans/2026-04-19-newsfeed-refactor.md.

• N1: Curated vs experimental feed split. Two HTML outputs (newsfeed.html, newsfeed-experimental.html) off a single pipeline; whitelist.yaml drives classification (websites, twitter_users, rss_feeds, domains). push fires on curated only.
• N2: Twitter as a first-class source. fetchers/twitter.py via twitterapi.io; handles come from whitelist.yaml → twitter_users; per-handle since_id state at state/twitter/<handle>.since; tweet card render; on-demand TTS like articles.
• N3: ElevenLabs TTS. Replaced OpenAI TTS. eleven_flash_v2_5 default, eleven_turbo_v2_5 fallback. Key at ~/.elevenlabs-key (primary) or ~/.bb-config [elevenlabs] api_key (fallback).
• N4: Strict on-demand TTS with budget gate. TTS_ENABLED=1 required; daily_char_budget (50K chars default, 5K testing); tts_log records every synth; 429 on budget exhaustion; no pre-generation during ingestion.

2026-04-16 — Early reliability fixes

• P1: iOS Safari blank screen. 961 items (1.6MB HTML) killed the renderer. Fix: server-side pagination via /api/items + IntersectionObserver infinite scroll. Initial HTML is 60 items (~120KB), rest lazy-loaded 40 at a time.
• P2: "news.google.com" as source label. Fix: three-layer defense on gnews wrapper URLs — entry.source.href at fetch time, _short_host() from the resolved URL, and _is_google_host() filter with a title-based _source_from_title() fallback. 0/961 items showed the wrapper host after deploy; legitimate google properties (blog.google, deepmind.google) correctly pass through.
• P3: TTS URL resolution. Fix: fallback chain (resolved_url → raw_url → canonical_url), gnews resolution attempt, DDG site-scoped search fallback. Further quality issues tracked under O2.