Add baoyu-skills package
This commit is contained in:
214
baoyu-skills/skills/baoyu-danger-x-to-markdown/SKILL.md
Normal file
214
baoyu-skills/skills/baoyu-danger-x-to-markdown/SKILL.md
Normal file
@@ -0,0 +1,214 @@
|
||||
---
|
||||
name: baoyu-danger-x-to-markdown
|
||||
description: Converts X (Twitter) tweets and articles to markdown with YAML front matter. Uses reverse-engineered API requiring user consent. Use when user mentions "X to markdown", "tweet to markdown", "save tweet", or provides x.com/twitter.com URLs for conversion.
|
||||
version: 1.117.3
|
||||
metadata:
|
||||
openclaw:
|
||||
homepage: https://github.com/JimLiu/baoyu-skills#baoyu-danger-x-to-markdown
|
||||
requires:
|
||||
anyBins:
|
||||
- bun
|
||||
- npx
|
||||
---
|
||||
|
||||
# X to Markdown
|
||||
|
||||
Converts X content to markdown:
|
||||
- Tweets/threads → Markdown with YAML front matter
|
||||
- X Articles → Full content extraction
|
||||
|
||||
## User Input Tools
|
||||
|
||||
When this skill prompts the user, follow this tool-selection rule (priority order):
|
||||
|
||||
1. **Prefer built-in user-input tools** exposed by the current agent runtime — e.g., `AskUserQuestion`, `request_user_input`, `clarify`, `ask_user`, or any equivalent.
|
||||
2. **Fallback**: if no such tool exists, emit a numbered plain-text message and ask the user to reply with the chosen number/answer for each question.
|
||||
3. **Batching**: if the tool supports multiple questions per call, combine all applicable questions into a single call; if only single-question, ask them one at a time in priority order.
|
||||
|
||||
Concrete `AskUserQuestion` references below are examples — substitute the local equivalent in other runtimes.
|
||||
|
||||
## Script Directory
|
||||
|
||||
Scripts located in `scripts/` subdirectory.
|
||||
|
||||
**Path Resolution**:
|
||||
1. `{baseDir}` = this SKILL.md's directory
|
||||
2. Script path = `{baseDir}/scripts/main.ts`
|
||||
3. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
|
||||
|
||||
## Consent Requirement
|
||||
|
||||
**Before any conversion**, check and obtain consent.
|
||||
|
||||
### Consent Flow
|
||||
|
||||
**Step 1**: Check consent file
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
cat ~/Library/Application\ Support/baoyu-skills/x-to-markdown/consent.json
|
||||
|
||||
# Linux
|
||||
cat ~/.local/share/baoyu-skills/x-to-markdown/consent.json
|
||||
```
|
||||
|
||||
**Step 2**: If `accepted: true` and `disclaimerVersion: "1.0"` → print warning and proceed:
|
||||
```
|
||||
Warning: Using reverse-engineered X API. Accepted on: <acceptedAt>
|
||||
```
|
||||
|
||||
**Step 3**: If missing or version mismatch → display disclaimer:
|
||||
```
|
||||
DISCLAIMER
|
||||
|
||||
This tool uses a reverse-engineered X API, NOT official.
|
||||
|
||||
Risks:
|
||||
- May break if X changes API
|
||||
- No guarantees or support
|
||||
- Possible account restrictions
|
||||
- Use at your own risk
|
||||
|
||||
Accept terms and continue?
|
||||
```
|
||||
|
||||
Use `AskUserQuestion` with options: "Yes, I accept" | "No, I decline"
|
||||
|
||||
**Step 4**: On accept → create consent file:
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"accepted": true,
|
||||
"acceptedAt": "<ISO timestamp>",
|
||||
"disclaimerVersion": "1.0"
|
||||
}
|
||||
```
|
||||
|
||||
**Step 5**: On decline → output "User declined. Exiting." and stop.
|
||||
|
||||
## Preferences (EXTEND.md)
|
||||
|
||||
Check EXTEND.md in priority order — the first one found wins:
|
||||
|
||||
| Priority | Path | Scope |
|
||||
|----------|------|-------|
|
||||
| 1 | `.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md` | Project |
|
||||
| 2 | `${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md` | XDG |
|
||||
| 3 | `$HOME/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md` | User home |
|
||||
|
||||
| Result | Action |
|
||||
|--------|--------|
|
||||
| Found | Read, parse, apply settings |
|
||||
| Not found | **MUST** run first-time setup (see below) — do NOT silently create defaults |
|
||||
|
||||
**EXTEND.md supports**: Download media by default, default output directory.
|
||||
|
||||
### First-Time Setup (BLOCKING)
|
||||
|
||||
**CRITICAL**: When EXTEND.md is not found, you **MUST use `AskUserQuestion`** to ask the user for their preferences before creating EXTEND.md. **NEVER** create EXTEND.md with defaults without asking. This is a **BLOCKING** operation — do NOT proceed with any conversion until setup is complete.
|
||||
|
||||
Use `AskUserQuestion` with ALL questions in ONE call:
|
||||
|
||||
**Question 1** — header: "Media", question: "How to handle images and videos in tweets?"
|
||||
- "Ask each time (Recommended)" — After saving markdown, ask whether to download media
|
||||
- "Always download" — Always download media to local imgs/ and videos/ directories
|
||||
- "Never download" — Keep original remote URLs in markdown
|
||||
|
||||
**Question 2** — header: "Output", question: "Default output directory?"
|
||||
- "x-to-markdown (Recommended)" — Save to ./x-to-markdown/{username}/{tweet-id}.md
|
||||
- (User may choose "Other" to type a custom path)
|
||||
|
||||
**Question 3** — header: "Save", question: "Where to save preferences?"
|
||||
- "User (Recommended)" — ~/.baoyu-skills/ (all projects)
|
||||
- "Project" — .baoyu-skills/ (this project only)
|
||||
|
||||
After user answers, create EXTEND.md at the chosen location, confirm "Preferences saved to [path]", then continue.
|
||||
|
||||
Full reference: [references/config/first-time-setup.md](references/config/first-time-setup.md)
|
||||
|
||||
### Supported Keys
|
||||
|
||||
| Key | Default | Values | Description |
|
||||
|-----|---------|--------|-------------|
|
||||
| `download_media` | `ask` | `ask` / `1` / `0` | `ask` = prompt each time, `1` = always download, `0` = never |
|
||||
| `default_output_dir` | empty | path or empty | Default output directory (empty = `./x-to-markdown/`) |
|
||||
|
||||
**Value priority**:
|
||||
1. CLI arguments (`--download-media`, `-o`)
|
||||
2. EXTEND.md
|
||||
3. Skill defaults
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
${BUN_X} {baseDir}/scripts/main.ts <url>
|
||||
${BUN_X} {baseDir}/scripts/main.ts <url> -o output.md
|
||||
${BUN_X} {baseDir}/scripts/main.ts <url> --download-media
|
||||
${BUN_X} {baseDir}/scripts/main.ts <url> --json
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `<url>` | Tweet or article URL |
|
||||
| `-o <path>` | Output path |
|
||||
| `--json` | JSON output |
|
||||
| `--download-media` | Download image/video assets to local `imgs/` and `videos/`, and rewrite markdown links to local relative paths |
|
||||
| `--login` | Refresh cookies only |
|
||||
|
||||
## Supported URLs
|
||||
|
||||
- `https://x.com/<user>/status/<id>`
|
||||
- `https://twitter.com/<user>/status/<id>`
|
||||
- `https://x.com/i/article/<id>`
|
||||
|
||||
## Output
|
||||
|
||||
```markdown
|
||||
---
|
||||
url: "https://x.com/user/status/123"
|
||||
author: "Name (@user)"
|
||||
tweetCount: 3
|
||||
coverImage: "https://pbs.twimg.com/media/example.jpg"
|
||||
---
|
||||
|
||||
Content...
|
||||
```
|
||||
|
||||
**File structure**: `x-to-markdown/{username}/{tweet-id}/{content-slug}.md`
|
||||
|
||||
When `--download-media` is enabled:
|
||||
- Images are saved to `imgs/` next to the markdown file
|
||||
- Videos are saved to `videos/` next to the markdown file
|
||||
- Markdown media links are rewritten to local relative paths
|
||||
|
||||
## Media Download Workflow
|
||||
|
||||
Based on `download_media` setting in EXTEND.md:
|
||||
|
||||
| Setting | Behavior |
|
||||
|---------|----------|
|
||||
| `1` (always) | Run script with `--download-media` flag |
|
||||
| `0` (never) | Run script without `--download-media` flag |
|
||||
| `ask` (default) | Follow the ask-each-time flow below |
|
||||
|
||||
### Ask-Each-Time Flow
|
||||
|
||||
1. Run script **without** `--download-media` → markdown saved
|
||||
2. Check saved markdown for remote media URLs (`https://` in image/video links)
|
||||
3. **If no remote media found** → done, no prompt needed
|
||||
4. **If remote media found** → use `AskUserQuestion`:
|
||||
- header: "Media", question: "Download N images/videos to local files?"
|
||||
- "Yes" — Download to local directories
|
||||
- "No" — Keep remote URLs
|
||||
5. If user confirms → run script **again** with `--download-media` (overwrites markdown with localized links)
|
||||
|
||||
## Authentication
|
||||
|
||||
1. **Environment variables** (preferred): `X_AUTH_TOKEN`, `X_CT0`
|
||||
2. **Chrome login** (fallback): Auto-opens Chrome, caches cookies locally
|
||||
|
||||
## Extension Support
|
||||
|
||||
Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: first-time-setup
|
||||
description: First-time setup flow for baoyu-danger-x-to-markdown preferences
|
||||
---
|
||||
|
||||
# First-Time Setup
|
||||
|
||||
## Overview
|
||||
|
||||
When no EXTEND.md is found, guide user through preference setup.
|
||||
|
||||
**BLOCKING OPERATION**: This setup MUST complete before ANY other workflow steps. Do NOT:
|
||||
- Start converting tweets or articles
|
||||
- Ask about URLs or output paths
|
||||
- Proceed to any conversion
|
||||
|
||||
ONLY ask the questions in this setup flow, save EXTEND.md, then continue.
|
||||
|
||||
## Setup Flow
|
||||
|
||||
```
|
||||
No EXTEND.md found
|
||||
|
|
||||
v
|
||||
+---------------------+
|
||||
| AskUserQuestion |
|
||||
| (all questions) |
|
||||
+---------------------+
|
||||
|
|
||||
v
|
||||
+---------------------+
|
||||
| Create EXTEND.md |
|
||||
+---------------------+
|
||||
|
|
||||
v
|
||||
Continue conversion
|
||||
```
|
||||
|
||||
## Questions
|
||||
|
||||
**Language**: Use user's input language or saved language preference.
|
||||
|
||||
Use AskUserQuestion with ALL questions in ONE call:
|
||||
|
||||
### Question 1: Download Media
|
||||
|
||||
```yaml
|
||||
header: "Media"
|
||||
question: "How to handle images and videos in tweets?"
|
||||
options:
|
||||
- label: "Ask each time (Recommended)"
|
||||
description: "After saving markdown, ask whether to download media"
|
||||
- label: "Always download"
|
||||
description: "Always download media to local imgs/ and videos/ directories"
|
||||
- label: "Never download"
|
||||
description: "Keep original remote URLs in markdown"
|
||||
```
|
||||
|
||||
### Question 2: Default Output Directory
|
||||
|
||||
```yaml
|
||||
header: "Output"
|
||||
question: "Default output directory?"
|
||||
options:
|
||||
- label: "x-to-markdown (Recommended)"
|
||||
description: "Save to ./x-to-markdown/{username}/{tweet-id}.md"
|
||||
```
|
||||
|
||||
Note: User will likely choose "Other" to type a custom path.
|
||||
|
||||
### Question 3: Save Location
|
||||
|
||||
```yaml
|
||||
header: "Save"
|
||||
question: "Where to save preferences?"
|
||||
options:
|
||||
- label: "User (Recommended)"
|
||||
description: "~/.baoyu-skills/ (all projects)"
|
||||
- label: "Project"
|
||||
description: ".baoyu-skills/ (this project only)"
|
||||
```
|
||||
|
||||
## Save Locations
|
||||
|
||||
| Choice | Path | Scope |
|
||||
|--------|------|-------|
|
||||
| User | `~/.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md` | All projects |
|
||||
| Project | `.baoyu-skills/baoyu-danger-x-to-markdown/EXTEND.md` | Current project |
|
||||
|
||||
## After Setup
|
||||
|
||||
1. Create directory if needed
|
||||
2. Write EXTEND.md
|
||||
3. Confirm: "Preferences saved to [path]"
|
||||
4. Continue with conversion using saved preferences
|
||||
|
||||
## EXTEND.md Template
|
||||
|
||||
```md
|
||||
download_media: [ask/1/0]
|
||||
default_output_dir: [path or empty]
|
||||
```
|
||||
|
||||
## Modifying Preferences Later
|
||||
|
||||
Users can edit EXTEND.md directly or delete it to trigger setup again.
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 1,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"name": "baoyu-danger-x-to-markdown-scripts",
|
||||
"dependencies": {
|
||||
"baoyu-chrome-cdp": "^0.1.1",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"baoyu-chrome-cdp": ["baoyu-chrome-cdp@0.1.1", "", {}, "sha512-OR3PQ7NzJDykCXl20TnkZuwvNQ0hsVC2czje93P72xQaA3vKOyPN/Q1CwEgKuYzP7Rka4Fdh4HvURj6AoNR7Tg=="],
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
import { resolveXToMarkdownChromeProfileDir } from "./paths.js";
|
||||
|
||||
export const DEFAULT_BEARER_TOKEN =
|
||||
"Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
|
||||
export const DEFAULT_USER_AGENT =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36";
|
||||
export const X_LOGIN_URL = "https://x.com/home";
|
||||
export const X_USER_DATA_DIR = resolveXToMarkdownChromeProfileDir();
|
||||
|
||||
export const X_COOKIE_NAMES = ["auth_token", "ct0", "gt", "twid"] as const;
|
||||
export const X_REQUIRED_COOKIES = ["auth_token", "ct0"] as const;
|
||||
|
||||
export const FALLBACK_QUERY_ID = "id8pHQbQi7eZ6P9mA1th1Q";
|
||||
export const FALLBACK_FEATURE_SWITCHES = [
|
||||
"profile_label_improvements_pcf_label_in_post_enabled",
|
||||
"responsive_web_profile_redirect_enabled",
|
||||
"rweb_tipjar_consumption_enabled",
|
||||
"verified_phone_label_enabled",
|
||||
"responsive_web_graphql_skip_user_profile_image_extensions_enabled",
|
||||
"responsive_web_graphql_timeline_navigation_enabled",
|
||||
];
|
||||
export const FALLBACK_FIELD_TOGGLES = ["withPayments", "withAuxiliaryUserLabels"];
|
||||
|
||||
export const FALLBACK_TWEET_QUERY_ID = "HJ9lpOL-ZlOk5CkCw0JW6Q";
|
||||
export const FALLBACK_TWEET_FEATURE_SWITCHES = [
|
||||
"creator_subscriptions_tweet_preview_api_enabled",
|
||||
"premium_content_api_read_enabled",
|
||||
"communities_web_enable_tweet_community_results_fetch",
|
||||
"c9s_tweet_anatomy_moderator_badge_enabled",
|
||||
"responsive_web_grok_analyze_button_fetch_trends_enabled",
|
||||
"responsive_web_grok_analyze_post_followups_enabled",
|
||||
"responsive_web_jetfuel_frame",
|
||||
"responsive_web_grok_share_attachment_enabled",
|
||||
"responsive_web_grok_annotations_enabled",
|
||||
"articles_preview_enabled",
|
||||
"responsive_web_edit_tweet_api_enabled",
|
||||
"graphql_is_translatable_rweb_tweet_is_translatable_enabled",
|
||||
"view_counts_everywhere_api_enabled",
|
||||
"longform_notetweets_consumption_enabled",
|
||||
"responsive_web_twitter_article_tweet_consumption_enabled",
|
||||
"tweet_awards_web_tipping_enabled",
|
||||
"responsive_web_grok_show_grok_translated_post",
|
||||
"responsive_web_grok_analysis_button_from_backend",
|
||||
"post_ctas_fetch_enabled",
|
||||
"creator_subscriptions_quote_tweet_preview_enabled",
|
||||
"freedom_of_speech_not_reach_fetch_enabled",
|
||||
"standardized_nudges_misinfo",
|
||||
"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled",
|
||||
"longform_notetweets_rich_text_read_enabled",
|
||||
"longform_notetweets_inline_media_enabled",
|
||||
"profile_label_improvements_pcf_label_in_post_enabled",
|
||||
"responsive_web_profile_redirect_enabled",
|
||||
"rweb_tipjar_consumption_enabled",
|
||||
"verified_phone_label_enabled",
|
||||
"responsive_web_grok_image_annotation_enabled",
|
||||
"responsive_web_grok_imagine_annotation_enabled",
|
||||
"responsive_web_grok_community_note_auto_translation_is_enabled",
|
||||
"responsive_web_graphql_skip_user_profile_image_extensions_enabled",
|
||||
"responsive_web_graphql_timeline_navigation_enabled",
|
||||
"responsive_web_enhance_cards_enabled",
|
||||
];
|
||||
export const FALLBACK_TWEET_FIELD_TOGGLES = [
|
||||
"withArticleRichContentState",
|
||||
"withArticlePlainText",
|
||||
"withGrokAnalyze",
|
||||
"withDisallowedReplyControls",
|
||||
"withPayments",
|
||||
"withAuxiliaryUserLabels",
|
||||
];
|
||||
|
||||
export const FALLBACK_TWEET_DETAIL_QUERY_ID = "_8aYOgEDz35BrBcBal1-_w";
|
||||
export const FALLBACK_TWEET_DETAIL_FEATURE_SWITCHES = [
|
||||
"rweb_video_screen_enabled",
|
||||
"profile_label_improvements_pcf_label_in_post_enabled",
|
||||
"rweb_tipjar_consumption_enabled",
|
||||
"verified_phone_label_enabled",
|
||||
"creator_subscriptions_tweet_preview_api_enabled",
|
||||
"responsive_web_graphql_timeline_navigation_enabled",
|
||||
"responsive_web_graphql_skip_user_profile_image_extensions_enabled",
|
||||
"premium_content_api_read_enabled",
|
||||
"communities_web_enable_tweet_community_results_fetch",
|
||||
"c9s_tweet_anatomy_moderator_badge_enabled",
|
||||
"responsive_web_grok_analyze_button_fetch_trends_enabled",
|
||||
"responsive_web_grok_analyze_post_followups_enabled",
|
||||
"responsive_web_jetfuel_frame",
|
||||
"responsive_web_grok_share_attachment_enabled",
|
||||
"articles_preview_enabled",
|
||||
"responsive_web_edit_tweet_api_enabled",
|
||||
"graphql_is_translatable_rweb_tweet_is_translatable_enabled",
|
||||
"view_counts_everywhere_api_enabled",
|
||||
"longform_notetweets_consumption_enabled",
|
||||
"responsive_web_twitter_article_tweet_consumption_enabled",
|
||||
"tweet_awards_web_tipping_enabled",
|
||||
"responsive_web_grok_show_grok_translated_post",
|
||||
"responsive_web_grok_analysis_button_from_backend",
|
||||
"creator_subscriptions_quote_tweet_preview_enabled",
|
||||
"freedom_of_speech_not_reach_fetch_enabled",
|
||||
"standardized_nudges_misinfo",
|
||||
"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled",
|
||||
"longform_notetweets_rich_text_read_enabled",
|
||||
"longform_notetweets_inline_media_enabled",
|
||||
"responsive_web_grok_image_annotation_enabled",
|
||||
"responsive_web_enhance_cards_enabled",
|
||||
];
|
||||
export const FALLBACK_TWEET_DETAIL_FEATURE_DEFAULTS: Record<string, boolean> = {
|
||||
rweb_video_screen_enabled: false,
|
||||
profile_label_improvements_pcf_label_in_post_enabled: true,
|
||||
rweb_tipjar_consumption_enabled: true,
|
||||
verified_phone_label_enabled: false,
|
||||
creator_subscriptions_tweet_preview_api_enabled: true,
|
||||
responsive_web_graphql_timeline_navigation_enabled: true,
|
||||
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
|
||||
premium_content_api_read_enabled: false,
|
||||
communities_web_enable_tweet_community_results_fetch: true,
|
||||
c9s_tweet_anatomy_moderator_badge_enabled: true,
|
||||
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
|
||||
responsive_web_grok_analyze_post_followups_enabled: true,
|
||||
responsive_web_jetfuel_frame: false,
|
||||
responsive_web_grok_share_attachment_enabled: true,
|
||||
articles_preview_enabled: true,
|
||||
responsive_web_edit_tweet_api_enabled: true,
|
||||
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
|
||||
view_counts_everywhere_api_enabled: true,
|
||||
longform_notetweets_consumption_enabled: true,
|
||||
responsive_web_twitter_article_tweet_consumption_enabled: true,
|
||||
tweet_awards_web_tipping_enabled: false,
|
||||
responsive_web_grok_show_grok_translated_post: false,
|
||||
responsive_web_grok_analysis_button_from_backend: true,
|
||||
creator_subscriptions_quote_tweet_preview_enabled: false,
|
||||
freedom_of_speech_not_reach_fetch_enabled: true,
|
||||
standardized_nudges_misinfo: true,
|
||||
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
|
||||
longform_notetweets_rich_text_read_enabled: true,
|
||||
longform_notetweets_inline_media_enabled: true,
|
||||
responsive_web_grok_image_annotation_enabled: true,
|
||||
responsive_web_enhance_cards_enabled: false,
|
||||
};
|
||||
export const FALLBACK_TWEET_DETAIL_FIELD_TOGGLES = [
|
||||
"withArticleRichContentState",
|
||||
"withArticlePlainText",
|
||||
"withGrokAnalyze",
|
||||
"withDisallowedReplyControls",
|
||||
];
|
||||
@@ -0,0 +1,85 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
|
||||
import { resolveXToMarkdownCookiePath } from "./paths.js";
|
||||
|
||||
export type CookieMap = Record<string, string>;
|
||||
|
||||
export type CookieFileData =
|
||||
| {
|
||||
cookies: CookieMap;
|
||||
updated_at: number;
|
||||
source?: string;
|
||||
}
|
||||
| {
|
||||
version: number;
|
||||
updatedAt: string;
|
||||
cookieMap: CookieMap;
|
||||
source?: string;
|
||||
};
|
||||
|
||||
export async function read_cookie_file(
|
||||
p: string = resolveXToMarkdownCookiePath()
|
||||
): Promise<CookieMap | null> {
|
||||
try {
|
||||
if (!fs.existsSync(p) || !fs.statSync(p).isFile()) return null;
|
||||
const raw = await readFile(p, "utf8");
|
||||
const data = JSON.parse(raw) as unknown;
|
||||
|
||||
if (data && typeof data === "object" && "cookies" in (data as any)) {
|
||||
const cookies = (data as any).cookies as unknown;
|
||||
if (cookies && typeof cookies === "object") {
|
||||
const out: CookieMap = {};
|
||||
for (const [k, v] of Object.entries(cookies as Record<string, unknown>)) {
|
||||
if (typeof v === "string") out[k] = v;
|
||||
}
|
||||
return Object.keys(out).length > 0 ? out : null;
|
||||
}
|
||||
}
|
||||
|
||||
if (data && typeof data === "object" && "cookieMap" in (data as any)) {
|
||||
const cookies = (data as any).cookieMap as unknown;
|
||||
if (cookies && typeof cookies === "object") {
|
||||
const out: CookieMap = {};
|
||||
for (const [k, v] of Object.entries(cookies as Record<string, unknown>)) {
|
||||
if (typeof v === "string") out[k] = v;
|
||||
}
|
||||
return Object.keys(out).length > 0 ? out : null;
|
||||
}
|
||||
}
|
||||
|
||||
if (data && typeof data === "object") {
|
||||
const out: CookieMap = {};
|
||||
for (const [k, v] of Object.entries(data as Record<string, unknown>)) {
|
||||
if (typeof v === "string") out[k] = v;
|
||||
}
|
||||
return Object.keys(out).length > 0 ? out : null;
|
||||
}
|
||||
|
||||
return null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function write_cookie_file(
|
||||
cookies: CookieMap,
|
||||
p: string = resolveXToMarkdownCookiePath(),
|
||||
source?: string
|
||||
): Promise<void> {
|
||||
const dir = path.dirname(p);
|
||||
await mkdir(dir, { recursive: true });
|
||||
|
||||
const payload: CookieFileData = {
|
||||
version: 1,
|
||||
updatedAt: new Date().toISOString(),
|
||||
cookieMap: cookies,
|
||||
source,
|
||||
};
|
||||
await writeFile(p, JSON.stringify(payload, null, 2), "utf8");
|
||||
}
|
||||
|
||||
export const readCookieFile = read_cookie_file;
|
||||
export const writeCookieFile = write_cookie_file;
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
import {
|
||||
CdpConnection,
|
||||
findChromeExecutable as findChromeExecutableBase,
|
||||
findExistingChromeDebugPort,
|
||||
gracefulKillChrome,
|
||||
getFreePort,
|
||||
launchChrome as launchChromeBase,
|
||||
openPageSession,
|
||||
sleep,
|
||||
waitForChromeDebugPort,
|
||||
type PlatformCandidates,
|
||||
} from "baoyu-chrome-cdp";
|
||||
|
||||
import process from "node:process";
|
||||
|
||||
import { read_cookie_file, write_cookie_file } from "./cookie-file.js";
|
||||
import { resolveXToMarkdownCookiePath } from "./paths.js";
|
||||
import { X_COOKIE_NAMES, X_REQUIRED_COOKIES, X_LOGIN_URL, X_USER_DATA_DIR } from "./constants.js";
|
||||
import type { CookieLike } from "./types.js";
|
||||
|
||||
const CHROME_CANDIDATES_FULL: PlatformCandidates = {
|
||||
darwin: [
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
"/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
|
||||
"/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta",
|
||||
"/Applications/Chromium.app/Contents/MacOS/Chromium",
|
||||
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
|
||||
],
|
||||
win32: [
|
||||
"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
|
||||
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
|
||||
"C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
|
||||
"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
|
||||
],
|
||||
default: [
|
||||
"/usr/bin/google-chrome",
|
||||
"/usr/bin/google-chrome-stable",
|
||||
"/usr/bin/chromium",
|
||||
"/usr/bin/chromium-browser",
|
||||
"/snap/bin/chromium",
|
||||
"/usr/bin/microsoft-edge",
|
||||
],
|
||||
};
|
||||
|
||||
function findChromeExecutable(): string | null {
|
||||
return findChromeExecutableBase({
|
||||
candidates: CHROME_CANDIDATES_FULL,
|
||||
envNames: ["X_CHROME_PATH"],
|
||||
}) ?? null;
|
||||
}
|
||||
|
||||
async function launchChrome(profileDir: string, port: number) {
|
||||
const chromePath = findChromeExecutable();
|
||||
if (!chromePath) throw new Error("Chrome executable not found.");
|
||||
return await launchChromeBase({
|
||||
chromePath,
|
||||
profileDir,
|
||||
port,
|
||||
url: X_LOGIN_URL,
|
||||
extraArgs: ["--disable-popup-blocking"],
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchXCookiesViaCdp(
|
||||
profileDir: string,
|
||||
timeoutMs: number,
|
||||
verbose: boolean,
|
||||
log?: (message: string) => void
|
||||
): Promise<Record<string, string>> {
|
||||
const existingPort = await findExistingChromeDebugPort({ profileDir });
|
||||
const reusing = existingPort !== null;
|
||||
const port = existingPort ?? await getFreePort("X_DEBUG_PORT");
|
||||
const chrome = reusing ? null : await launchChrome(profileDir, port);
|
||||
|
||||
let cdp: CdpConnection | null = null;
|
||||
let targetId: string | null = null;
|
||||
try {
|
||||
const wsUrl = await waitForChromeDebugPort(port, 30_000, { includeLastError: true });
|
||||
cdp = await CdpConnection.connect(wsUrl, 15_000);
|
||||
|
||||
const page = await openPageSession({
|
||||
cdp,
|
||||
reusing,
|
||||
url: X_LOGIN_URL,
|
||||
matchTarget: (target) => target.type === "page" && (
|
||||
target.url.includes("x.com") || target.url.includes("twitter.com")
|
||||
),
|
||||
enableNetwork: true,
|
||||
});
|
||||
const { sessionId } = page;
|
||||
targetId = page.targetId;
|
||||
|
||||
if (verbose) {
|
||||
log?.(reusing
|
||||
? `[x-cookies] Reusing existing Chrome on port ${port}. Waiting for cookies...`
|
||||
: "[x-cookies] Chrome opened. If needed, complete X login in the window. Waiting for cookies...");
|
||||
}
|
||||
|
||||
const start = Date.now();
|
||||
let last: Record<string, string> = {};
|
||||
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const { cookies } = await cdp.send<{ cookies: CookieLike[] }>(
|
||||
"Network.getCookies",
|
||||
{ urls: ["https://x.com/", "https://twitter.com/"] },
|
||||
{ sessionId, timeoutMs: 10_000 }
|
||||
);
|
||||
|
||||
const m = buildXCookieMap((cookies ?? []).filter(Boolean));
|
||||
last = m;
|
||||
if (hasRequiredXCookies(m)) {
|
||||
return m;
|
||||
}
|
||||
|
||||
await sleep(1000);
|
||||
}
|
||||
|
||||
throw new Error(`Timed out waiting for X cookies. Last keys: ${Object.keys(last).join(", ")}`);
|
||||
} finally {
|
||||
if (cdp) {
|
||||
if (reusing && targetId) {
|
||||
try {
|
||||
await cdp.send("Target.closeTarget", { targetId }, { timeoutMs: 5_000 });
|
||||
} catch {}
|
||||
}
|
||||
cdp.close();
|
||||
}
|
||||
|
||||
if (chrome) await gracefulKillChrome(chrome, port);
|
||||
}
|
||||
}
|
||||
|
||||
function resolveCookieDomain(cookie: CookieLike): string | null {
|
||||
const rawDomain = cookie.domain?.trim();
|
||||
if (rawDomain) {
|
||||
return rawDomain.startsWith(".") ? rawDomain.slice(1) : rawDomain;
|
||||
}
|
||||
const rawUrl = cookie.url?.trim();
|
||||
if (rawUrl) {
|
||||
try {
|
||||
return new URL(rawUrl).hostname;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function pickCookieValue<T extends CookieLike>(cookies: T[], name: string): string | undefined {
|
||||
const matches = cookies.filter((cookie) => cookie.name === name && typeof cookie.value === "string");
|
||||
if (matches.length === 0) return undefined;
|
||||
|
||||
const preferred = matches.find((cookie) => {
|
||||
const domain = resolveCookieDomain(cookie);
|
||||
return domain === "x.com" && (cookie.path ?? "/") === "/";
|
||||
});
|
||||
const xDomain = matches.find((cookie) => (resolveCookieDomain(cookie) ?? "").endsWith("x.com"));
|
||||
const twitterDomain = matches.find((cookie) => (resolveCookieDomain(cookie) ?? "").endsWith("twitter.com"));
|
||||
return (preferred ?? xDomain ?? twitterDomain ?? matches[0])?.value;
|
||||
}
|
||||
|
||||
function buildXCookieMap<T extends CookieLike>(cookies: T[]): Record<string, string> {
|
||||
const cookieMap: Record<string, string> = {};
|
||||
for (const name of X_COOKIE_NAMES) {
|
||||
const value = pickCookieValue(cookies, name);
|
||||
if (value) cookieMap[name] = value;
|
||||
}
|
||||
return cookieMap;
|
||||
}
|
||||
|
||||
export function hasRequiredXCookies(cookieMap: Record<string, string>): boolean {
|
||||
return X_REQUIRED_COOKIES.every((name) => Boolean(cookieMap[name]));
|
||||
}
|
||||
|
||||
function filterXCookieMap(cookieMap: Record<string, string>): Record<string, string> {
|
||||
const filtered: Record<string, string> = {};
|
||||
for (const name of X_COOKIE_NAMES) {
|
||||
const value = cookieMap[name];
|
||||
if (value) filtered[name] = value;
|
||||
}
|
||||
return filtered;
|
||||
}
|
||||
|
||||
function buildInlineCookiesFromEnv(): CookieLike[] {
|
||||
const cookies: CookieLike[] = [];
|
||||
const authToken = process.env.X_AUTH_TOKEN?.trim();
|
||||
const ct0 = process.env.X_CT0?.trim();
|
||||
const gt = process.env.X_GUEST_TOKEN?.trim();
|
||||
const twid = process.env.X_TWID?.trim();
|
||||
|
||||
if (authToken) {
|
||||
cookies.push({ name: "auth_token", value: authToken, domain: "x.com", path: "/" });
|
||||
}
|
||||
if (ct0) {
|
||||
cookies.push({ name: "ct0", value: ct0, domain: "x.com", path: "/" });
|
||||
}
|
||||
if (gt) {
|
||||
cookies.push({ name: "gt", value: gt, domain: "x.com", path: "/" });
|
||||
}
|
||||
if (twid) {
|
||||
cookies.push({ name: "twid", value: twid, domain: "x.com", path: "/" });
|
||||
}
|
||||
|
||||
return cookies;
|
||||
}
|
||||
|
||||
async function loadXCookiesFromInline(log?: (message: string) => void): Promise<Record<string, string>> {
|
||||
const inline = buildInlineCookiesFromEnv();
|
||||
if (inline.length === 0) return {};
|
||||
|
||||
const cookieMap = buildXCookieMap(
|
||||
inline.filter((cookie): cookie is CookieLike => Boolean(cookie?.name && typeof cookie.value === "string"))
|
||||
);
|
||||
|
||||
if (Object.keys(cookieMap).length > 0) {
|
||||
log?.(`[x-cookies] Loaded X cookies from env: ${Object.keys(cookieMap).length} cookie(s).`);
|
||||
} else {
|
||||
log?.("[x-cookies] Env cookies provided but no X cookies matched.");
|
||||
}
|
||||
|
||||
return cookieMap;
|
||||
}
|
||||
|
||||
async function loadXCookiesFromFile(log?: (message: string) => void): Promise<Record<string, string>> {
|
||||
const cookiePath = resolveXToMarkdownCookiePath();
|
||||
const fileMap = filterXCookieMap((await read_cookie_file(cookiePath)) ?? {});
|
||||
if (Object.keys(fileMap).length > 0) {
|
||||
log?.(`[x-cookies] Loaded X cookies from file: ${cookiePath} (${Object.keys(fileMap).length} cookie(s))`);
|
||||
}
|
||||
return fileMap;
|
||||
}
|
||||
|
||||
async function loadXCookiesFromCdp(log?: (message: string) => void): Promise<Record<string, string>> {
|
||||
try {
|
||||
const cookieMap = await fetchXCookiesViaCdp(X_USER_DATA_DIR, 5 * 60 * 1000, true, log);
|
||||
if (!hasRequiredXCookies(cookieMap)) return cookieMap;
|
||||
|
||||
const cookiePath = resolveXToMarkdownCookiePath();
|
||||
try {
|
||||
await write_cookie_file(cookieMap, cookiePath, "cdp");
|
||||
log?.(`[x-cookies] Cookies saved to ${cookiePath}`);
|
||||
} catch (error) {
|
||||
log?.(
|
||||
`[x-cookies] Failed to write cookie file (${cookiePath}): ${
|
||||
error instanceof Error ? error.message : String(error ?? "")
|
||||
}`
|
||||
);
|
||||
}
|
||||
if (cookieMap.auth_token) log?.(`[x-cookies] auth_token: ${cookieMap.auth_token.slice(0, 20)}...`);
|
||||
if (cookieMap.ct0) log?.(`[x-cookies] ct0: ${cookieMap.ct0.slice(0, 20)}...`);
|
||||
return cookieMap;
|
||||
} catch (error) {
|
||||
log?.(
|
||||
`[x-cookies] Failed to load cookies via Chrome DevTools Protocol: ${
|
||||
error instanceof Error ? error.message : String(error ?? "")
|
||||
}`
|
||||
);
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
export async function loadXCookies(log?: (message: string) => void): Promise<Record<string, string>> {
|
||||
const inlineMap = await loadXCookiesFromInline(log);
|
||||
const fileMap = await loadXCookiesFromFile(log);
|
||||
const combined = { ...fileMap, ...inlineMap };
|
||||
|
||||
if (hasRequiredXCookies(combined)) return combined;
|
||||
|
||||
const cdpMap = await loadXCookiesFromCdp(log);
|
||||
return { ...fileMap, ...cdpMap, ...inlineMap };
|
||||
}
|
||||
|
||||
export async function refreshXCookies(log?: (message: string) => void): Promise<Record<string, string>> {
|
||||
return loadXCookiesFromCdp(log);
|
||||
}
|
||||
|
||||
export function buildCookieHeader(cookieMap: Record<string, string>): string | undefined {
|
||||
const entries = Object.entries(cookieMap).filter(([, value]) => value);
|
||||
if (entries.length === 0) return undefined;
|
||||
return entries.map(([key, value]) => `${key}=${value}`).join("; ");
|
||||
}
|
||||
@@ -0,0 +1,389 @@
|
||||
import {
|
||||
DEFAULT_BEARER_TOKEN,
|
||||
DEFAULT_USER_AGENT,
|
||||
FALLBACK_FEATURE_SWITCHES,
|
||||
FALLBACK_FIELD_TOGGLES,
|
||||
FALLBACK_QUERY_ID,
|
||||
FALLBACK_TWEET_DETAIL_FEATURE_DEFAULTS,
|
||||
FALLBACK_TWEET_DETAIL_FEATURE_SWITCHES,
|
||||
FALLBACK_TWEET_DETAIL_FIELD_TOGGLES,
|
||||
FALLBACK_TWEET_DETAIL_QUERY_ID,
|
||||
FALLBACK_TWEET_FEATURE_SWITCHES,
|
||||
FALLBACK_TWEET_FIELD_TOGGLES,
|
||||
FALLBACK_TWEET_QUERY_ID,
|
||||
} from "./constants.js";
|
||||
import {
|
||||
buildFeatureMap,
|
||||
buildFieldToggleMap,
|
||||
buildRequestHeaders,
|
||||
buildTweetFieldToggleMap,
|
||||
fetchHomeHtml,
|
||||
fetchText,
|
||||
parseStringList,
|
||||
} from "./http.js";
|
||||
import type { ArticleQueryInfo } from "./types.js";
|
||||
|
||||
function isNonEmptyObject(value: unknown): value is Record<string, unknown> {
|
||||
return Boolean(value && typeof value === "object" && Object.keys(value as Record<string, unknown>).length > 0);
|
||||
}
|
||||
|
||||
function unwrapTweetResult(result: any): any {
|
||||
if (!result) return null;
|
||||
if (result.__typename === "TweetWithVisibilityResults" && result.tweet) {
|
||||
return result.tweet;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function extractArticleFromTweet(payload: unknown): unknown {
|
||||
const root = (payload as { data?: any }).data ?? payload;
|
||||
const result = root?.tweetResult?.result ?? root?.tweet_result?.result ?? root?.tweet_result;
|
||||
const tweet = unwrapTweetResult(result);
|
||||
const legacy = tweet?.legacy ?? {};
|
||||
const article = legacy?.article ?? tweet?.article;
|
||||
return (
|
||||
article?.article_results?.result ??
|
||||
legacy?.article_results?.result ??
|
||||
tweet?.article_results?.result ??
|
||||
null
|
||||
);
|
||||
}
|
||||
|
||||
function extractTweetFromPayload(payload: unknown): unknown {
|
||||
const root = (payload as { data?: any }).data ?? payload;
|
||||
const result = root?.tweetResult?.result ?? root?.tweet_result?.result ?? root?.tweet_result;
|
||||
return unwrapTweetResult(result);
|
||||
}
|
||||
|
||||
function extractArticleFromEntity(payload: unknown): unknown {
|
||||
const root = (payload as { data?: any }).data ?? payload;
|
||||
return (
|
||||
root?.article_result_by_rest_id?.result ??
|
||||
root?.article_result_by_rest_id ??
|
||||
root?.article_entity_result?.result ??
|
||||
null
|
||||
);
|
||||
}
|
||||
|
||||
async function resolveArticleQueryInfo(userAgent: string): Promise<ArticleQueryInfo> {
|
||||
const html = await fetchHomeHtml(userAgent);
|
||||
|
||||
const bundleMatch = html.match(/"bundle\\.TwitterArticles":"([a-z0-9]+)"/);
|
||||
if (!bundleMatch) {
|
||||
return {
|
||||
queryId: FALLBACK_QUERY_ID,
|
||||
featureSwitches: FALLBACK_FEATURE_SWITCHES,
|
||||
fieldToggles: FALLBACK_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
const bundleHash = bundleMatch[1];
|
||||
const chunkUrl = `https://abs.twimg.com/responsive-web/client-web/bundle.TwitterArticles.${bundleHash}a.js`;
|
||||
const chunk = await fetchText(chunkUrl, {
|
||||
headers: {
|
||||
"user-agent": userAgent,
|
||||
},
|
||||
});
|
||||
|
||||
const queryIdMatch = chunk.match(/queryId:\"([^\"]+)\",operationName:\"ArticleEntityResultByRestId\"/);
|
||||
const featureMatch = chunk.match(
|
||||
/operationName:\"ArticleEntityResultByRestId\"[\s\S]*?featureSwitches:\[(.*?)\]/
|
||||
);
|
||||
const fieldToggleMatch = chunk.match(
|
||||
/operationName:\"ArticleEntityResultByRestId\"[\s\S]*?fieldToggles:\[(.*?)\]/
|
||||
);
|
||||
|
||||
const featureSwitches = parseStringList(featureMatch?.[1]);
|
||||
const fieldToggles = parseStringList(fieldToggleMatch?.[1]);
|
||||
|
||||
return {
|
||||
queryId: queryIdMatch?.[1] ?? FALLBACK_QUERY_ID,
|
||||
featureSwitches: featureSwitches.length > 0 ? featureSwitches : FALLBACK_FEATURE_SWITCHES,
|
||||
fieldToggles: fieldToggles.length > 0 ? fieldToggles : FALLBACK_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveMainChunkHash(html: string): string | null {
|
||||
const match = html.match(/main\\.([a-z0-9]+)\\.js/);
|
||||
return match?.[1] ?? null;
|
||||
}
|
||||
|
||||
function resolveApiChunkHash(html: string): string | null {
|
||||
const match = html.match(/api:\"([a-zA-Z0-9_-]+)\"/);
|
||||
return match?.[1] ?? null;
|
||||
}
|
||||
|
||||
async function resolveTweetDetailQueryInfo(userAgent: string): Promise<ArticleQueryInfo> {
|
||||
const html = await fetchHomeHtml(userAgent);
|
||||
const apiHash = resolveApiChunkHash(html);
|
||||
if (!apiHash) {
|
||||
return {
|
||||
queryId: FALLBACK_TWEET_DETAIL_QUERY_ID,
|
||||
featureSwitches: FALLBACK_TWEET_DETAIL_FEATURE_SWITCHES,
|
||||
fieldToggles: FALLBACK_TWEET_DETAIL_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
const chunkUrl = `https://abs.twimg.com/responsive-web/client-web/api.${apiHash}a.js`;
|
||||
const chunk = await fetchText(chunkUrl, {
|
||||
headers: {
|
||||
"user-agent": userAgent,
|
||||
},
|
||||
});
|
||||
|
||||
const queryIdMatch = chunk.match(/queryId:\"([^\"]+)\",operationName:\"TweetDetail\"/);
|
||||
const featureMatch = chunk.match(
|
||||
/operationName:\"TweetDetail\"[\s\S]*?featureSwitches:\[(.*?)\]/
|
||||
);
|
||||
const fieldToggleMatch = chunk.match(
|
||||
/operationName:\"TweetDetail\"[\s\S]*?fieldToggles:\[(.*?)\]/
|
||||
);
|
||||
|
||||
const featureSwitches = parseStringList(featureMatch?.[1]);
|
||||
const fieldToggles = parseStringList(fieldToggleMatch?.[1]);
|
||||
|
||||
return {
|
||||
queryId: queryIdMatch?.[1] ?? FALLBACK_TWEET_DETAIL_QUERY_ID,
|
||||
featureSwitches: featureSwitches.length > 0 ? featureSwitches : FALLBACK_TWEET_DETAIL_FEATURE_SWITCHES,
|
||||
fieldToggles: fieldToggles.length > 0 ? fieldToggles : FALLBACK_TWEET_DETAIL_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
function buildTweetDetailFieldToggleMap(keys: string[]): Record<string, boolean> {
|
||||
const toggles = buildFieldToggleMap(keys);
|
||||
if (Object.prototype.hasOwnProperty.call(toggles, "withArticlePlainText")) {
|
||||
toggles.withArticlePlainText = false;
|
||||
}
|
||||
if (Object.prototype.hasOwnProperty.call(toggles, "withGrokAnalyze")) {
|
||||
toggles.withGrokAnalyze = false;
|
||||
}
|
||||
if (Object.prototype.hasOwnProperty.call(toggles, "withDisallowedReplyControls")) {
|
||||
toggles.withDisallowedReplyControls = false;
|
||||
}
|
||||
return toggles;
|
||||
}
|
||||
|
||||
async function resolveTweetQueryInfo(userAgent: string): Promise<ArticleQueryInfo> {
|
||||
const html = await fetchHomeHtml(userAgent);
|
||||
const mainHash = resolveMainChunkHash(html);
|
||||
if (!mainHash) {
|
||||
return {
|
||||
queryId: FALLBACK_TWEET_QUERY_ID,
|
||||
featureSwitches: FALLBACK_TWEET_FEATURE_SWITCHES,
|
||||
fieldToggles: FALLBACK_TWEET_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
const chunkUrl = `https://abs.twimg.com/responsive-web/client-web/main.${mainHash}.js`;
|
||||
const chunk = await fetchText(chunkUrl, {
|
||||
headers: {
|
||||
"user-agent": userAgent,
|
||||
},
|
||||
});
|
||||
|
||||
const queryIdMatch = chunk.match(/queryId:\"([^\"]+)\",operationName:\"TweetResultByRestId\"/);
|
||||
const featureMatch = chunk.match(
|
||||
/operationName:\"TweetResultByRestId\"[\s\S]*?featureSwitches:\[(.*?)\]/
|
||||
);
|
||||
const fieldToggleMatch = chunk.match(
|
||||
/operationName:\"TweetResultByRestId\"[\s\S]*?fieldToggles:\[(.*?)\]/
|
||||
);
|
||||
|
||||
const featureSwitches = parseStringList(featureMatch?.[1]);
|
||||
const fieldToggles = parseStringList(fieldToggleMatch?.[1]);
|
||||
|
||||
return {
|
||||
queryId: queryIdMatch?.[1] ?? FALLBACK_TWEET_QUERY_ID,
|
||||
featureSwitches: featureSwitches.length > 0 ? featureSwitches : FALLBACK_TWEET_FEATURE_SWITCHES,
|
||||
fieldToggles: fieldToggles.length > 0 ? fieldToggles : FALLBACK_TWEET_FIELD_TOGGLES,
|
||||
html,
|
||||
};
|
||||
}
|
||||
|
||||
async function fetchTweetResult(
|
||||
tweetId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
userAgent: string,
|
||||
bearerToken: string
|
||||
): Promise<unknown> {
|
||||
const queryInfo = await resolveTweetQueryInfo(userAgent);
|
||||
const features = buildFeatureMap(queryInfo.html, queryInfo.featureSwitches);
|
||||
const fieldToggles = buildTweetFieldToggleMap(queryInfo.fieldToggles);
|
||||
|
||||
const url = new URL(`https://x.com/i/api/graphql/${queryInfo.queryId}/TweetResultByRestId`);
|
||||
url.searchParams.set(
|
||||
"variables",
|
||||
JSON.stringify({
|
||||
tweetId,
|
||||
withCommunity: false,
|
||||
includePromotedContent: false,
|
||||
withVoice: true,
|
||||
})
|
||||
);
|
||||
if (Object.keys(features).length > 0) {
|
||||
url.searchParams.set("features", JSON.stringify(features));
|
||||
}
|
||||
if (Object.keys(fieldToggles).length > 0) {
|
||||
url.searchParams.set("fieldToggles", JSON.stringify(fieldToggles));
|
||||
}
|
||||
|
||||
const response = await fetch(url.toString(), {
|
||||
headers: buildRequestHeaders(cookieMap, userAgent, bearerToken),
|
||||
});
|
||||
|
||||
const text = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`X API error (${response.status}): ${text.slice(0, 400)}`);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to parse response JSON: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchTweetDetail(
|
||||
tweetId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
cursor?: string
|
||||
): Promise<unknown> {
|
||||
const userAgent = process.env.X_USER_AGENT?.trim() || DEFAULT_USER_AGENT;
|
||||
const bearerToken = process.env.X_BEARER_TOKEN?.trim() || DEFAULT_BEARER_TOKEN;
|
||||
const queryInfo = await resolveTweetDetailQueryInfo(userAgent);
|
||||
const features = buildFeatureMap(
|
||||
queryInfo.html,
|
||||
queryInfo.featureSwitches,
|
||||
FALLBACK_TWEET_DETAIL_FEATURE_DEFAULTS
|
||||
);
|
||||
const fieldToggles = buildTweetDetailFieldToggleMap(queryInfo.fieldToggles);
|
||||
|
||||
const url = new URL(`https://x.com/i/api/graphql/${queryInfo.queryId}/TweetDetail`);
|
||||
url.searchParams.set(
|
||||
"variables",
|
||||
JSON.stringify({
|
||||
focalTweetId: tweetId,
|
||||
cursor,
|
||||
referrer: cursor ? "tweet" : undefined,
|
||||
with_rux_injections: false,
|
||||
includePromotedContent: true,
|
||||
withCommunity: true,
|
||||
withQuickPromoteEligibilityTweetFields: true,
|
||||
withBirdwatchNotes: true,
|
||||
withVoice: true,
|
||||
withV2Timeline: true,
|
||||
withDownvotePerspective: false,
|
||||
withReactionsMetadata: false,
|
||||
withReactionsPerspective: false,
|
||||
withSuperFollowsTweetFields: false,
|
||||
withSuperFollowsUserFields: false,
|
||||
})
|
||||
);
|
||||
if (Object.keys(features).length > 0) {
|
||||
url.searchParams.set("features", JSON.stringify(features));
|
||||
}
|
||||
if (Object.keys(fieldToggles).length > 0) {
|
||||
url.searchParams.set("fieldToggles", JSON.stringify(fieldToggles));
|
||||
}
|
||||
|
||||
const response = await fetch(url.toString(), {
|
||||
headers: buildRequestHeaders(cookieMap, userAgent, bearerToken),
|
||||
});
|
||||
|
||||
const text = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`X API error (${response.status}): ${text.slice(0, 400)}`);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to parse response JSON: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchArticleEntityById(
|
||||
articleEntityId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
userAgent: string,
|
||||
bearerToken: string
|
||||
): Promise<unknown> {
|
||||
const queryInfo = await resolveArticleQueryInfo(userAgent);
|
||||
const features = buildFeatureMap(queryInfo.html, queryInfo.featureSwitches);
|
||||
const fieldToggles = buildFieldToggleMap(queryInfo.fieldToggles);
|
||||
|
||||
const url = new URL(`https://x.com/i/api/graphql/${queryInfo.queryId}/ArticleEntityResultByRestId`);
|
||||
url.searchParams.set("variables", JSON.stringify({ articleEntityId }));
|
||||
if (Object.keys(features).length > 0) {
|
||||
url.searchParams.set("features", JSON.stringify(features));
|
||||
}
|
||||
if (Object.keys(fieldToggles).length > 0) {
|
||||
url.searchParams.set("fieldToggles", JSON.stringify(fieldToggles));
|
||||
}
|
||||
|
||||
const response = await fetch(url.toString(), {
|
||||
headers: buildRequestHeaders(cookieMap, userAgent, bearerToken),
|
||||
});
|
||||
|
||||
const text = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`X API error (${response.status}): ${text.slice(0, 400)}`);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to parse response JSON: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchXArticle(
|
||||
articleId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
raw: boolean
|
||||
): Promise<unknown> {
|
||||
const userAgent = process.env.X_USER_AGENT?.trim() || DEFAULT_USER_AGENT;
|
||||
const bearerToken = process.env.X_BEARER_TOKEN?.trim() || DEFAULT_BEARER_TOKEN;
|
||||
|
||||
const tweetPayload = await fetchTweetResult(articleId, cookieMap, userAgent, bearerToken);
|
||||
if (raw) {
|
||||
return tweetPayload;
|
||||
}
|
||||
|
||||
const articleFromTweet = extractArticleFromTweet(tweetPayload);
|
||||
if (isNonEmptyObject(articleFromTweet)) {
|
||||
return articleFromTweet;
|
||||
}
|
||||
|
||||
const articlePayload = await fetchArticleEntityById(articleId, cookieMap, userAgent, bearerToken);
|
||||
const articleFromEntity = extractArticleFromEntity(articlePayload);
|
||||
if (isNonEmptyObject(articleFromEntity)) {
|
||||
return articleFromEntity;
|
||||
}
|
||||
return articleFromEntity ?? articlePayload;
|
||||
}
|
||||
|
||||
export async function fetchXTweet(
|
||||
tweetId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
raw: boolean
|
||||
): Promise<unknown> {
|
||||
const userAgent = process.env.X_USER_AGENT?.trim() || DEFAULT_USER_AGENT;
|
||||
const bearerToken = process.env.X_BEARER_TOKEN?.trim() || DEFAULT_BEARER_TOKEN;
|
||||
|
||||
const tweetPayload = await fetchTweetResult(tweetId, cookieMap, userAgent, bearerToken);
|
||||
if (raw) {
|
||||
return tweetPayload;
|
||||
}
|
||||
|
||||
const tweet = extractTweetFromPayload(tweetPayload);
|
||||
if (isNonEmptyObject(tweet)) {
|
||||
return tweet;
|
||||
}
|
||||
return tweet ?? tweetPayload;
|
||||
}
|
||||
117
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/http.ts
Normal file
117
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/http.ts
Normal file
@@ -0,0 +1,117 @@
|
||||
import { buildCookieHeader } from "./cookies.js";
|
||||
|
||||
let cachedHomeHtml: { userAgent: string; html: string } | null = null;
|
||||
|
||||
export async function fetchText(url: string, init?: RequestInit): Promise<string> {
|
||||
const response = await fetch(url, init);
|
||||
const text = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`Request failed (${response.status}) for ${url}: ${text.slice(0, 200)}`);
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
export async function fetchHomeHtml(userAgent: string): Promise<string> {
|
||||
if (cachedHomeHtml?.userAgent === userAgent) {
|
||||
return cachedHomeHtml.html;
|
||||
}
|
||||
const html = await fetchText("https://x.com", {
|
||||
headers: {
|
||||
"user-agent": userAgent,
|
||||
},
|
||||
});
|
||||
cachedHomeHtml = { userAgent, html };
|
||||
return html;
|
||||
}
|
||||
|
||||
export function parseStringList(raw: string | undefined): string[] {
|
||||
if (!raw) return [];
|
||||
return raw
|
||||
.split(",")
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean)
|
||||
.map((item) => item.replace(/^\"|\"$/g, ""));
|
||||
}
|
||||
|
||||
export function resolveFeatureValue(html: string, key: string): boolean | undefined {
|
||||
const keyPattern = key.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
const unescaped = new RegExp(`"${keyPattern}"\\s*:\\s*\\{"value"\\s*:\\s*(true|false)`);
|
||||
const escaped = new RegExp(`\\\\"${keyPattern}\\\\"\\s*:\\s*\\\\{\\\\"value\\\\"\\s*:\\s*(true|false)`);
|
||||
const match = html.match(unescaped) ?? html.match(escaped);
|
||||
if (!match) return undefined;
|
||||
return match[1] === "true";
|
||||
}
|
||||
|
||||
export function buildFeatureMap(
|
||||
html: string,
|
||||
keys: string[],
|
||||
defaults?: Record<string, boolean>
|
||||
): Record<string, boolean> {
|
||||
const features: Record<string, boolean> = {};
|
||||
for (const key of keys) {
|
||||
const value = resolveFeatureValue(html, key);
|
||||
if (value !== undefined) {
|
||||
features[key] = value;
|
||||
} else if (defaults && Object.prototype.hasOwnProperty.call(defaults, key)) {
|
||||
features[key] = defaults[key] ?? true;
|
||||
} else {
|
||||
features[key] = true;
|
||||
}
|
||||
}
|
||||
if (!Object.prototype.hasOwnProperty.call(features, "responsive_web_graphql_exclude_directive_enabled")) {
|
||||
features.responsive_web_graphql_exclude_directive_enabled = true;
|
||||
}
|
||||
return features;
|
||||
}
|
||||
|
||||
export function buildFieldToggleMap(keys: string[]): Record<string, boolean> {
|
||||
const toggles: Record<string, boolean> = {};
|
||||
for (const key of keys) {
|
||||
toggles[key] = true;
|
||||
}
|
||||
return toggles;
|
||||
}
|
||||
|
||||
export function buildTweetFieldToggleMap(keys: string[]): Record<string, boolean> {
|
||||
const toggles: Record<string, boolean> = {};
|
||||
for (const key of keys) {
|
||||
if (key === "withGrokAnalyze" || key === "withDisallowedReplyControls") {
|
||||
toggles[key] = false;
|
||||
} else {
|
||||
toggles[key] = true;
|
||||
}
|
||||
}
|
||||
return toggles;
|
||||
}
|
||||
|
||||
export function buildRequestHeaders(
|
||||
cookieMap: Record<string, string>,
|
||||
userAgent: string,
|
||||
bearerToken: string
|
||||
): Record<string, string> {
|
||||
const headers: Record<string, string> = {
|
||||
authorization: bearerToken,
|
||||
"user-agent": userAgent,
|
||||
accept: "application/json",
|
||||
"x-twitter-active-user": "yes",
|
||||
"x-twitter-client-language": "en",
|
||||
"accept-language": "en",
|
||||
};
|
||||
|
||||
if (cookieMap.auth_token) {
|
||||
headers["x-twitter-auth-type"] = "OAuth2Session";
|
||||
}
|
||||
|
||||
const cookieHeader = buildCookieHeader(cookieMap);
|
||||
if (cookieHeader) {
|
||||
headers.cookie = cookieHeader;
|
||||
}
|
||||
if (cookieMap.ct0) {
|
||||
headers["x-csrf-token"] = cookieMap.ct0;
|
||||
}
|
||||
if (process.env.X_CLIENT_TRANSACTION_ID?.trim()) {
|
||||
headers["x-client-transaction-id"] = process.env.X_CLIENT_TRANSACTION_ID.trim();
|
||||
}
|
||||
|
||||
return headers;
|
||||
}
|
||||
640
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/main.ts
Normal file
640
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/main.ts
Normal file
@@ -0,0 +1,640 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import readline from "node:readline";
|
||||
import process from "node:process";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
|
||||
import { fetchXArticle } from "./graphql.js";
|
||||
import { formatArticleMarkdown } from "./markdown.js";
|
||||
import { localizeMarkdownMedia, type LocalizeMarkdownMediaResult } from "./media-localizer.js";
|
||||
import { resolveReferencedTweetsFromArticle } from "./referenced-tweets.js";
|
||||
import { hasRequiredXCookies, loadXCookies, refreshXCookies } from "./cookies.js";
|
||||
import { resolveXToMarkdownConsentPath } from "./paths.js";
|
||||
import { tweetToMarkdown } from "./tweet-to-markdown.js";
|
||||
|
||||
type CliArgs = {
|
||||
url: string | null;
|
||||
output: string | null;
|
||||
json: boolean;
|
||||
login: boolean;
|
||||
downloadMedia: boolean;
|
||||
help: boolean;
|
||||
};
|
||||
|
||||
type ConsentRecord = {
|
||||
version: number;
|
||||
accepted: boolean;
|
||||
acceptedAt: string;
|
||||
disclaimerVersion: string;
|
||||
};
|
||||
|
||||
const DISCLAIMER_VERSION = "1.0";
|
||||
|
||||
function formatScriptCommand(fallback: string): string {
|
||||
const raw = process.argv[1];
|
||||
const displayPath = raw
|
||||
? (() => {
|
||||
const relative = path.relative(process.cwd(), raw);
|
||||
return relative && !relative.startsWith("..") ? relative : raw;
|
||||
})()
|
||||
: fallback;
|
||||
const quotedPath = displayPath.includes(" ")
|
||||
? `"${displayPath.replace(/"/g, '\\"')}"`
|
||||
: displayPath;
|
||||
return `npx -y bun ${quotedPath}`;
|
||||
}
|
||||
|
||||
function printUsage(exitCode: number): never {
|
||||
const cmd = formatScriptCommand("scripts/main.ts");
|
||||
console.log(`X (Twitter) to Markdown
|
||||
|
||||
Usage:
|
||||
${cmd} <url>
|
||||
${cmd} --url <url>
|
||||
|
||||
Options:
|
||||
--output <path>, -o Output path (file or dir). Default: ./x-to-markdown/<slug>/
|
||||
--json Output as JSON
|
||||
--download-media Download images/videos to local ./imgs and ./videos next to markdown
|
||||
--login Refresh cookies only, then exit
|
||||
--help, -h Show help
|
||||
|
||||
Examples:
|
||||
${cmd} https://x.com/username/status/1234567890
|
||||
${cmd} https://x.com/i/article/1234567890 -o ./article.md
|
||||
${cmd} https://x.com/username/status/1234567890 -o ./out/
|
||||
${cmd} https://x.com/username/status/1234567890 --download-media
|
||||
${cmd} https://x.com/username/status/1234567890 --json | jq -r '.markdownPath'
|
||||
${cmd} --login
|
||||
`);
|
||||
process.exit(exitCode);
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): CliArgs {
|
||||
const out: CliArgs = {
|
||||
url: null,
|
||||
output: null,
|
||||
json: false,
|
||||
login: false,
|
||||
downloadMedia: false,
|
||||
help: false,
|
||||
};
|
||||
|
||||
const positional: string[] = [];
|
||||
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i]!;
|
||||
|
||||
if (a === "--help" || a === "-h") {
|
||||
out.help = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a === "--json") {
|
||||
out.json = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a === "--login") {
|
||||
out.login = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a === "--download-media") {
|
||||
out.downloadMedia = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a === "--url") {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error("Missing value for --url");
|
||||
out.url = v;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a === "--output" || a === "-o") {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`Missing value for ${a}`);
|
||||
out.output = v;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (a.startsWith("-")) {
|
||||
throw new Error(`Unknown option: ${a}`);
|
||||
}
|
||||
|
||||
positional.push(a);
|
||||
}
|
||||
|
||||
if (!out.url && positional.length > 0) {
|
||||
out.url = positional[0]!;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
function normalizeInputUrl(input: string): string {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return "";
|
||||
try {
|
||||
return new URL(trimmed).toString();
|
||||
} catch {
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
|
||||
function parseArticleId(input: string): string | null {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return null;
|
||||
|
||||
try {
|
||||
const parsed = new URL(trimmed);
|
||||
const match = parsed.pathname.match(/\/(?:i\/)?article\/(\d+)/);
|
||||
if (match?.[1]) return match[1];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseTweetId(input: string): string | null {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return null;
|
||||
if (/^\d+$/.test(trimmed)) return trimmed;
|
||||
|
||||
try {
|
||||
const parsed = new URL(trimmed);
|
||||
const match = parsed.pathname.match(/\/status(?:es)?\/(\d+)/);
|
||||
if (match?.[1]) return match[1];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseTweetUsername(input: string): string | null {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return null;
|
||||
try {
|
||||
const parsed = new URL(trimmed);
|
||||
const match = parsed.pathname.match(/^\/([^/]+)\/status(?:es)?\/\d+/);
|
||||
if (match?.[1]) return match[1];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function sanitizeSlug(input: string): string {
|
||||
return input
|
||||
.trim()
|
||||
.replace(/^@/, "")
|
||||
.replace(/[^a-zA-Z0-9_-]+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^[-_]+|[-_]+$/g, "")
|
||||
.slice(0, 120);
|
||||
}
|
||||
|
||||
function extractContentSlug(markdown: string): string {
|
||||
const headingMatch = markdown.match(/^#\s+(.+)$/m);
|
||||
if (headingMatch?.[1]) {
|
||||
return sanitizeSlug(headingMatch[1].slice(0, 60)).toLowerCase();
|
||||
}
|
||||
const lines = markdown.split("\n");
|
||||
let inFrontmatter = false;
|
||||
for (const line of lines) {
|
||||
if (line === "---") {
|
||||
inFrontmatter = !inFrontmatter;
|
||||
continue;
|
||||
}
|
||||
if (inFrontmatter) continue;
|
||||
const trimmed = line.trim();
|
||||
if (trimmed) {
|
||||
return sanitizeSlug(trimmed.slice(0, 60)).toLowerCase();
|
||||
}
|
||||
}
|
||||
return "untitled";
|
||||
}
|
||||
|
||||
function resolveSlugAndId(normalizedUrl: string, kind: "tweet" | "article"): { slug: string; idPart: string } {
|
||||
const articleId = kind === "article" ? parseArticleId(normalizedUrl) : null;
|
||||
const tweetId = kind === "tweet" ? parseTweetId(normalizedUrl) : null;
|
||||
const username = kind === "tweet" ? parseTweetUsername(normalizedUrl) : null;
|
||||
|
||||
const idPart = articleId ?? tweetId ?? String(Date.now());
|
||||
const userSlug = username ? sanitizeSlug(username) : null;
|
||||
const slug = userSlug ?? idPart;
|
||||
return { slug, idPart };
|
||||
}
|
||||
|
||||
function extractFrontmatterUrls(markdown: string): string[] {
|
||||
const match = markdown.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!match?.[1]) return [];
|
||||
|
||||
const lines = match[1].split("\n");
|
||||
const urls: string[] = [];
|
||||
for (const line of lines) {
|
||||
const m = line.match(/^(url|requestedUrl):\s*["']([^"']+)["']\s*$/);
|
||||
if (m?.[2]) {
|
||||
urls.push(m[2]);
|
||||
}
|
||||
}
|
||||
return urls;
|
||||
}
|
||||
|
||||
function frontmatterMatchesTarget(
|
||||
markdown: string,
|
||||
normalizedUrl: string,
|
||||
kind: "tweet" | "article"
|
||||
): boolean {
|
||||
const urls = extractFrontmatterUrls(markdown);
|
||||
if (urls.length === 0) return false;
|
||||
|
||||
const targetId = kind === "article" ? parseArticleId(normalizedUrl) : parseTweetId(normalizedUrl);
|
||||
if (!targetId) return false;
|
||||
|
||||
for (const url of urls) {
|
||||
const candidateId = kind === "article" ? parseArticleId(url) : parseTweetId(url);
|
||||
if (candidateId && candidateId === targetId) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
function listMarkdownFiles(dirPath: string): string[] {
|
||||
try {
|
||||
return fs
|
||||
.readdirSync(dirPath)
|
||||
.filter((name) => name.toLowerCase().endsWith(".md"))
|
||||
.map((name) => path.join(dirPath, name))
|
||||
.sort();
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
function resolveExistingMarkdownPath(
|
||||
normalizedUrl: string,
|
||||
kind: "tweet" | "article",
|
||||
argsOutput: string | null
|
||||
): string | null {
|
||||
const { slug, idPart } = resolveSlugAndId(normalizedUrl, kind);
|
||||
const candidateDirs = new Set<string>();
|
||||
const candidateFiles = new Set<string>();
|
||||
|
||||
if (argsOutput) {
|
||||
const resolved = path.resolve(argsOutput);
|
||||
const looksDir = argsOutput.endsWith("/") || argsOutput.endsWith("\\");
|
||||
try {
|
||||
if (fs.existsSync(resolved)) {
|
||||
const stat = fs.statSync(resolved);
|
||||
if (stat.isFile()) {
|
||||
candidateFiles.add(resolved);
|
||||
} else if (stat.isDirectory()) {
|
||||
candidateDirs.add(path.join(resolved, slug, idPart));
|
||||
candidateDirs.add(resolved);
|
||||
}
|
||||
} else if (looksDir) {
|
||||
candidateDirs.add(path.join(resolved, slug, idPart));
|
||||
}
|
||||
} catch {
|
||||
// ignore and continue
|
||||
}
|
||||
} else {
|
||||
candidateDirs.add(path.resolve(process.cwd(), "x-to-markdown", slug, idPart));
|
||||
}
|
||||
|
||||
for (const filePath of candidateFiles) {
|
||||
if (!filePath.toLowerCase().endsWith(".md")) continue;
|
||||
try {
|
||||
const markdown = fs.readFileSync(filePath, "utf8");
|
||||
if (frontmatterMatchesTarget(markdown, normalizedUrl, kind)) {
|
||||
return filePath;
|
||||
}
|
||||
} catch {
|
||||
// ignore and continue
|
||||
}
|
||||
}
|
||||
|
||||
for (const dirPath of candidateDirs) {
|
||||
if (!fs.existsSync(dirPath)) continue;
|
||||
let stat: fs.Stats;
|
||||
try {
|
||||
stat = fs.statSync(dirPath);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!stat.isDirectory()) continue;
|
||||
|
||||
const markdownFiles = listMarkdownFiles(dirPath);
|
||||
for (const filePath of markdownFiles) {
|
||||
try {
|
||||
const markdown = fs.readFileSync(filePath, "utf8");
|
||||
if (frontmatterMatchesTarget(markdown, normalizedUrl, kind)) {
|
||||
return filePath;
|
||||
}
|
||||
} catch {
|
||||
// ignore and continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
async function resolveOutputPath(
|
||||
normalizedUrl: string,
|
||||
kind: "tweet" | "article",
|
||||
argsOutput: string | null,
|
||||
contentSlug: string,
|
||||
log: (message: string) => void
|
||||
): Promise<{ outputDir: string; markdownPath: string; slug: string }> {
|
||||
const articleId = kind === "article" ? parseArticleId(normalizedUrl) : null;
|
||||
const tweetId = kind === "tweet" ? parseTweetId(normalizedUrl) : null;
|
||||
const username = kind === "tweet" ? parseTweetUsername(normalizedUrl) : null;
|
||||
|
||||
const userSlug = username ? sanitizeSlug(username) : null;
|
||||
const idPart = articleId ?? tweetId ?? String(Date.now());
|
||||
const slug = userSlug ?? idPart;
|
||||
|
||||
const defaultFileName = `${contentSlug}.md`;
|
||||
|
||||
if (argsOutput) {
|
||||
const wantsDir = argsOutput.endsWith("/") || argsOutput.endsWith("\\");
|
||||
const resolved = path.resolve(argsOutput);
|
||||
try {
|
||||
if (wantsDir || (fs.existsSync(resolved) && fs.statSync(resolved).isDirectory())) {
|
||||
const outputDir = path.join(resolved, slug, idPart);
|
||||
await mkdir(outputDir, { recursive: true });
|
||||
return { outputDir, markdownPath: path.join(outputDir, defaultFileName), slug };
|
||||
}
|
||||
} catch {
|
||||
// treat as file path
|
||||
}
|
||||
|
||||
const outputDir = path.dirname(resolved);
|
||||
await mkdir(outputDir, { recursive: true });
|
||||
return { outputDir, markdownPath: resolved, slug };
|
||||
}
|
||||
|
||||
const outputDir = path.resolve(process.cwd(), "x-to-markdown", slug, idPart);
|
||||
await mkdir(outputDir, { recursive: true });
|
||||
return { outputDir, markdownPath: path.join(outputDir, defaultFileName), slug };
|
||||
}
|
||||
|
||||
function formatMetaMarkdown(meta: Record<string, string | number | null | undefined>): string {
|
||||
const lines = ["---"];
|
||||
for (const [key, value] of Object.entries(meta)) {
|
||||
if (value === undefined || value === null || value === "") continue;
|
||||
if (typeof value === "number") {
|
||||
lines.push(`${key}: ${value}`);
|
||||
} else {
|
||||
lines.push(`${key}: ${JSON.stringify(value)}`);
|
||||
}
|
||||
}
|
||||
lines.push("---");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function promptYesNo(question: string): Promise<boolean> {
|
||||
if (!process.stdin.isTTY) return false;
|
||||
|
||||
const rl = readline.createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stderr,
|
||||
});
|
||||
|
||||
try {
|
||||
const answer = await new Promise<string>((resolve) => rl.question(question, resolve));
|
||||
const normalized = answer.trim().toLowerCase();
|
||||
return normalized === "y" || normalized === "yes";
|
||||
} finally {
|
||||
rl.close();
|
||||
}
|
||||
}
|
||||
|
||||
function isValidConsent(value: unknown): value is ConsentRecord {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const record = value as Partial<ConsentRecord>;
|
||||
return (
|
||||
record.accepted === true &&
|
||||
record.disclaimerVersion === DISCLAIMER_VERSION &&
|
||||
typeof record.acceptedAt === "string" &&
|
||||
record.acceptedAt.length > 0
|
||||
);
|
||||
}
|
||||
|
||||
async function ensureConsent(log: (message: string) => void): Promise<void> {
|
||||
const consentPath = resolveXToMarkdownConsentPath();
|
||||
|
||||
try {
|
||||
if (fs.existsSync(consentPath) && fs.statSync(consentPath).isFile()) {
|
||||
const raw = await readFile(consentPath, "utf8");
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (isValidConsent(parsed)) {
|
||||
log(
|
||||
`⚠️ Warning: Using reverse-engineered X API (not official). Accepted on: ${(parsed as ConsentRecord).acceptedAt}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// fall through to prompt
|
||||
}
|
||||
|
||||
log(`⚠️ DISCLAIMER
|
||||
|
||||
This tool uses a reverse-engineered X (Twitter) API, NOT an official API.
|
||||
|
||||
Risks:
|
||||
- May break without notice if X changes their API
|
||||
- No official support or guarantees
|
||||
- Account restrictions possible if API usage detected
|
||||
- Use at your own risk
|
||||
`);
|
||||
|
||||
if (!process.stdin.isTTY) {
|
||||
throw new Error(
|
||||
`Consent required. Run in a TTY or create ${consentPath} with accepted: true and disclaimerVersion: ${DISCLAIMER_VERSION}`
|
||||
);
|
||||
}
|
||||
|
||||
const accepted = await promptYesNo("Do you accept these terms and wish to continue? (y/N): ");
|
||||
if (!accepted) {
|
||||
throw new Error("User declined the disclaimer. Exiting.");
|
||||
}
|
||||
|
||||
await mkdir(path.dirname(consentPath), { recursive: true });
|
||||
const payload: ConsentRecord = {
|
||||
version: 1,
|
||||
accepted: true,
|
||||
acceptedAt: new Date().toISOString(),
|
||||
disclaimerVersion: DISCLAIMER_VERSION,
|
||||
};
|
||||
await writeFile(consentPath, JSON.stringify(payload, null, 2), "utf8");
|
||||
log(`[x-to-markdown] Consent saved to: ${consentPath}`);
|
||||
}
|
||||
|
||||
async function convertArticleToMarkdown(
|
||||
inputUrl: string,
|
||||
articleId: string,
|
||||
log: (message: string) => void
|
||||
): Promise<string> {
|
||||
log("[x-to-markdown] Loading cookies...");
|
||||
const cookieMap = await loadXCookies(log);
|
||||
if (!hasRequiredXCookies(cookieMap)) {
|
||||
throw new Error("Missing auth cookies. Provide X_AUTH_TOKEN and X_CT0 or log in via Chrome.");
|
||||
}
|
||||
|
||||
log(`[x-to-markdown] Fetching article ${articleId}...`);
|
||||
const article = await fetchXArticle(articleId, cookieMap, false);
|
||||
const referencedTweets = await resolveReferencedTweetsFromArticle(article, cookieMap, { log });
|
||||
const { markdown: body, coverUrl } = formatArticleMarkdown(article, { referencedTweets });
|
||||
|
||||
const title = typeof (article as any)?.title === "string" ? String((article as any).title).trim() : "";
|
||||
const meta = formatMetaMarkdown({
|
||||
url: `https://x.com/i/article/${articleId}`,
|
||||
requestedUrl: inputUrl,
|
||||
title: title || null,
|
||||
coverImage: coverUrl,
|
||||
});
|
||||
|
||||
return [meta, body.trimEnd()].filter(Boolean).join("\n\n").trimEnd();
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) printUsage(0);
|
||||
if (!args.login && !args.url) printUsage(1);
|
||||
|
||||
const log = (message: string) => console.error(message);
|
||||
await ensureConsent(log);
|
||||
|
||||
if (args.login) {
|
||||
log("[x-to-markdown] Refreshing cookies via browser login...");
|
||||
const cookieMap = await refreshXCookies(log);
|
||||
if (!hasRequiredXCookies(cookieMap)) {
|
||||
throw new Error("Missing auth cookies after login. Please ensure you are logged in to X.");
|
||||
}
|
||||
log("[x-to-markdown] Cookies refreshed.");
|
||||
return;
|
||||
}
|
||||
|
||||
const normalizedUrl = normalizeInputUrl(args.url ?? "");
|
||||
const articleId = parseArticleId(normalizedUrl);
|
||||
const tweetId = parseTweetId(normalizedUrl);
|
||||
if (!articleId && !tweetId) {
|
||||
throw new Error("Invalid X url. Examples: https://x.com/<user>/status/<id> or https://x.com/i/article/<id>");
|
||||
}
|
||||
|
||||
const kind = articleId ? ("article" as const) : ("tweet" as const);
|
||||
|
||||
if (args.downloadMedia) {
|
||||
const existingMarkdownPath = resolveExistingMarkdownPath(normalizedUrl, kind, args.output);
|
||||
if (existingMarkdownPath) {
|
||||
log(`[x-to-markdown] Reusing existing markdown: ${existingMarkdownPath}`);
|
||||
const existingMarkdown = await readFile(existingMarkdownPath, "utf8");
|
||||
const mediaResult = await localizeMarkdownMedia(existingMarkdown, {
|
||||
markdownPath: existingMarkdownPath,
|
||||
log,
|
||||
});
|
||||
const didLocalize =
|
||||
mediaResult.downloadedImages > 0 ||
|
||||
mediaResult.downloadedVideos > 0 ||
|
||||
mediaResult.markdown !== existingMarkdown;
|
||||
|
||||
if (didLocalize) {
|
||||
await writeFile(existingMarkdownPath, mediaResult.markdown, "utf8");
|
||||
log(
|
||||
`[x-to-markdown] Media localized: images=${mediaResult.downloadedImages}, videos=${mediaResult.downloadedVideos}`
|
||||
);
|
||||
log(`[x-to-markdown] Saved: ${existingMarkdownPath}`);
|
||||
|
||||
const { slug } = resolveSlugAndId(normalizedUrl, kind);
|
||||
if (args.json) {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
url: articleId ? `https://x.com/i/article/${articleId}` : normalizedUrl,
|
||||
requestedUrl: normalizedUrl,
|
||||
type: kind,
|
||||
slug,
|
||||
outputDir: path.dirname(existingMarkdownPath),
|
||||
markdownPath: existingMarkdownPath,
|
||||
downloadMedia: true,
|
||||
downloadedImages: mediaResult.downloadedImages,
|
||||
downloadedVideos: mediaResult.downloadedVideos,
|
||||
imageDir: mediaResult.imageDir,
|
||||
videoDir: mediaResult.videoDir,
|
||||
},
|
||||
null,
|
||||
2
|
||||
)
|
||||
);
|
||||
} else {
|
||||
console.log(existingMarkdownPath);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
log("[x-to-markdown] Existing markdown already localized; rebuilding content to refresh placement.");
|
||||
}
|
||||
}
|
||||
|
||||
let markdown =
|
||||
kind === "article" && articleId
|
||||
? await convertArticleToMarkdown(normalizedUrl, articleId, log)
|
||||
: await tweetToMarkdown(normalizedUrl, { log });
|
||||
|
||||
const contentSlug = extractContentSlug(markdown);
|
||||
const { outputDir, markdownPath, slug } = await resolveOutputPath(normalizedUrl, kind, args.output, contentSlug, log);
|
||||
|
||||
let mediaResult: LocalizeMarkdownMediaResult | null = null;
|
||||
|
||||
if (args.downloadMedia) {
|
||||
mediaResult = await localizeMarkdownMedia(markdown, {
|
||||
markdownPath,
|
||||
log,
|
||||
});
|
||||
markdown = mediaResult.markdown;
|
||||
log(
|
||||
`[x-to-markdown] Media localized: images=${mediaResult.downloadedImages}, videos=${mediaResult.downloadedVideos}`
|
||||
);
|
||||
}
|
||||
|
||||
await writeFile(markdownPath, markdown, "utf8");
|
||||
log(`[x-to-markdown] Saved: ${markdownPath}`);
|
||||
|
||||
if (args.json) {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
url: articleId ? `https://x.com/i/article/${articleId}` : normalizedUrl,
|
||||
requestedUrl: normalizedUrl,
|
||||
type: kind,
|
||||
slug,
|
||||
outputDir,
|
||||
markdownPath,
|
||||
downloadMedia: args.downloadMedia,
|
||||
downloadedImages: mediaResult?.downloadedImages ?? 0,
|
||||
downloadedVideos: mediaResult?.downloadedVideos ?? 0,
|
||||
imageDir: mediaResult?.imageDir ?? null,
|
||||
videoDir: mediaResult?.videoDir ?? null,
|
||||
},
|
||||
null,
|
||||
2
|
||||
)
|
||||
);
|
||||
} else {
|
||||
console.log(markdownPath);
|
||||
}
|
||||
}
|
||||
|
||||
await main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error ?? ""));
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,179 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
import { formatArticleMarkdown } from "./markdown.js";
|
||||
|
||||
test("formatArticleMarkdown renders MARKDOWN entities from atomic blocks", () => {
|
||||
const article = {
|
||||
title: "Atomic Markdown Example",
|
||||
content_state: {
|
||||
blocks: [
|
||||
{
|
||||
type: "unstyled",
|
||||
text: "Before the snippet.",
|
||||
entityRanges: [],
|
||||
},
|
||||
{
|
||||
type: "atomic",
|
||||
text: " ",
|
||||
entityRanges: [{ key: 0, offset: 0, length: 1 }],
|
||||
},
|
||||
{
|
||||
type: "unstyled",
|
||||
text: "After the snippet.",
|
||||
entityRanges: [],
|
||||
},
|
||||
],
|
||||
entityMap: {
|
||||
"0": {
|
||||
key: "5",
|
||||
value: {
|
||||
type: "MARKDOWN",
|
||||
mutability: "Mutable",
|
||||
data: {
|
||||
markdown: "```python\nprint('hello from x article')\n```\n",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const { markdown } = formatArticleMarkdown(article);
|
||||
|
||||
assert.ok(markdown.includes("Before the snippet."));
|
||||
assert.ok(markdown.includes("```python\nprint('hello from x article')\n```"));
|
||||
assert.ok(markdown.includes("After the snippet."));
|
||||
assert.strictEqual(markdown, `# Atomic Markdown Example
|
||||
|
||||
Before the snippet.
|
||||
|
||||
\`\`\`python
|
||||
print('hello from x article')
|
||||
\`\`\`
|
||||
|
||||
After the snippet.`);
|
||||
});
|
||||
|
||||
test("formatArticleMarkdown renders article video media as poster plus video link", () => {
|
||||
const posterUrl = "https://pbs.twimg.com/amplify_video_thumb/123/img/poster.jpg";
|
||||
const videoUrl = "https://video.twimg.com/amplify_video/123/vid/avc1/720x720/demo.mp4?tag=21";
|
||||
const article = {
|
||||
title: "Video Example",
|
||||
content_state: {
|
||||
blocks: [
|
||||
{
|
||||
type: "unstyled",
|
||||
text: "Intro text.",
|
||||
entityRanges: [],
|
||||
},
|
||||
{
|
||||
type: "atomic",
|
||||
text: " ",
|
||||
entityRanges: [{ key: 0, offset: 0, length: 1 }],
|
||||
},
|
||||
],
|
||||
entityMap: {
|
||||
"0": {
|
||||
key: "0",
|
||||
value: {
|
||||
type: "MEDIA",
|
||||
mutability: "Immutable",
|
||||
data: {
|
||||
caption: "Demo reel",
|
||||
mediaItems: [{ mediaId: "vid-1" }],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
media_entities: [
|
||||
{
|
||||
media_id: "vid-1",
|
||||
media_info: {
|
||||
__typename: "ApiVideo",
|
||||
preview_image: {
|
||||
original_img_url: posterUrl,
|
||||
},
|
||||
variants: [
|
||||
{
|
||||
content_type: "video/mp4",
|
||||
bit_rate: 256000,
|
||||
url: videoUrl,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const { markdown } = formatArticleMarkdown(article);
|
||||
|
||||
assert.ok(markdown.includes("Intro text."));
|
||||
assert.ok(markdown.includes(``));
|
||||
assert.ok(markdown.includes(`[video](${videoUrl})`));
|
||||
assert.ok(!markdown.includes(``));
|
||||
assert.ok(!markdown.includes("## Media"));
|
||||
});
|
||||
|
||||
test("formatArticleMarkdown renders unused article videos in trailing media section", () => {
|
||||
const posterUrl = "https://pbs.twimg.com/amplify_video_thumb/456/img/poster.jpg";
|
||||
const videoUrl = "https://video.twimg.com/amplify_video/456/vid/avc1/1080x1080/demo.mp4?tag=21";
|
||||
const article = {
|
||||
title: "Trailing Media Example",
|
||||
plain_text: "Body text.",
|
||||
media_entities: [
|
||||
{
|
||||
media_id: "vid-2",
|
||||
media_info: {
|
||||
__typename: "ApiVideo",
|
||||
preview_image: {
|
||||
original_img_url: posterUrl,
|
||||
},
|
||||
variants: [
|
||||
{
|
||||
content_type: "video/mp4",
|
||||
bit_rate: 832000,
|
||||
url: videoUrl,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const { markdown, coverUrl } = formatArticleMarkdown(article);
|
||||
|
||||
assert.strictEqual(coverUrl, null);
|
||||
assert.ok(markdown.includes("## Media"));
|
||||
assert.ok(markdown.includes(``));
|
||||
assert.ok(markdown.includes(`[video](${videoUrl})`));
|
||||
});
|
||||
|
||||
test("formatArticleMarkdown keeps coverUrl as preview image for video cover media", () => {
|
||||
const posterUrl = "https://pbs.twimg.com/amplify_video_thumb/789/img/poster.jpg";
|
||||
const videoUrl = "https://video.twimg.com/amplify_video/789/vid/avc1/720x720/demo.mp4?tag=21";
|
||||
const article = {
|
||||
title: "Video Cover Example",
|
||||
plain_text: "Body text.",
|
||||
cover_media: {
|
||||
media_info: {
|
||||
__typename: "ApiVideo",
|
||||
preview_image: {
|
||||
original_img_url: posterUrl,
|
||||
},
|
||||
variants: [
|
||||
{
|
||||
content_type: "video/mp4",
|
||||
bit_rate: 1280000,
|
||||
url: videoUrl,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const { coverUrl } = formatArticleMarkdown(article);
|
||||
|
||||
assert.strictEqual(coverUrl, posterUrl);
|
||||
});
|
||||
@@ -0,0 +1,725 @@
|
||||
import type {
|
||||
ArticleBlock,
|
||||
ArticleContentState,
|
||||
ArticleEntity,
|
||||
ArticleEntityMapEntry,
|
||||
ArticleMediaInfo,
|
||||
} from "./types.js";
|
||||
|
||||
export type ReferencedTweetInfo = {
|
||||
id: string;
|
||||
url: string;
|
||||
authorName?: string;
|
||||
authorUsername?: string;
|
||||
text?: string;
|
||||
};
|
||||
|
||||
export type FormatArticleOptions = {
|
||||
referencedTweets?: Map<string, ReferencedTweetInfo>;
|
||||
};
|
||||
|
||||
type ResolvedMediaAsset =
|
||||
| {
|
||||
kind: "image";
|
||||
url: string;
|
||||
}
|
||||
| {
|
||||
kind: "video";
|
||||
url: string;
|
||||
posterUrl?: string;
|
||||
};
|
||||
|
||||
function coerceArticleEntity(value: unknown): ArticleEntity | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const candidate = value as ArticleEntity;
|
||||
if (
|
||||
typeof candidate.title === "string" ||
|
||||
typeof candidate.plain_text === "string" ||
|
||||
typeof candidate.preview_text === "string" ||
|
||||
candidate.content_state
|
||||
) {
|
||||
return candidate;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function escapeMarkdownAlt(text: string): string {
|
||||
return text.replace(/[\[\]]/g, "\\$&");
|
||||
}
|
||||
|
||||
function normalizeCaption(caption?: string): string {
|
||||
const trimmed = caption?.trim();
|
||||
if (!trimmed) return "";
|
||||
return trimmed.replace(/\s+/g, " ");
|
||||
}
|
||||
|
||||
function summarizeTweetText(text?: string): string {
|
||||
const trimmed = text?.trim();
|
||||
if (!trimmed) return "";
|
||||
const normalized = trimmed
|
||||
.split(/\r?\n+/)
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean)
|
||||
.join(" ");
|
||||
if (normalized.length <= 280) return normalized;
|
||||
return `${normalized.slice(0, 277)}...`;
|
||||
}
|
||||
|
||||
function buildTweetUrl(tweetId?: string, username?: string): string | null {
|
||||
if (!tweetId) return null;
|
||||
if (username) {
|
||||
return `https://x.com/${username}/status/${tweetId}`;
|
||||
}
|
||||
return `https://x.com/i/web/status/${tweetId}`;
|
||||
}
|
||||
|
||||
type EntityLookup = {
|
||||
byIndex: Map<number, ArticleEntityMapEntry>;
|
||||
byLogicalKey: Map<number, ArticleEntityMapEntry>;
|
||||
};
|
||||
|
||||
function buildEntityLookup(
|
||||
entityMap: ArticleContentState["entityMap"] | undefined
|
||||
): EntityLookup {
|
||||
const lookup: EntityLookup = {
|
||||
byIndex: new Map<number, ArticleEntityMapEntry>(),
|
||||
byLogicalKey: new Map<number, ArticleEntityMapEntry>(),
|
||||
};
|
||||
|
||||
if (!entityMap) return lookup;
|
||||
|
||||
for (const [idx, entry] of Object.entries(entityMap)) {
|
||||
const idxNum = Number(idx);
|
||||
if (Number.isFinite(idxNum)) {
|
||||
lookup.byIndex.set(idxNum, entry);
|
||||
}
|
||||
|
||||
const logicalKey = parseInt(entry?.key ?? "", 10);
|
||||
if (Number.isFinite(logicalKey) && !lookup.byLogicalKey.has(logicalKey)) {
|
||||
lookup.byLogicalKey.set(logicalKey, entry);
|
||||
}
|
||||
}
|
||||
|
||||
return lookup;
|
||||
}
|
||||
|
||||
function resolveEntityEntry(
|
||||
entityKey: number | undefined,
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
lookup: EntityLookup
|
||||
): ArticleEntityMapEntry | undefined {
|
||||
if (entityKey === undefined) return undefined;
|
||||
|
||||
const byLogicalKey = lookup.byLogicalKey.get(entityKey);
|
||||
if (byLogicalKey) return byLogicalKey;
|
||||
|
||||
const byIndex = lookup.byIndex.get(entityKey);
|
||||
if (byIndex) return byIndex;
|
||||
|
||||
if (!entityMap) return undefined;
|
||||
return entityMap[String(entityKey)];
|
||||
}
|
||||
|
||||
function resolveVideoUrl(info?: ArticleMediaInfo): string | undefined {
|
||||
if (!info) return undefined;
|
||||
const variants = info.variants ?? [];
|
||||
const mp4 = variants
|
||||
.filter((variant) => variant?.content_type?.includes("video"))
|
||||
.sort((a, b) => (b.bit_rate ?? 0) - (a.bit_rate ?? 0))[0];
|
||||
return mp4?.url ?? variants.find((variant) => typeof variant?.url === "string")?.url;
|
||||
}
|
||||
|
||||
function resolveMediaAsset(info?: ArticleMediaInfo): ResolvedMediaAsset | undefined {
|
||||
if (!info) return undefined;
|
||||
|
||||
const posterUrl = info.preview_image?.original_img_url ?? info.original_img_url;
|
||||
const videoUrl = resolveVideoUrl(info);
|
||||
if (videoUrl) {
|
||||
return {
|
||||
kind: "video",
|
||||
url: videoUrl,
|
||||
posterUrl,
|
||||
};
|
||||
}
|
||||
|
||||
const imageUrl = info.original_img_url ?? info.preview_image?.original_img_url;
|
||||
if (imageUrl) {
|
||||
return {
|
||||
kind: "image",
|
||||
url: imageUrl,
|
||||
};
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function resolveFallbackMediaAsset(rawUrl?: string): ResolvedMediaAsset | undefined {
|
||||
if (!rawUrl) return undefined;
|
||||
|
||||
if (/^https:\/\/video\.twimg\.com\//i.test(rawUrl) || /\.(mp4|m4v|mov|webm)(?:$|[?#])/i.test(rawUrl)) {
|
||||
return {
|
||||
kind: "video",
|
||||
url: rawUrl,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
kind: "image",
|
||||
url: rawUrl,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveCoverUrl(info?: ArticleMediaInfo): string | undefined {
|
||||
if (!info) return undefined;
|
||||
return info.original_img_url ?? info.preview_image?.original_img_url;
|
||||
}
|
||||
|
||||
function buildMediaIdentity(asset: ResolvedMediaAsset): string {
|
||||
return asset.kind === "video"
|
||||
? `video:${asset.url}:${asset.posterUrl ?? ""}`
|
||||
: `image:${asset.url}`;
|
||||
}
|
||||
|
||||
function renderMediaLines(
|
||||
asset: ResolvedMediaAsset,
|
||||
altText: string,
|
||||
usedUrls: Set<string>
|
||||
): string[] {
|
||||
if (asset.kind === "video") {
|
||||
const lines: string[] = [];
|
||||
if (asset.posterUrl && !usedUrls.has(asset.posterUrl)) {
|
||||
usedUrls.add(asset.posterUrl);
|
||||
lines.push(``);
|
||||
}
|
||||
if (!usedUrls.has(asset.url)) {
|
||||
usedUrls.add(asset.url);
|
||||
lines.push(`[video](${asset.url})`);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
if (usedUrls.has(asset.url)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
usedUrls.add(asset.url);
|
||||
return [``];
|
||||
}
|
||||
|
||||
function buildMediaById(article: ArticleEntity): Map<string, ResolvedMediaAsset> {
|
||||
const map = new Map<string, ResolvedMediaAsset>();
|
||||
for (const entity of article.media_entities ?? []) {
|
||||
if (!entity?.media_id) continue;
|
||||
const asset = resolveMediaAsset(entity.media_info);
|
||||
if (asset) {
|
||||
map.set(entity.media_id, asset);
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
function collectMediaAssets(article: ArticleEntity): ResolvedMediaAsset[] {
|
||||
const assets: ResolvedMediaAsset[] = [];
|
||||
const seen = new Set<string>();
|
||||
const addAsset = (asset?: ResolvedMediaAsset) => {
|
||||
if (!asset) return;
|
||||
const identity = buildMediaIdentity(asset);
|
||||
if (seen.has(identity)) return;
|
||||
seen.add(identity);
|
||||
assets.push(asset);
|
||||
};
|
||||
|
||||
for (const entity of article.media_entities ?? []) {
|
||||
addAsset(resolveMediaAsset(entity?.media_info));
|
||||
}
|
||||
|
||||
return assets;
|
||||
}
|
||||
|
||||
function resolveEntityMediaLines(
|
||||
entityKey: number | undefined,
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
entityLookup: EntityLookup,
|
||||
mediaById: Map<string, ResolvedMediaAsset>,
|
||||
usedUrls: Set<string>
|
||||
): string[] {
|
||||
if (entityKey === undefined) return [];
|
||||
const entry = resolveEntityEntry(entityKey, entityMap, entityLookup);
|
||||
const value = entry?.value;
|
||||
if (!value) return [];
|
||||
const type = value.type;
|
||||
if (type !== "MEDIA" && type !== "IMAGE") return [];
|
||||
|
||||
const caption = normalizeCaption(value.data?.caption);
|
||||
const altText = caption ? escapeMarkdownAlt(caption) : "";
|
||||
const lines: string[] = [];
|
||||
|
||||
const mediaItems = value.data?.mediaItems ?? [];
|
||||
for (const item of mediaItems) {
|
||||
const mediaId =
|
||||
typeof item?.mediaId === "string"
|
||||
? item.mediaId
|
||||
: typeof item?.media_id === "string"
|
||||
? item.media_id
|
||||
: undefined;
|
||||
const asset = mediaId ? mediaById.get(mediaId) : undefined;
|
||||
if (asset) {
|
||||
lines.push(...renderMediaLines(asset, altText, usedUrls));
|
||||
}
|
||||
}
|
||||
|
||||
const fallbackUrl = typeof value.data?.url === "string" ? value.data.url : undefined;
|
||||
const fallbackAsset = resolveFallbackMediaAsset(fallbackUrl);
|
||||
if (fallbackAsset) {
|
||||
lines.push(...renderMediaLines(fallbackAsset, altText, usedUrls));
|
||||
}
|
||||
|
||||
return lines;
|
||||
}
|
||||
|
||||
function resolveEntityTweetLines(
|
||||
entityKey: number | undefined,
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
entityLookup: EntityLookup,
|
||||
referencedTweets?: Map<string, ReferencedTweetInfo>
|
||||
): string[] {
|
||||
if (entityKey === undefined) return [];
|
||||
const entry = resolveEntityEntry(entityKey, entityMap, entityLookup);
|
||||
const value = entry?.value;
|
||||
if (!value || value.type !== "TWEET") return [];
|
||||
|
||||
const tweetId = typeof value.data?.tweetId === "string" ? value.data.tweetId : "";
|
||||
if (!tweetId) return [];
|
||||
|
||||
const referenced = referencedTweets?.get(tweetId);
|
||||
const url =
|
||||
referenced?.url ??
|
||||
buildTweetUrl(tweetId, referenced?.authorUsername) ??
|
||||
`https://x.com/i/web/status/${tweetId}`;
|
||||
|
||||
const authorText =
|
||||
referenced?.authorName && referenced?.authorUsername
|
||||
? `${referenced.authorName} (@${referenced.authorUsername})`
|
||||
: referenced?.authorUsername
|
||||
? `@${referenced.authorUsername}`
|
||||
: referenced?.authorName;
|
||||
|
||||
const lines: string[] = [];
|
||||
lines.push(`> 引用推文${authorText ? `：${authorText}` : ""}`);
|
||||
|
||||
const summary = summarizeTweetText(referenced?.text);
|
||||
if (summary) {
|
||||
lines.push(`> ${summary}`);
|
||||
}
|
||||
|
||||
lines.push(`> ${url}`);
|
||||
return lines;
|
||||
}
|
||||
|
||||
function resolveEntityMarkdownLines(
|
||||
entityKey: number | undefined,
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
entityLookup: EntityLookup
|
||||
): string[] {
|
||||
if (entityKey === undefined) return [];
|
||||
const entry = resolveEntityEntry(entityKey, entityMap, entityLookup);
|
||||
const value = entry?.value;
|
||||
if (!value || value.type !== "MARKDOWN") return [];
|
||||
|
||||
const markdown = typeof value.data?.markdown === "string" ? value.data.markdown : "";
|
||||
const normalized = markdown.replace(/\r\n/g, "\n").trimEnd();
|
||||
if (!normalized) return [];
|
||||
return normalized.split("\n");
|
||||
}
|
||||
|
||||
function buildMediaLinkMap(
|
||||
entityMap: ArticleContentState["entityMap"] | undefined
|
||||
): Map<number, string> {
|
||||
const map = new Map<number, string>();
|
||||
if (!entityMap) return map;
|
||||
|
||||
const mediaEntries: { idx: number; key: number }[] = [];
|
||||
const linkEntries: { key: number; url: string }[] = [];
|
||||
|
||||
for (const [idx, entry] of Object.entries(entityMap)) {
|
||||
const value = entry?.value;
|
||||
if (!value) continue;
|
||||
const key = parseInt(entry?.key ?? "", 10);
|
||||
if (isNaN(key)) continue;
|
||||
|
||||
if (value.type === "MEDIA" || value.type === "IMAGE") {
|
||||
mediaEntries.push({ idx: Number(idx), key });
|
||||
} else if (value.type === "LINK" && typeof value.data?.url === "string") {
|
||||
linkEntries.push({ key, url: value.data.url });
|
||||
}
|
||||
}
|
||||
|
||||
if (mediaEntries.length === 0 || linkEntries.length === 0) return map;
|
||||
|
||||
mediaEntries.sort((a, b) => a.key - b.key);
|
||||
linkEntries.sort((a, b) => a.key - b.key);
|
||||
|
||||
const pool = [...linkEntries];
|
||||
for (const media of mediaEntries) {
|
||||
if (pool.length === 0) break;
|
||||
let linkIdx = pool.findIndex((l) => l.key > media.key);
|
||||
if (linkIdx === -1) linkIdx = 0;
|
||||
const link = pool.splice(linkIdx, 1)[0]!;
|
||||
map.set(media.idx, link.url);
|
||||
map.set(media.key, link.url);
|
||||
}
|
||||
|
||||
return map;
|
||||
}
|
||||
|
||||
function renderInlineLinks(
|
||||
text: string,
|
||||
entityRanges: Array<{ key?: number; offset?: number; length?: number }>,
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
entityLookup: EntityLookup,
|
||||
mediaLinkMap: Map<number, string>
|
||||
): string {
|
||||
if (!entityMap || entityRanges.length === 0) return text;
|
||||
|
||||
const valid = entityRanges.filter(
|
||||
(r) =>
|
||||
typeof r.key === "number" &&
|
||||
typeof r.offset === "number" &&
|
||||
typeof r.length === "number" &&
|
||||
r.length > 0
|
||||
);
|
||||
if (valid.length === 0) return text;
|
||||
|
||||
const sorted = [...valid].sort((a, b) => (b.offset ?? 0) - (a.offset ?? 0));
|
||||
|
||||
let result = text;
|
||||
for (const range of sorted) {
|
||||
const offset = range.offset!;
|
||||
const length = range.length!;
|
||||
const key = range.key!;
|
||||
|
||||
const entry = resolveEntityEntry(key, entityMap, entityLookup);
|
||||
const value = entry?.value;
|
||||
if (!value) continue;
|
||||
|
||||
let url: string | undefined;
|
||||
if (value.type === "LINK" && typeof value.data?.url === "string") {
|
||||
url = value.data.url;
|
||||
} else if (value.type === "MEDIA" || value.type === "IMAGE") {
|
||||
url = mediaLinkMap.get(key);
|
||||
}
|
||||
|
||||
if (!url) continue;
|
||||
|
||||
const linkText = result.slice(offset, offset + length);
|
||||
result =
|
||||
result.slice(0, offset) +
|
||||
`[${linkText}](${url})` +
|
||||
result.slice(offset + length);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function renderContentBlocks(
|
||||
blocks: ArticleBlock[],
|
||||
entityMap: ArticleContentState["entityMap"] | undefined,
|
||||
entityLookup: EntityLookup,
|
||||
mediaById: Map<string, ResolvedMediaAsset>,
|
||||
usedUrls: Set<string>,
|
||||
mediaLinkMap: Map<number, string>,
|
||||
referencedTweets?: Map<string, ReferencedTweetInfo>
|
||||
): string[] {
|
||||
const lines: string[] = [];
|
||||
let previousKind: "list" | "quote" | "heading" | "text" | "code" | "media" | null = null;
|
||||
let listKind: "ordered" | "unordered" | null = null;
|
||||
let orderedIndex = 0;
|
||||
let inCodeBlock = false;
|
||||
|
||||
const pushBlock = (
|
||||
blockLines: string[],
|
||||
kind: "list" | "quote" | "heading" | "text" | "media"
|
||||
) => {
|
||||
if (blockLines.length === 0) return;
|
||||
if (
|
||||
lines.length > 0 &&
|
||||
previousKind &&
|
||||
!(previousKind === kind && (kind === "list" || kind === "quote" || kind === "media"))
|
||||
) {
|
||||
lines.push("");
|
||||
}
|
||||
lines.push(...blockLines);
|
||||
previousKind = kind;
|
||||
};
|
||||
|
||||
const collectMediaLines = (block: ArticleBlock): string[] => {
|
||||
const ranges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
|
||||
const mediaLines: string[] = [];
|
||||
for (const range of ranges) {
|
||||
if (typeof range?.key !== "number") continue;
|
||||
mediaLines.push(
|
||||
...resolveEntityMediaLines(range.key, entityMap, entityLookup, mediaById, usedUrls)
|
||||
);
|
||||
}
|
||||
return mediaLines;
|
||||
};
|
||||
|
||||
const collectTweetLines = (block: ArticleBlock): string[] => {
|
||||
const ranges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
|
||||
const tweetLines: string[] = [];
|
||||
for (const range of ranges) {
|
||||
if (typeof range?.key !== "number") continue;
|
||||
tweetLines.push(
|
||||
...resolveEntityTweetLines(range.key, entityMap, entityLookup, referencedTweets)
|
||||
);
|
||||
}
|
||||
return tweetLines;
|
||||
};
|
||||
|
||||
const collectLinkLines = (block: ArticleBlock): string[] => {
|
||||
const ranges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
|
||||
const linkLines: string[] = [];
|
||||
for (const range of ranges) {
|
||||
if (typeof range?.key !== "number") continue;
|
||||
const entry = resolveEntityEntry(range.key, entityMap, entityLookup);
|
||||
const value = entry?.value;
|
||||
if (value?.type !== "LINK") continue;
|
||||
const url = typeof value.data?.url === "string" ? value.data.url : "";
|
||||
if (url) {
|
||||
linkLines.push(url);
|
||||
}
|
||||
}
|
||||
return [...new Set(linkLines)];
|
||||
};
|
||||
|
||||
const collectMarkdownLines = (block: ArticleBlock): string[] => {
|
||||
const ranges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
|
||||
const markdownLines: string[] = [];
|
||||
for (const range of ranges) {
|
||||
if (typeof range?.key !== "number") continue;
|
||||
markdownLines.push(...resolveEntityMarkdownLines(range.key, entityMap, entityLookup));
|
||||
}
|
||||
return markdownLines;
|
||||
};
|
||||
|
||||
const pushTrailingMedia = (mediaLines: string[]) => {
|
||||
if (mediaLines.length > 0) {
|
||||
pushBlock(mediaLines, "media");
|
||||
}
|
||||
};
|
||||
|
||||
for (const block of blocks) {
|
||||
const type = typeof block?.type === "string" ? block.type : "unstyled";
|
||||
const rawText = typeof block?.text === "string" ? block.text : "";
|
||||
const ranges = Array.isArray(block.entityRanges) ? block.entityRanges : [];
|
||||
const text =
|
||||
type !== "atomic" && type !== "code-block"
|
||||
? renderInlineLinks(rawText, ranges, entityMap, entityLookup, mediaLinkMap)
|
||||
: rawText;
|
||||
|
||||
if (type === "code-block") {
|
||||
if (!inCodeBlock) {
|
||||
if (lines.length > 0) {
|
||||
lines.push("");
|
||||
}
|
||||
lines.push("```");
|
||||
inCodeBlock = true;
|
||||
}
|
||||
lines.push(text);
|
||||
previousKind = "code";
|
||||
listKind = null;
|
||||
orderedIndex = 0;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (type === "atomic") {
|
||||
if (inCodeBlock) {
|
||||
lines.push("```");
|
||||
inCodeBlock = false;
|
||||
previousKind = "code";
|
||||
}
|
||||
listKind = null;
|
||||
orderedIndex = 0;
|
||||
|
||||
const tweetLines = collectTweetLines(block);
|
||||
if (tweetLines.length > 0) {
|
||||
pushBlock(tweetLines, "quote");
|
||||
}
|
||||
|
||||
const markdownLines = collectMarkdownLines(block);
|
||||
if (markdownLines.length > 0) {
|
||||
pushBlock(markdownLines, "text");
|
||||
}
|
||||
|
||||
const mediaLines = collectMediaLines(block);
|
||||
if (mediaLines.length > 0) {
|
||||
pushBlock(mediaLines, "media");
|
||||
}
|
||||
|
||||
const linkLines = collectLinkLines(block);
|
||||
if (linkLines.length > 0) {
|
||||
pushBlock(linkLines, "text");
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (inCodeBlock) {
|
||||
lines.push("```");
|
||||
inCodeBlock = false;
|
||||
previousKind = "code";
|
||||
}
|
||||
|
||||
if (type === "unordered-list-item") {
|
||||
listKind = "unordered";
|
||||
orderedIndex = 0;
|
||||
pushBlock([`- ${text}`], "list");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
continue;
|
||||
}
|
||||
|
||||
if (type === "ordered-list-item") {
|
||||
if (listKind !== "ordered") {
|
||||
orderedIndex = 0;
|
||||
}
|
||||
listKind = "ordered";
|
||||
orderedIndex += 1;
|
||||
pushBlock([`${orderedIndex}. ${text}`], "list");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
continue;
|
||||
}
|
||||
|
||||
listKind = null;
|
||||
orderedIndex = 0;
|
||||
|
||||
switch (type) {
|
||||
case "header-one":
|
||||
pushBlock([`# ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "header-two":
|
||||
pushBlock([`## ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "header-three":
|
||||
pushBlock([`### ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "header-four":
|
||||
pushBlock([`#### ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "header-five":
|
||||
pushBlock([`##### ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "header-six":
|
||||
pushBlock([`###### ${text}`], "heading");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
case "blockquote": {
|
||||
const quoteLines = text.length > 0 ? text.split("\n") : [""];
|
||||
pushBlock(quoteLines.map((line) => `> ${line}`), "quote");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
}
|
||||
default:
|
||||
if (/^XIMGPH_\d+$/.test(text.trim())) {
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
}
|
||||
pushBlock([text], "text");
|
||||
pushTrailingMedia(collectMediaLines(block));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (inCodeBlock) {
|
||||
lines.push("```");
|
||||
}
|
||||
|
||||
return lines;
|
||||
}
|
||||
|
||||
export type FormatArticleResult = {
|
||||
markdown: string;
|
||||
coverUrl: string | null;
|
||||
};
|
||||
|
||||
export function extractReferencedTweetIds(article: unknown): string[] {
|
||||
const candidate = coerceArticleEntity(article);
|
||||
const entityMap = candidate?.content_state?.entityMap;
|
||||
if (!entityMap) return [];
|
||||
|
||||
const ids: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const entry of Object.values(entityMap)) {
|
||||
const value = entry?.value;
|
||||
if (value?.type !== "TWEET") continue;
|
||||
const tweetId = typeof value.data?.tweetId === "string" ? value.data.tweetId : "";
|
||||
if (!tweetId || seen.has(tweetId)) continue;
|
||||
seen.add(tweetId);
|
||||
ids.push(tweetId);
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
export function formatArticleMarkdown(
|
||||
article: unknown,
|
||||
options: FormatArticleOptions = {}
|
||||
): FormatArticleResult {
|
||||
const candidate = coerceArticleEntity(article);
|
||||
if (!candidate) {
|
||||
return { markdown: `\`\`\`json\n${JSON.stringify(article, null, 2)}\n\`\`\``, coverUrl: null };
|
||||
}
|
||||
|
||||
const lines: string[] = [];
|
||||
const usedUrls = new Set<string>();
|
||||
const mediaById = buildMediaById(candidate);
|
||||
const title = typeof candidate.title === "string" ? candidate.title.trim() : "";
|
||||
if (title) {
|
||||
lines.push(`# ${title}`);
|
||||
}
|
||||
|
||||
const coverUrl = resolveCoverUrl(candidate.cover_media?.media_info) ?? null;
|
||||
if (coverUrl) {
|
||||
usedUrls.add(coverUrl);
|
||||
}
|
||||
|
||||
const blocks = candidate.content_state?.blocks;
|
||||
const entityMap = candidate.content_state?.entityMap;
|
||||
const entityLookup = buildEntityLookup(entityMap);
|
||||
if (Array.isArray(blocks) && blocks.length > 0) {
|
||||
const mediaLinkMap = buildMediaLinkMap(entityMap);
|
||||
const rendered = renderContentBlocks(
|
||||
blocks,
|
||||
entityMap,
|
||||
entityLookup,
|
||||
mediaById,
|
||||
usedUrls,
|
||||
mediaLinkMap,
|
||||
options.referencedTweets
|
||||
);
|
||||
if (rendered.length > 0) {
|
||||
if (lines.length > 0) lines.push("");
|
||||
lines.push(...rendered);
|
||||
}
|
||||
} else if (typeof candidate.plain_text === "string") {
|
||||
if (lines.length > 0) lines.push("");
|
||||
lines.push(candidate.plain_text.trim());
|
||||
} else if (typeof candidate.preview_text === "string") {
|
||||
if (lines.length > 0) lines.push("");
|
||||
lines.push(candidate.preview_text.trim());
|
||||
}
|
||||
|
||||
const trailingMediaLines: string[] = [];
|
||||
for (const asset of collectMediaAssets(candidate)) {
|
||||
trailingMediaLines.push(...renderMediaLines(asset, "", usedUrls));
|
||||
}
|
||||
if (trailingMediaLines.length > 0) {
|
||||
lines.push("", "## Media", "");
|
||||
lines.push(...trailingMediaLines);
|
||||
}
|
||||
|
||||
return { markdown: lines.join("\n").trimEnd(), coverUrl };
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
import path from "node:path";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
|
||||
type MediaKind = "image" | "video";
|
||||
type MediaHint = "image" | "unknown";
|
||||
|
||||
type MarkdownLinkCandidate = {
|
||||
url: string;
|
||||
hint: MediaHint;
|
||||
};
|
||||
|
||||
export type LocalizeMarkdownMediaOptions = {
|
||||
markdownPath: string;
|
||||
log?: (message: string) => void;
|
||||
};
|
||||
|
||||
export type LocalizeMarkdownMediaResult = {
|
||||
markdown: string;
|
||||
downloadedImages: number;
|
||||
downloadedVideos: number;
|
||||
imageDir: string | null;
|
||||
videoDir: string | null;
|
||||
};
|
||||
|
||||
const MARKDOWN_LINK_RE = /(!?\[[^\]\n]*\])\((<)?(https?:\/\/[^)\s>]+)(>)?\)/g;
|
||||
|
||||
const IMAGE_EXTENSIONS = new Set([
|
||||
"jpg",
|
||||
"jpeg",
|
||||
"png",
|
||||
"webp",
|
||||
"gif",
|
||||
"bmp",
|
||||
"avif",
|
||||
"heic",
|
||||
"heif",
|
||||
"svg",
|
||||
]);
|
||||
|
||||
const VIDEO_EXTENSIONS = new Set(["mp4", "m4v", "mov", "webm", "mkv"]);
|
||||
|
||||
const MIME_EXTENSION_MAP: Record<string, string> = {
|
||||
"image/jpeg": "jpg",
|
||||
"image/jpg": "jpg",
|
||||
"image/png": "png",
|
||||
"image/webp": "webp",
|
||||
"image/gif": "gif",
|
||||
"image/bmp": "bmp",
|
||||
"image/avif": "avif",
|
||||
"image/heic": "heic",
|
||||
"image/heif": "heif",
|
||||
"image/svg+xml": "svg",
|
||||
"video/mp4": "mp4",
|
||||
"video/webm": "webm",
|
||||
"video/quicktime": "mov",
|
||||
"video/x-m4v": "m4v",
|
||||
};
|
||||
|
||||
const DOWNLOAD_USER_AGENT =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36";
|
||||
|
||||
function normalizeContentType(raw: string | null): string {
|
||||
return raw?.split(";")[0]?.trim().toLowerCase() ?? "";
|
||||
}
|
||||
|
||||
function normalizeExtension(raw: string | undefined | null): string | undefined {
|
||||
if (!raw) return undefined;
|
||||
const trimmed = raw.replace(/^\./, "").trim().toLowerCase();
|
||||
if (!trimmed) return undefined;
|
||||
if (trimmed === "jpeg") return "jpg";
|
||||
if (trimmed === "jpg") return "jpg";
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
function resolveExtensionFromUrl(rawUrl: string): string | undefined {
|
||||
try {
|
||||
const parsed = new URL(rawUrl);
|
||||
const extFromPath = normalizeExtension(path.posix.extname(parsed.pathname));
|
||||
if (extFromPath) return extFromPath;
|
||||
const extFromFormat = normalizeExtension(parsed.searchParams.get("format"));
|
||||
if (extFromFormat) return extFromFormat;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function resolveKindFromContentType(contentType: string): MediaKind | undefined {
|
||||
if (!contentType) return undefined;
|
||||
if (contentType.startsWith("image/")) return "image";
|
||||
if (contentType.startsWith("video/")) return "video";
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function resolveKindFromExtension(ext: string | undefined): MediaKind | undefined {
|
||||
if (!ext) return undefined;
|
||||
if (IMAGE_EXTENSIONS.has(ext)) return "image";
|
||||
if (VIDEO_EXTENSIONS.has(ext)) return "video";
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function resolveKindFromHostname(rawUrl: string): MediaKind | undefined {
|
||||
try {
|
||||
const hostname = new URL(rawUrl).hostname.toLowerCase();
|
||||
if (hostname.includes("video.twimg.com")) return "video";
|
||||
if (hostname.includes("pbs.twimg.com")) return "image";
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function resolveMediaKind(
|
||||
rawUrl: string,
|
||||
contentType: string,
|
||||
extension: string | undefined,
|
||||
hint: MediaHint
|
||||
): MediaKind | undefined {
|
||||
const kindFromType = resolveKindFromContentType(contentType);
|
||||
if (kindFromType) return kindFromType;
|
||||
|
||||
const kindFromExtension = resolveKindFromExtension(extension);
|
||||
if (kindFromExtension) return kindFromExtension;
|
||||
|
||||
const kindFromHost = resolveKindFromHostname(rawUrl);
|
||||
if (kindFromHost) return kindFromHost;
|
||||
|
||||
if (contentType && contentType !== "application/octet-stream") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return hint === "image" ? "image" : undefined;
|
||||
}
|
||||
|
||||
function resolveOutputExtension(
|
||||
contentType: string,
|
||||
extension: string | undefined,
|
||||
kind: MediaKind
|
||||
): string {
|
||||
const extFromMime = normalizeExtension(MIME_EXTENSION_MAP[contentType]);
|
||||
if (extFromMime) return extFromMime;
|
||||
|
||||
const normalizedExt = normalizeExtension(extension);
|
||||
if (normalizedExt) return normalizedExt;
|
||||
|
||||
return kind === "video" ? "mp4" : "jpg";
|
||||
}
|
||||
|
||||
function safeDecodeURIComponent(value: string): string {
|
||||
try {
|
||||
return decodeURIComponent(value);
|
||||
} catch {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
function sanitizeFileSegment(input: string): string {
|
||||
return input
|
||||
.replace(/[^a-zA-Z0-9_-]+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^[-_]+|[-_]+$/g, "")
|
||||
.slice(0, 48);
|
||||
}
|
||||
|
||||
function resolveFileStem(rawUrl: string, extension: string): string {
|
||||
try {
|
||||
const parsed = new URL(rawUrl);
|
||||
const base = path.posix.basename(parsed.pathname);
|
||||
if (!base) return "";
|
||||
const decodedBase = safeDecodeURIComponent(base);
|
||||
const normalizedExt = normalizeExtension(extension);
|
||||
const stripExt = normalizedExt ? new RegExp(`\\.${normalizedExt}$`, "i") : null;
|
||||
const rawStem = stripExt ? decodedBase.replace(stripExt, "") : decodedBase;
|
||||
return sanitizeFileSegment(rawStem);
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
function buildFileName(kind: MediaKind, index: number, sourceUrl: string, extension: string): string {
|
||||
const stem = resolveFileStem(sourceUrl, extension);
|
||||
const prefix = kind === "image" ? "img" : "video";
|
||||
const serial = String(index).padStart(3, "0");
|
||||
const suffix = stem ? `-${stem}` : "";
|
||||
return `${prefix}-${serial}${suffix}.${extension}`;
|
||||
}
|
||||
|
||||
const FRONTMATTER_COVER_RE = /^(coverImage:\s*")(https?:\/\/[^"]+)(")/m;
|
||||
|
||||
function toHighResUrl(rawUrl: string): string {
|
||||
try {
|
||||
const parsed = new URL(rawUrl);
|
||||
if (parsed.hostname !== "pbs.twimg.com") return rawUrl;
|
||||
const ext = path.posix.extname(parsed.pathname).replace(/^\./, "").toLowerCase();
|
||||
if (!ext || !IMAGE_EXTENSIONS.has(ext)) return rawUrl;
|
||||
parsed.pathname = parsed.pathname.replace(new RegExp(`\\.${ext}$`), "");
|
||||
parsed.searchParams.set("format", ext === "jpeg" ? "jpg" : ext);
|
||||
parsed.searchParams.set("name", "4096x4096");
|
||||
return parsed.toString();
|
||||
} catch {
|
||||
return rawUrl;
|
||||
}
|
||||
}
|
||||
|
||||
function isPlausibleMediaUrl(rawUrl: string): boolean {
|
||||
const ext = resolveExtensionFromUrl(rawUrl);
|
||||
if (ext && (IMAGE_EXTENSIONS.has(ext) || VIDEO_EXTENSIONS.has(ext))) return true;
|
||||
if (resolveKindFromHostname(rawUrl) !== undefined) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
function collectMarkdownLinkCandidates(markdown: string): MarkdownLinkCandidate[] {
|
||||
const candidates: MarkdownLinkCandidate[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
const fmMatch = markdown.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (fmMatch) {
|
||||
const coverMatch = fmMatch[1]?.match(FRONTMATTER_COVER_RE);
|
||||
if (coverMatch?.[2] && !seen.has(coverMatch[2])) {
|
||||
seen.add(coverMatch[2]);
|
||||
candidates.push({ url: coverMatch[2], hint: "image" });
|
||||
}
|
||||
}
|
||||
|
||||
MARKDOWN_LINK_RE.lastIndex = 0;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = MARKDOWN_LINK_RE.exec(markdown))) {
|
||||
const label = match[1] ?? "";
|
||||
const rawUrl = match[3] ?? "";
|
||||
if (!rawUrl || seen.has(rawUrl)) continue;
|
||||
const isImage = label.startsWith("![");
|
||||
if (!isImage && !isPlausibleMediaUrl(rawUrl)) continue;
|
||||
seen.add(rawUrl);
|
||||
candidates.push({
|
||||
url: rawUrl,
|
||||
hint: isImage ? "image" : "unknown",
|
||||
});
|
||||
}
|
||||
|
||||
return candidates;
|
||||
}
|
||||
|
||||
function rewriteMarkdownMediaLinks(markdown: string, replacements: Map<string, string>): string {
|
||||
if (replacements.size === 0) return markdown;
|
||||
MARKDOWN_LINK_RE.lastIndex = 0;
|
||||
|
||||
let result = markdown.replace(MARKDOWN_LINK_RE, (full, label, _openAngle, rawUrl) => {
|
||||
const localPath = replacements.get(rawUrl);
|
||||
if (!localPath) return full;
|
||||
return `${label}(${localPath})`;
|
||||
});
|
||||
|
||||
result = result.replace(FRONTMATTER_COVER_RE, (full, prefix, rawUrl, suffix) => {
|
||||
const localPath = replacements.get(rawUrl);
|
||||
if (!localPath) return full;
|
||||
return `${prefix}${localPath}${suffix}`;
|
||||
});
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function localizeMarkdownMedia(
|
||||
markdown: string,
|
||||
options: LocalizeMarkdownMediaOptions
|
||||
): Promise<LocalizeMarkdownMediaResult> {
|
||||
const log = options.log ?? (() => {});
|
||||
const markdownDir = path.dirname(options.markdownPath);
|
||||
const candidates = collectMarkdownLinkCandidates(markdown);
|
||||
|
||||
if (candidates.length === 0) {
|
||||
return {
|
||||
markdown,
|
||||
downloadedImages: 0,
|
||||
downloadedVideos: 0,
|
||||
imageDir: null,
|
||||
videoDir: null,
|
||||
};
|
||||
}
|
||||
|
||||
const replacements = new Map<string, string>();
|
||||
let downloadedImages = 0;
|
||||
let downloadedVideos = 0;
|
||||
|
||||
for (const candidate of candidates) {
|
||||
try {
|
||||
const downloadUrl = toHighResUrl(candidate.url);
|
||||
const response = await fetch(downloadUrl, {
|
||||
method: "GET",
|
||||
redirect: "follow",
|
||||
headers: {
|
||||
"user-agent": DOWNLOAD_USER_AGENT,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
log(`[x-to-markdown] Skip media (${response.status}): ${candidate.url}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const sourceUrl = response.url || candidate.url;
|
||||
const contentType = normalizeContentType(response.headers.get("content-type"));
|
||||
const extension = resolveExtensionFromUrl(sourceUrl) ?? resolveExtensionFromUrl(candidate.url);
|
||||
const kind = resolveMediaKind(sourceUrl, contentType, extension, candidate.hint);
|
||||
if (!kind) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const outputExtension = resolveOutputExtension(contentType, extension, kind);
|
||||
const nextIndex = kind === "image" ? downloadedImages + 1 : downloadedVideos + 1;
|
||||
const dirName = kind === "image" ? "imgs" : "videos";
|
||||
const targetDir = path.join(markdownDir, dirName);
|
||||
await mkdir(targetDir, { recursive: true });
|
||||
|
||||
const fileName = buildFileName(kind, nextIndex, sourceUrl, outputExtension);
|
||||
const absolutePath = path.join(targetDir, fileName);
|
||||
const relativePath = path.posix.join(dirName, fileName);
|
||||
const bytes = Buffer.from(await response.arrayBuffer());
|
||||
await writeFile(absolutePath, bytes);
|
||||
replacements.set(candidate.url, relativePath);
|
||||
|
||||
if (kind === "image") {
|
||||
downloadedImages = nextIndex;
|
||||
} else {
|
||||
downloadedVideos = nextIndex;
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error ?? "");
|
||||
log(`[x-to-markdown] Failed to download media ${candidate.url}: ${message}`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
markdown: rewriteMarkdownMediaLinks(markdown, replacements),
|
||||
downloadedImages,
|
||||
downloadedVideos,
|
||||
imageDir: downloadedImages > 0 ? path.join(markdownDir, "imgs") : null,
|
||||
videoDir: downloadedVideos > 0 ? path.join(markdownDir, "videos") : null,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "baoyu-danger-x-to-markdown-scripts",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"baoyu-chrome-cdp": "^0.1.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
import { execSync } from "node:child_process";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
|
||||
const APP_DATA_DIR = "baoyu-skills";
|
||||
const X_TO_MARKDOWN_DATA_DIR = "x-to-markdown";
|
||||
const COOKIE_FILE_NAME = "cookies.json";
|
||||
const PROFILE_DIR_NAME = "chrome-profile";
|
||||
const CONSENT_FILE_NAME = "consent.json";
|
||||
|
||||
export function resolveUserDataRoot(): string {
|
||||
if (process.platform === "win32") {
|
||||
return process.env.APPDATA ?? path.join(os.homedir(), "AppData", "Roaming");
|
||||
}
|
||||
if (process.platform === "darwin") {
|
||||
return path.join(os.homedir(), "Library", "Application Support");
|
||||
}
|
||||
return process.env.XDG_DATA_HOME ?? path.join(os.homedir(), ".local", "share");
|
||||
}
|
||||
|
||||
export function resolveXToMarkdownDataDir(): string {
|
||||
const override = process.env.X_DATA_DIR?.trim();
|
||||
if (override) return path.resolve(override);
|
||||
return path.join(resolveUserDataRoot(), APP_DATA_DIR, X_TO_MARKDOWN_DATA_DIR);
|
||||
}
|
||||
|
||||
export function resolveXToMarkdownCookiePath(): string {
|
||||
const override = process.env.X_COOKIE_PATH?.trim();
|
||||
if (override) return path.resolve(override);
|
||||
return path.join(resolveXToMarkdownDataDir(), COOKIE_FILE_NAME);
|
||||
}
|
||||
|
||||
let _wslHome: string | null | undefined;
|
||||
function getWslWindowsHome(): string | null {
|
||||
if (_wslHome !== undefined) return _wslHome;
|
||||
if (!process.env.WSL_DISTRO_NAME) { _wslHome = null; return null; }
|
||||
try {
|
||||
const raw = execSync('cmd.exe /C "echo %USERPROFILE%"', { encoding: 'utf-8', timeout: 5000 }).trim().replace(/\r/g, '');
|
||||
_wslHome = execSync(`wslpath -u "${raw}"`, { encoding: 'utf-8', timeout: 5000 }).trim() || null;
|
||||
} catch { _wslHome = null; }
|
||||
return _wslHome;
|
||||
}
|
||||
|
||||
export function resolveXToMarkdownChromeProfileDir(): string {
|
||||
const override = process.env.BAOYU_CHROME_PROFILE_DIR?.trim() || process.env.X_CHROME_PROFILE_DIR?.trim();
|
||||
if (override) return path.resolve(override);
|
||||
const wslHome = getWslWindowsHome();
|
||||
if (wslHome) return path.join(wslHome, ".local", "share", APP_DATA_DIR, PROFILE_DIR_NAME);
|
||||
return path.join(resolveUserDataRoot(), APP_DATA_DIR, PROFILE_DIR_NAME);
|
||||
}
|
||||
|
||||
export function resolveXToMarkdownConsentPath(): string {
|
||||
return path.join(resolveXToMarkdownDataDir(), CONSENT_FILE_NAME);
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
import { fetchXTweet } from "./graphql.js";
|
||||
import {
|
||||
extractReferencedTweetIds,
|
||||
type ReferencedTweetInfo,
|
||||
} from "./markdown.js";
|
||||
|
||||
type ResolveReferencedTweetsOptions = {
|
||||
log?: (message: string) => void;
|
||||
};
|
||||
|
||||
function extractReferencedTweetInfo(tweet: any, fallbackTweetId: string): ReferencedTweetInfo {
|
||||
const userCore = tweet?.core?.user_results?.result?.core;
|
||||
const userLegacy = tweet?.core?.user_results?.result?.legacy;
|
||||
|
||||
const authorName =
|
||||
typeof userCore?.name === "string"
|
||||
? userCore.name
|
||||
: typeof userLegacy?.name === "string"
|
||||
? userLegacy.name
|
||||
: undefined;
|
||||
|
||||
const authorUsername =
|
||||
typeof userCore?.screen_name === "string"
|
||||
? userCore.screen_name
|
||||
: typeof userLegacy?.screen_name === "string"
|
||||
? userLegacy.screen_name
|
||||
: undefined;
|
||||
|
||||
const text =
|
||||
tweet?.note_tweet?.note_tweet_results?.result?.text ??
|
||||
tweet?.legacy?.full_text ??
|
||||
tweet?.legacy?.text ??
|
||||
undefined;
|
||||
|
||||
const tweetId =
|
||||
typeof tweet?.rest_id === "string" && tweet.rest_id.length > 0
|
||||
? tweet.rest_id
|
||||
: fallbackTweetId;
|
||||
|
||||
const url = authorUsername
|
||||
? `https://x.com/${authorUsername}/status/${tweetId}`
|
||||
: `https://x.com/i/web/status/${tweetId}`;
|
||||
|
||||
return {
|
||||
id: tweetId,
|
||||
url,
|
||||
authorName,
|
||||
authorUsername,
|
||||
text: typeof text === "string" ? text : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export async function resolveReferencedTweetsFromArticle(
|
||||
article: unknown,
|
||||
cookieMap: Record<string, string>,
|
||||
options: ResolveReferencedTweetsOptions = {}
|
||||
): Promise<Map<string, ReferencedTweetInfo>> {
|
||||
const log = options.log ?? (() => {});
|
||||
const ids = extractReferencedTweetIds(article);
|
||||
const referencedTweets = new Map<string, ReferencedTweetInfo>();
|
||||
|
||||
for (const id of ids) {
|
||||
try {
|
||||
const tweet = await fetchXTweet(id, cookieMap, false);
|
||||
const info = extractReferencedTweetInfo(tweet, id);
|
||||
referencedTweets.set(id, info);
|
||||
} catch (error) {
|
||||
log(
|
||||
`[x-to-markdown] Failed to fetch referenced tweet ${id}: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`
|
||||
);
|
||||
referencedTweets.set(id, {
|
||||
id,
|
||||
url: `https://x.com/i/web/status/${id}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return referencedTweets;
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
type ThreadLike = {
|
||||
requestedId?: string;
|
||||
rootId?: string;
|
||||
tweets?: unknown[];
|
||||
totalTweets?: number;
|
||||
user?: any;
|
||||
};
|
||||
|
||||
type TweetPhoto = {
|
||||
src: string;
|
||||
alt?: string;
|
||||
};
|
||||
|
||||
type TweetVideo = {
|
||||
url: string;
|
||||
poster?: string;
|
||||
alt?: string;
|
||||
type?: string;
|
||||
};
|
||||
|
||||
export type ThreadTweetsMarkdownOptions = {
|
||||
username?: string;
|
||||
headingLevel?: number;
|
||||
startIndex?: number;
|
||||
includeTweetUrls?: boolean;
|
||||
};
|
||||
|
||||
export type ThreadMarkdownOptions = ThreadTweetsMarkdownOptions & {
|
||||
includeHeader?: boolean;
|
||||
title?: string;
|
||||
sourceUrl?: string;
|
||||
};
|
||||
|
||||
function coerceThread(value: unknown): ThreadLike | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const candidate = value as ThreadLike;
|
||||
if (!Array.isArray(candidate.tweets)) return null;
|
||||
return candidate;
|
||||
}
|
||||
|
||||
function escapeMarkdownAlt(text: string): string {
|
||||
return text.replace(/[\[\]]/g, "\\$&");
|
||||
}
|
||||
|
||||
function normalizeAlt(text?: string | null): string {
|
||||
const trimmed = text?.trim();
|
||||
if (!trimmed) return "";
|
||||
return trimmed.replace(/\s+/g, " ");
|
||||
}
|
||||
|
||||
function parseTweetText(tweet: any): string {
|
||||
const noteText = tweet?.note_tweet?.note_tweet_results?.result?.text;
|
||||
const legacyText = tweet?.legacy?.full_text ?? tweet?.legacy?.text ?? "";
|
||||
return (noteText ?? legacyText ?? "").trim();
|
||||
}
|
||||
|
||||
function parsePhotos(tweet: any): TweetPhoto[] {
|
||||
const media = tweet?.legacy?.extended_entities?.media ?? [];
|
||||
return media
|
||||
.reduce((acc: TweetPhoto[], item: any) => {
|
||||
if (item?.type !== "photo") {
|
||||
return acc;
|
||||
}
|
||||
const src = item.media_url_https ?? item.media_url;
|
||||
if (!src) {
|
||||
return acc;
|
||||
}
|
||||
const alt = normalizeAlt(item.ext_alt_text);
|
||||
acc.push({ src, alt });
|
||||
return acc;
|
||||
}, [])
|
||||
.filter((photo) => Boolean(photo.src));
|
||||
}
|
||||
|
||||
function parseVideos(tweet: any): TweetVideo[] {
|
||||
const media = tweet?.legacy?.extended_entities?.media ?? [];
|
||||
return media
|
||||
.reduce((acc: TweetVideo[], item: any) => {
|
||||
if (!item?.type || !["animated_gif", "video"].includes(item.type)) {
|
||||
return acc;
|
||||
}
|
||||
const variants = item?.video_info?.variants ?? [];
|
||||
const sources = variants
|
||||
.map((variant: any) => ({
|
||||
contentType: variant?.content_type,
|
||||
url: variant?.url,
|
||||
bitrate: variant?.bitrate ?? 0,
|
||||
}))
|
||||
.filter((variant: any) => Boolean(variant.url));
|
||||
|
||||
const videoSources = sources.filter((variant: any) =>
|
||||
String(variant.contentType ?? "").includes("video")
|
||||
);
|
||||
const sorted = (videoSources.length > 0 ? videoSources : sources).sort(
|
||||
(a: any, b: any) => (b.bitrate ?? 0) - (a.bitrate ?? 0)
|
||||
);
|
||||
const best = sorted[0];
|
||||
if (!best?.url) {
|
||||
return acc;
|
||||
}
|
||||
const alt = normalizeAlt(item.ext_alt_text);
|
||||
acc.push({
|
||||
url: best.url,
|
||||
poster: item.media_url_https ?? item.media_url ?? undefined,
|
||||
alt,
|
||||
type: item.type,
|
||||
});
|
||||
return acc;
|
||||
}, [])
|
||||
.filter((video) => Boolean(video.url));
|
||||
}
|
||||
|
||||
function unwrapTweetResult(result: any): any {
|
||||
if (!result) return null;
|
||||
if (result.__typename === "TweetWithVisibilityResults" && result.tweet) {
|
||||
return result.tweet;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function resolveTweetId(tweet: any): string | undefined {
|
||||
return tweet?.legacy?.id_str ?? tweet?.rest_id;
|
||||
}
|
||||
|
||||
function buildTweetUrl(username: string | undefined, tweetId: string | undefined): string | null {
|
||||
if (!tweetId) return null;
|
||||
if (username) {
|
||||
return `https://x.com/${username}/status/${tweetId}`;
|
||||
}
|
||||
return `https://x.com/i/web/status/${tweetId}`;
|
||||
}
|
||||
|
||||
function formatTweetMarkdown(
|
||||
tweet: any,
|
||||
index: number,
|
||||
options: ThreadTweetsMarkdownOptions
|
||||
): string[] {
|
||||
const headingLevel = options.headingLevel ?? 2;
|
||||
const includeTweetUrls = options.includeTweetUrls ?? true;
|
||||
const headingPrefix = "#".repeat(Math.min(Math.max(headingLevel, 1), 6));
|
||||
const tweetId = resolveTweetId(tweet);
|
||||
const tweetUrl = includeTweetUrls ? buildTweetUrl(options.username, tweetId) : null;
|
||||
|
||||
const lines: string[] = [];
|
||||
lines.push(`${headingPrefix} ${index}`);
|
||||
if (tweetUrl) {
|
||||
lines.push(tweetUrl);
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
const text = parseTweetText(tweet);
|
||||
const photos = parsePhotos(tweet);
|
||||
const videos = parseVideos(tweet);
|
||||
const quoted = unwrapTweetResult(tweet?.quoted_status_result?.result);
|
||||
|
||||
const bodyLines: string[] = [];
|
||||
if (text) {
|
||||
bodyLines.push(...text.split(/\r?\n/));
|
||||
}
|
||||
|
||||
const quotedLines = formatQuotedTweetMarkdown(quoted);
|
||||
if (quotedLines.length > 0) {
|
||||
if (bodyLines.length > 0) bodyLines.push("");
|
||||
bodyLines.push(...quotedLines);
|
||||
}
|
||||
|
||||
const photoLines = photos.map((photo) => {
|
||||
const alt = photo.alt ? escapeMarkdownAlt(photo.alt) : "";
|
||||
return ``;
|
||||
});
|
||||
if (photoLines.length > 0) {
|
||||
if (bodyLines.length > 0) bodyLines.push("");
|
||||
bodyLines.push(...photoLines);
|
||||
}
|
||||
|
||||
const videoLines: string[] = [];
|
||||
for (const video of videos) {
|
||||
if (video.poster) {
|
||||
const alt = video.alt ? escapeMarkdownAlt(video.alt) : "video";
|
||||
videoLines.push(``);
|
||||
}
|
||||
videoLines.push(`[${video.type ?? "video"}](${video.url})`);
|
||||
}
|
||||
if (videoLines.length > 0) {
|
||||
if (bodyLines.length > 0) bodyLines.push("");
|
||||
bodyLines.push(...videoLines);
|
||||
}
|
||||
|
||||
if (bodyLines.length === 0) {
|
||||
bodyLines.push("_No text or media._");
|
||||
}
|
||||
|
||||
lines.push(...bodyLines);
|
||||
return lines;
|
||||
}
|
||||
|
||||
function formatQuotedTweetMarkdown(quoted: any): string[] {
|
||||
if (!quoted) return [];
|
||||
const quotedUser = quoted?.core?.user_results?.result?.legacy;
|
||||
const quotedUsername = quotedUser?.screen_name;
|
||||
const quotedName = quotedUser?.name;
|
||||
const quotedAuthor =
|
||||
quotedUsername && quotedName
|
||||
? `${quotedName} (@${quotedUsername})`
|
||||
: quotedUsername
|
||||
? `@${quotedUsername}`
|
||||
: quotedName ?? "Unknown";
|
||||
|
||||
const quotedId = resolveTweetId(quoted);
|
||||
const quotedUrl =
|
||||
buildTweetUrl(quotedUsername, quotedId) ??
|
||||
(quotedId ? `https://x.com/i/web/status/${quotedId}` : "unavailable");
|
||||
|
||||
const quotedText = parseTweetText(quoted);
|
||||
const lines: string[] = [];
|
||||
lines.push(`Author: ${quotedAuthor}`);
|
||||
lines.push(`URL: ${quotedUrl}`);
|
||||
if (quotedText) {
|
||||
lines.push("", ...quotedText.split(/\r?\n/));
|
||||
} else {
|
||||
lines.push("", "(no content)");
|
||||
}
|
||||
|
||||
return lines.map((line) => `> ${line}`.trimEnd());
|
||||
}
|
||||
|
||||
export function formatThreadTweetsMarkdown(
|
||||
tweets: unknown[],
|
||||
options: ThreadTweetsMarkdownOptions = {}
|
||||
): string {
|
||||
const lines: string[] = [];
|
||||
const startIndex = options.startIndex ?? 1;
|
||||
if (!Array.isArray(tweets) || tweets.length === 0) {
|
||||
return "";
|
||||
}
|
||||
|
||||
tweets.forEach((tweet, index) => {
|
||||
if (lines.length > 0) {
|
||||
lines.push("");
|
||||
}
|
||||
lines.push(...formatTweetMarkdown(tweet, startIndex + index, options));
|
||||
});
|
||||
|
||||
return lines.join("\n").trimEnd();
|
||||
}
|
||||
|
||||
export function formatThreadMarkdown(
|
||||
thread: unknown,
|
||||
options: ThreadMarkdownOptions = {}
|
||||
): string {
|
||||
const candidate = coerceThread(thread);
|
||||
if (!candidate) {
|
||||
return `\`\`\`json\n${JSON.stringify(thread, null, 2)}\n\`\`\``;
|
||||
}
|
||||
|
||||
const tweets = candidate.tweets ?? [];
|
||||
const firstTweet = tweets[0] as any;
|
||||
const user = candidate.user ?? firstTweet?.core?.user_results?.result?.legacy;
|
||||
const username = user?.screen_name;
|
||||
const name = user?.name;
|
||||
|
||||
const includeHeader = options.includeHeader ?? true;
|
||||
const lines: string[] = [];
|
||||
if (includeHeader) {
|
||||
if (options.title) {
|
||||
lines.push(`# ${options.title}`);
|
||||
} else if (username) {
|
||||
lines.push(`# Thread by @${username}${name ? ` (${name})` : ""}`);
|
||||
} else {
|
||||
lines.push("# Thread");
|
||||
}
|
||||
|
||||
const sourceUrl = options.sourceUrl ?? buildTweetUrl(username, candidate.rootId ?? candidate.requestedId);
|
||||
if (sourceUrl) {
|
||||
lines.push(`Source: ${sourceUrl}`);
|
||||
}
|
||||
if (typeof candidate.totalTweets === "number") {
|
||||
lines.push(`Tweets: ${candidate.totalTweets}`);
|
||||
}
|
||||
}
|
||||
|
||||
const tweetMarkdown = formatThreadTweetsMarkdown(tweets, {
|
||||
...options,
|
||||
username,
|
||||
});
|
||||
|
||||
if (tweetMarkdown) {
|
||||
if (lines.length > 0) {
|
||||
lines.push("");
|
||||
}
|
||||
lines.push(tweetMarkdown);
|
||||
}
|
||||
|
||||
return lines.join("\n").trimEnd();
|
||||
}
|
||||
311
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/thread.ts
Normal file
311
baoyu-skills/skills/baoyu-danger-x-to-markdown/scripts/thread.ts
Normal file
@@ -0,0 +1,311 @@
|
||||
import { fetchTweetDetail } from "./graphql.js";
|
||||
|
||||
type TweetEntry = {
|
||||
tweet: any;
|
||||
user?: any;
|
||||
};
|
||||
|
||||
type ParsedEntries = {
|
||||
entries: TweetEntry[];
|
||||
moreCursor?: string;
|
||||
topCursor?: string;
|
||||
bottomCursor?: string;
|
||||
};
|
||||
|
||||
type ThreadResult = {
|
||||
requestedId: string;
|
||||
rootId: string;
|
||||
tweets: any[];
|
||||
totalTweets: number;
|
||||
user?: any;
|
||||
responses?: unknown[];
|
||||
};
|
||||
|
||||
function unwrapTweetResult(result: any): any {
|
||||
if (!result) return null;
|
||||
if (result.__typename === "TweetWithVisibilityResults" && result.tweet) {
|
||||
return result.tweet;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function extractTweetEntry(itemContent: any): TweetEntry | null {
|
||||
const result = itemContent?.tweet_results?.result;
|
||||
if (!result) return null;
|
||||
const resolved = unwrapTweetResult(result?.tweet ?? result);
|
||||
if (!resolved) return null;
|
||||
const user = resolved?.core?.user_results?.result?.legacy;
|
||||
return { tweet: resolved, user };
|
||||
}
|
||||
|
||||
function parseInstruction(instruction?: any): ParsedEntries {
|
||||
const { entries: entities, moduleItems } = instruction || {};
|
||||
const entries: TweetEntry[] = [];
|
||||
let moreCursor: string | undefined;
|
||||
let topCursor: string | undefined;
|
||||
let bottomCursor: string | undefined;
|
||||
|
||||
const parseItems = (items: any[]) => {
|
||||
items?.forEach((item) => {
|
||||
const itemContent = item?.item?.itemContent ?? item?.itemContent;
|
||||
if (!itemContent) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (
|
||||
itemContent.cursorType &&
|
||||
["ShowMore", "ShowMoreThreads"].includes(itemContent.cursorType) &&
|
||||
itemContent.itemType === "TimelineTimelineCursor"
|
||||
) {
|
||||
moreCursor = itemContent.value;
|
||||
return;
|
||||
}
|
||||
|
||||
const entry = extractTweetEntry(itemContent);
|
||||
if (entry) {
|
||||
entries.push(entry);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
if (moduleItems) {
|
||||
parseItems(moduleItems);
|
||||
}
|
||||
|
||||
for (const entity of entities ?? []) {
|
||||
if (entity?.content?.clientEventInfo?.component === "you_might_also_like") {
|
||||
continue;
|
||||
}
|
||||
|
||||
const { itemContent, items, cursorType, entryType, value } = entity?.content ?? {};
|
||||
if (cursorType === "Bottom" && entryType === "TimelineTimelineCursor") {
|
||||
bottomCursor = value;
|
||||
}
|
||||
|
||||
if (
|
||||
itemContent?.cursorType === "Bottom" &&
|
||||
itemContent?.itemType === "TimelineTimelineCursor"
|
||||
) {
|
||||
bottomCursor = bottomCursor ?? itemContent?.value;
|
||||
}
|
||||
|
||||
if (cursorType === "Top" && entryType === "TimelineTimelineCursor") {
|
||||
topCursor = topCursor ?? value;
|
||||
}
|
||||
|
||||
if (itemContent) {
|
||||
const entry = extractTweetEntry(itemContent);
|
||||
if (entry) {
|
||||
entries.push(entry);
|
||||
}
|
||||
if (
|
||||
itemContent.cursorType &&
|
||||
["ShowMore", "ShowMoreThreads"].includes(itemContent.cursorType) &&
|
||||
itemContent.itemType === "TimelineTimelineCursor"
|
||||
) {
|
||||
moreCursor = moreCursor ?? itemContent.value;
|
||||
}
|
||||
|
||||
if (itemContent.cursorType === "Top" && itemContent.itemType === "TimelineTimelineCursor") {
|
||||
topCursor = topCursor ?? itemContent.value;
|
||||
}
|
||||
}
|
||||
|
||||
if (items) {
|
||||
parseItems(items);
|
||||
}
|
||||
}
|
||||
|
||||
return { entries, moreCursor, topCursor, bottomCursor };
|
||||
}
|
||||
|
||||
function parseTweetsAndToken(response: any): ParsedEntries {
|
||||
const instruction =
|
||||
response?.data?.threaded_conversation_with_injections_v2?.instructions?.find(
|
||||
(ins: any) => ins?.type === "TimelineAddEntries" || ins?.type === "TimelineAddToModule"
|
||||
) ??
|
||||
response?.data?.threaded_conversation_with_injections?.instructions?.find(
|
||||
(ins: any) => ins?.type === "TimelineAddEntries" || ins?.type === "TimelineAddToModule"
|
||||
);
|
||||
|
||||
return parseInstruction(instruction);
|
||||
}
|
||||
|
||||
function toTimestamp(value: string | undefined): number {
|
||||
if (!value) return 0;
|
||||
const parsed = Date.parse(value);
|
||||
return Number.isNaN(parsed) ? 0 : parsed;
|
||||
}
|
||||
|
||||
export async function fetchTweetThread(
|
||||
tweetId: string,
|
||||
cookieMap: Record<string, string>,
|
||||
includeResponses = false
|
||||
): Promise<ThreadResult | null> {
|
||||
const responses: unknown[] = [];
|
||||
const res = await fetchTweetDetail(tweetId, cookieMap);
|
||||
if (includeResponses) {
|
||||
responses.push(res);
|
||||
}
|
||||
|
||||
let { entries, moreCursor, topCursor, bottomCursor } = parseTweetsAndToken(res);
|
||||
if (!entries.length) {
|
||||
const errorMessage = res?.errors?.[0]?.message;
|
||||
if (errorMessage) {
|
||||
throw new Error(errorMessage);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
let allEntries = entries.slice();
|
||||
const root = allEntries.find((entry) => entry.tweet?.legacy?.id_str === tweetId);
|
||||
if (!root) {
|
||||
throw new Error("Can not fetch the root tweet");
|
||||
}
|
||||
|
||||
let rootEntry = root.tweet.legacy;
|
||||
|
||||
const isSameThread = (entry: TweetEntry) => {
|
||||
const tweet = entry.tweet?.legacy;
|
||||
if (!tweet) return false;
|
||||
return (
|
||||
tweet.user_id_str === rootEntry.user_id_str &&
|
||||
tweet.conversation_id_str === rootEntry.conversation_id_str &&
|
||||
(tweet.id_str === rootEntry.id_str ||
|
||||
tweet.in_reply_to_user_id_str === rootEntry.user_id_str ||
|
||||
tweet.in_reply_to_status_id_str === rootEntry.conversation_id_str ||
|
||||
!tweet.in_reply_to_user_id_str)
|
||||
);
|
||||
};
|
||||
|
||||
const inThread = (items: TweetEntry[]) => items.some(isSameThread);
|
||||
|
||||
let hasThread = inThread(entries);
|
||||
let maxRequestCount = 1000;
|
||||
let topHasThread = true;
|
||||
|
||||
while (topCursor && topHasThread && maxRequestCount > 0) {
|
||||
const newRes = await fetchTweetDetail(tweetId, cookieMap, topCursor);
|
||||
if (includeResponses) {
|
||||
responses.push(newRes);
|
||||
}
|
||||
|
||||
const parsed = parseTweetsAndToken(newRes);
|
||||
topHasThread = inThread(parsed.entries);
|
||||
topCursor = parsed.topCursor;
|
||||
allEntries = parsed.entries.concat(allEntries);
|
||||
maxRequestCount--;
|
||||
}
|
||||
|
||||
async function checkMoreTweets(focalId: string) {
|
||||
while (moreCursor && hasThread && maxRequestCount > 0) {
|
||||
const newRes = await fetchTweetDetail(focalId, cookieMap, moreCursor);
|
||||
if (includeResponses) {
|
||||
responses.push(newRes);
|
||||
}
|
||||
|
||||
const parsed = parseTweetsAndToken(newRes);
|
||||
moreCursor = parsed.moreCursor;
|
||||
bottomCursor = bottomCursor ?? parsed.bottomCursor;
|
||||
|
||||
hasThread = inThread(parsed.entries);
|
||||
allEntries = allEntries.concat(parsed.entries);
|
||||
maxRequestCount--;
|
||||
}
|
||||
|
||||
if (bottomCursor) {
|
||||
const newRes = await fetchTweetDetail(focalId, cookieMap, bottomCursor);
|
||||
if (includeResponses) {
|
||||
responses.push(newRes);
|
||||
}
|
||||
|
||||
const parsed = parseTweetsAndToken(newRes);
|
||||
allEntries = allEntries.concat(parsed.entries);
|
||||
bottomCursor = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
await checkMoreTweets(tweetId);
|
||||
|
||||
const allThreadEntries = allEntries.filter(
|
||||
(entry) => entry.tweet?.legacy?.id_str === tweetId || isSameThread(entry)
|
||||
);
|
||||
const lastEntity = allThreadEntries[allThreadEntries.length - 1];
|
||||
if (lastEntity?.tweet?.legacy?.id_str) {
|
||||
const lastRes = await fetchTweetDetail(lastEntity.tweet.legacy.id_str, cookieMap);
|
||||
if (includeResponses) {
|
||||
responses.push(lastRes);
|
||||
}
|
||||
|
||||
const parsed = parseTweetsAndToken(lastRes);
|
||||
hasThread = inThread(parsed.entries);
|
||||
allEntries = allEntries.concat(parsed.entries);
|
||||
moreCursor = parsed.moreCursor;
|
||||
bottomCursor = parsed.bottomCursor;
|
||||
maxRequestCount--;
|
||||
|
||||
await checkMoreTweets(lastEntity.tweet.legacy.id_str);
|
||||
}
|
||||
|
||||
const distinctEntries: TweetEntry[] = [];
|
||||
const entriesMap = allEntries.reduce((acc, entry) => {
|
||||
const id = entry.tweet?.legacy?.id_str ?? entry.tweet?.rest_id;
|
||||
if (id && !acc.has(id)) {
|
||||
distinctEntries.push(entry);
|
||||
acc.set(id, entry);
|
||||
}
|
||||
return acc;
|
||||
}, new Map<string, TweetEntry>());
|
||||
allEntries = distinctEntries;
|
||||
|
||||
while (rootEntry.in_reply_to_status_id_str) {
|
||||
const parent = entriesMap.get(rootEntry.in_reply_to_status_id_str)?.tweet?.legacy;
|
||||
if (
|
||||
parent &&
|
||||
parent.user_id_str === rootEntry.user_id_str &&
|
||||
parent.conversation_id_str === rootEntry.conversation_id_str &&
|
||||
parent.id_str !== rootEntry.id_str
|
||||
) {
|
||||
rootEntry = parent;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
allEntries = allEntries.sort((a, b) => {
|
||||
const aTime = toTimestamp(a.tweet?.legacy?.created_at);
|
||||
const bTime = toTimestamp(b.tweet?.legacy?.created_at);
|
||||
return aTime - bTime;
|
||||
});
|
||||
|
||||
const rootIndex = allEntries.findIndex(
|
||||
(entry) => entry.tweet?.legacy?.id_str === rootEntry.id_str
|
||||
);
|
||||
if (rootIndex > 0) {
|
||||
allEntries = allEntries.slice(rootIndex);
|
||||
}
|
||||
|
||||
const threadEntries = allEntries.filter(
|
||||
(entry) => entry.tweet?.legacy?.id_str === tweetId || isSameThread(entry)
|
||||
);
|
||||
|
||||
if (!threadEntries.length) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const tweets = threadEntries.map((entry) => entry.tweet);
|
||||
const user = threadEntries[0].user ?? threadEntries[0].tweet?.core?.user_results?.result?.legacy;
|
||||
const result: ThreadResult = {
|
||||
requestedId: tweetId,
|
||||
rootId: rootEntry.id_str ?? tweetId,
|
||||
tweets,
|
||||
totalTweets: tweets.length,
|
||||
user,
|
||||
};
|
||||
|
||||
if (includeResponses) {
|
||||
result.responses = responses;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
import { fetchXArticle } from "./graphql.js";
|
||||
import type { ArticleEntity } from "./types.js";
|
||||
|
||||
function coerceArticleEntity(value: unknown): ArticleEntity | null {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
const candidate = value as ArticleEntity;
|
||||
if (
|
||||
typeof candidate.title === "string" ||
|
||||
typeof candidate.plain_text === "string" ||
|
||||
typeof candidate.preview_text === "string" ||
|
||||
candidate.content_state
|
||||
) {
|
||||
return candidate;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function hasArticleContent(article: ArticleEntity): boolean {
|
||||
const blocks = article.content_state?.blocks;
|
||||
if (Array.isArray(blocks) && blocks.length > 0) {
|
||||
return true;
|
||||
}
|
||||
if (typeof article.plain_text === "string" && article.plain_text.trim()) {
|
||||
return true;
|
||||
}
|
||||
if (typeof article.preview_text === "string" && article.preview_text.trim()) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function parseArticleIdFromUrl(raw: string | undefined): string | null {
|
||||
if (!raw) return null;
|
||||
try {
|
||||
const parsed = new URL(raw);
|
||||
const match = parsed.pathname.match(/\/(?:i\/)?article\/(\d+)/);
|
||||
if (match?.[1]) return match[1];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractArticleIdFromUrls(urls: any[] | undefined): string | null {
|
||||
if (!Array.isArray(urls)) return null;
|
||||
for (const url of urls) {
|
||||
const candidate =
|
||||
url?.expanded_url ?? url?.url ?? (url?.display_url ? `https://${url.display_url}` : undefined);
|
||||
const id = parseArticleIdFromUrl(candidate);
|
||||
if (id) return id;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function extractArticleEntityFromTweet(tweet: any): unknown | null {
|
||||
return (
|
||||
tweet?.article?.article_results?.result ??
|
||||
tweet?.article?.result ??
|
||||
tweet?.legacy?.article?.article_results?.result ??
|
||||
tweet?.legacy?.article?.result ??
|
||||
tweet?.article_results?.result ??
|
||||
null
|
||||
);
|
||||
}
|
||||
|
||||
export function extractArticleIdFromTweet(tweet: any): string | null {
|
||||
const embedded = extractArticleEntityFromTweet(tweet);
|
||||
const embeddedArticle = embedded as { rest_id?: string } | null;
|
||||
if (embeddedArticle?.rest_id) {
|
||||
return embeddedArticle.rest_id;
|
||||
}
|
||||
|
||||
const noteUrls = tweet?.note_tweet?.note_tweet_results?.result?.entity_set?.urls;
|
||||
const legacyUrls = tweet?.legacy?.entities?.urls;
|
||||
return extractArticleIdFromUrls(noteUrls) ?? extractArticleIdFromUrls(legacyUrls);
|
||||
}
|
||||
|
||||
export async function resolveArticleEntityFromTweet(
|
||||
tweet: any,
|
||||
cookieMap: Record<string, string>
|
||||
): Promise<unknown | null> {
|
||||
if (!tweet) return null;
|
||||
const embedded = extractArticleEntityFromTweet(tweet);
|
||||
const embeddedArticle = coerceArticleEntity(embedded);
|
||||
if (embeddedArticle && hasArticleContent(embeddedArticle)) {
|
||||
return embedded;
|
||||
}
|
||||
|
||||
const articleId = extractArticleIdFromTweet(tweet);
|
||||
if (!articleId) {
|
||||
return embedded ?? null;
|
||||
}
|
||||
|
||||
const fetched = await fetchXArticle(articleId, cookieMap, false);
|
||||
return fetched ?? embedded ?? null;
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
|
||||
import * as path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { hasRequiredXCookies, loadXCookies } from "./cookies.js";
|
||||
import { fetchTweetThread } from "./thread.js";
|
||||
import { formatArticleMarkdown } from "./markdown.js";
|
||||
import { resolveReferencedTweetsFromArticle } from "./referenced-tweets.js";
|
||||
import { formatThreadTweetsMarkdown } from "./thread-markdown.js";
|
||||
import { resolveArticleEntityFromTweet } from "./tweet-article.js";
|
||||
|
||||
type TweetToMarkdownOptions = {
|
||||
log?: (message: string) => void;
|
||||
};
|
||||
|
||||
function parseArgs(): { url?: string } {
|
||||
const args = process.argv.slice(2);
|
||||
let url: string | undefined;
|
||||
|
||||
for (const arg of args) {
|
||||
if (!arg.startsWith("-") && !url) {
|
||||
url = arg;
|
||||
}
|
||||
}
|
||||
|
||||
return { url };
|
||||
}
|
||||
|
||||
function normalizeInputUrl(input: string): string {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return "";
|
||||
try {
|
||||
return new URL(trimmed).toString();
|
||||
} catch {
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
|
||||
function formatScriptCommand(fallback: string): string {
|
||||
const raw = process.argv[1];
|
||||
const displayPath = raw
|
||||
? (() => {
|
||||
const relative = path.relative(process.cwd(), raw);
|
||||
return relative && !relative.startsWith("..") ? relative : raw;
|
||||
})()
|
||||
: fallback;
|
||||
const quotedPath = displayPath.includes(" ")
|
||||
? `"${displayPath.replace(/"/g, '\\"')}"`
|
||||
: displayPath;
|
||||
return `npx -y bun ${quotedPath}`;
|
||||
}
|
||||
|
||||
function parseTweetId(input: string): string | null {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return null;
|
||||
if (/^\d+$/.test(trimmed)) return trimmed;
|
||||
|
||||
try {
|
||||
const parsed = new URL(trimmed);
|
||||
const match = parsed.pathname.match(/\/status(?:es)?\/(\d+)/);
|
||||
if (match?.[1]) return match[1];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function buildTweetUrl(username: string | undefined, tweetId: string | undefined): string | null {
|
||||
if (!tweetId) return null;
|
||||
if (username) {
|
||||
return `https://x.com/${username}/status/${tweetId}`;
|
||||
}
|
||||
return `https://x.com/i/web/status/${tweetId}`;
|
||||
}
|
||||
|
||||
function formatMetaMarkdown(meta: Record<string, string | number | null | undefined>): string {
|
||||
const lines = ["---"];
|
||||
for (const [key, value] of Object.entries(meta)) {
|
||||
if (value === undefined || value === null || value === "") continue;
|
||||
if (typeof value === "number") {
|
||||
lines.push(`${key}: ${value}`);
|
||||
} else {
|
||||
lines.push(`${key}: ${JSON.stringify(value)}`);
|
||||
}
|
||||
}
|
||||
lines.push("---");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function extractTweetText(tweet: any): string {
|
||||
const noteText = tweet?.note_tweet?.note_tweet_results?.result?.text;
|
||||
const legacyText = tweet?.legacy?.full_text ?? tweet?.legacy?.text ?? "";
|
||||
return (noteText ?? legacyText ?? "").trim();
|
||||
}
|
||||
|
||||
function isOnlyUrl(text: string): boolean {
|
||||
const trimmed = text.trim();
|
||||
if (!trimmed) return true;
|
||||
return /^https?:\/\/\S+$/.test(trimmed);
|
||||
}
|
||||
|
||||
export async function tweetToMarkdown(
|
||||
inputUrl: string,
|
||||
options: TweetToMarkdownOptions = {}
|
||||
): Promise<string> {
|
||||
const normalizedUrl = normalizeInputUrl(inputUrl);
|
||||
const tweetId = parseTweetId(normalizedUrl);
|
||||
if (!tweetId) {
|
||||
throw new Error("Invalid tweet url. Example: https://x.com/<user>/status/<tweet_id>");
|
||||
}
|
||||
|
||||
const log = options.log ?? (() => {});
|
||||
log("[tweet-to-markdown] Loading cookies...");
|
||||
const cookieMap = await loadXCookies(log);
|
||||
if (!hasRequiredXCookies(cookieMap)) {
|
||||
throw new Error("Missing auth cookies. Provide X_AUTH_TOKEN and X_CT0 or log in via Chrome.");
|
||||
}
|
||||
|
||||
log(`[tweet-to-markdown] Fetching thread for ${tweetId}...`);
|
||||
const thread = await fetchTweetThread(tweetId, cookieMap);
|
||||
if (!thread) {
|
||||
throw new Error("Failed to fetch thread.");
|
||||
}
|
||||
|
||||
const tweets = thread.tweets ?? [];
|
||||
if (tweets.length === 0) {
|
||||
throw new Error("No tweets found in thread.");
|
||||
}
|
||||
|
||||
const firstTweet = tweets[0] as any;
|
||||
const user = thread.user ?? firstTweet?.core?.user_results?.result?.legacy;
|
||||
const username = user?.screen_name;
|
||||
const name = user?.name;
|
||||
const author =
|
||||
username && name ? `${name} (@${username})` : username ? `@${username}` : name ?? null;
|
||||
const authorUrl = username ? `https://x.com/${username}` : undefined;
|
||||
const requestedUrl = normalizedUrl || buildTweetUrl(username, tweetId) || inputUrl.trim();
|
||||
const rootUrl = buildTweetUrl(username, thread.rootId ?? tweetId) ?? requestedUrl;
|
||||
|
||||
const articleEntity = await resolveArticleEntityFromTweet(firstTweet, cookieMap);
|
||||
let coverImage: string | null = null;
|
||||
let remainingTweets = tweets;
|
||||
const parts: string[] = [];
|
||||
|
||||
if (articleEntity) {
|
||||
const referencedTweets = await resolveReferencedTweetsFromArticle(articleEntity, cookieMap, { log });
|
||||
const articleResult = formatArticleMarkdown(articleEntity, { referencedTweets });
|
||||
coverImage = articleResult.coverUrl;
|
||||
const articleMarkdown = articleResult.markdown.trimEnd();
|
||||
if (articleMarkdown) {
|
||||
parts.push(articleMarkdown);
|
||||
const firstTweetText = extractTweetText(firstTweet);
|
||||
if (isOnlyUrl(firstTweetText)) {
|
||||
remainingTweets = tweets.slice(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const meta = formatMetaMarkdown({
|
||||
url: rootUrl,
|
||||
requestedUrl: requestedUrl,
|
||||
author,
|
||||
authorName: name ?? null,
|
||||
authorUsername: username ?? null,
|
||||
authorUrl: authorUrl ?? null,
|
||||
tweetCount: thread.totalTweets ?? tweets.length,
|
||||
coverImage,
|
||||
});
|
||||
|
||||
parts.unshift(meta);
|
||||
|
||||
if (remainingTweets.length > 0) {
|
||||
const hasArticle = parts.length > 1;
|
||||
if (hasArticle) {
|
||||
parts.push("## Thread");
|
||||
}
|
||||
const tweetMarkdown = formatThreadTweetsMarkdown(remainingTweets, {
|
||||
username,
|
||||
headingLevel: hasArticle ? 3 : 2,
|
||||
startIndex: 1,
|
||||
includeTweetUrls: true,
|
||||
});
|
||||
if (tweetMarkdown) {
|
||||
parts.push(tweetMarkdown);
|
||||
}
|
||||
}
|
||||
|
||||
return parts.join("\n\n").trimEnd();
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { url } = parseArgs();
|
||||
if (!url) {
|
||||
console.error("Usage:");
|
||||
console.error(` ${formatScriptCommand("scripts/tweet-to-markdown.ts")} <tweet url>`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const markdown = await tweetToMarkdown(url, { log: console.log });
|
||||
console.log(markdown);
|
||||
}
|
||||
|
||||
const isCliExecution =
|
||||
process.argv[1] && fileURLToPath(import.meta.url) === path.resolve(process.argv[1]);
|
||||
|
||||
if (isCliExecution) {
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : error);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
export type CookieLike = {
|
||||
name?: string;
|
||||
value?: string;
|
||||
domain?: string;
|
||||
path?: string;
|
||||
url?: string;
|
||||
};
|
||||
|
||||
export type ArticleQueryInfo = {
|
||||
queryId: string;
|
||||
featureSwitches: string[];
|
||||
fieldToggles: string[];
|
||||
html: string;
|
||||
};
|
||||
|
||||
export type ArticleEntityRange = {
|
||||
key?: number;
|
||||
offset?: number;
|
||||
length?: number;
|
||||
};
|
||||
|
||||
export type ArticleBlock = {
|
||||
type?: string;
|
||||
text?: string;
|
||||
entityRanges?: ArticleEntityRange[];
|
||||
};
|
||||
|
||||
export type ArticleEntityMapMediaItem = {
|
||||
mediaId?: string;
|
||||
media_id?: string;
|
||||
localMediaId?: string;
|
||||
};
|
||||
|
||||
export type ArticleEntityMapEntry = {
|
||||
key?: string;
|
||||
value?: {
|
||||
type?: string;
|
||||
mutability?: string;
|
||||
data?: {
|
||||
caption?: string;
|
||||
markdown?: string;
|
||||
mediaItems?: ArticleEntityMapMediaItem[];
|
||||
url?: string;
|
||||
tweetId?: string;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
export type ArticleContentState = {
|
||||
blocks?: ArticleBlock[];
|
||||
entityMap?: Record<string, ArticleEntityMapEntry>;
|
||||
};
|
||||
|
||||
export type ArticleMediaInfo = {
|
||||
__typename?: string;
|
||||
original_img_url?: string;
|
||||
preview_image?: {
|
||||
original_img_url?: string;
|
||||
};
|
||||
variants?: Array<{
|
||||
content_type?: string;
|
||||
url?: string;
|
||||
bit_rate?: number;
|
||||
}>;
|
||||
};
|
||||
|
||||
export type ArticleMediaEntity = {
|
||||
media_id?: string;
|
||||
media_info?: ArticleMediaInfo;
|
||||
};
|
||||
|
||||
export type ArticleEntity = {
|
||||
title?: string;
|
||||
plain_text?: string;
|
||||
preview_text?: string;
|
||||
content_state?: ArticleContentState;
|
||||
cover_media?: {
|
||||
media_info?: ArticleMediaInfo;
|
||||
};
|
||||
media_entities?: ArticleMediaEntity[];
|
||||
};
|
||||
Reference in New Issue
Block a user