Add baoyu-skills package
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
# Adapters & Media
|
||||
|
||||
Read when choosing an adapter, handling media, or answering adapter-specific questions.
|
||||
|
||||
## Built-in Adapters
|
||||
|
||||
| Adapter | URLs | Key Features |
|
||||
|---------|------|-------------|
|
||||
| `x` | x.com, twitter.com | Tweets, threads, X Articles, media, login detection |
|
||||
| `youtube` | youtube.com, youtu.be | Transcript/captions, chapters, cover image, metadata |
|
||||
| `hn` | news.ycombinator.com | Threaded comments, story metadata, nested replies |
|
||||
| `generic` | Any URL (fallback) | Defuddle extraction, Readability fallback, auto-scroll, network idle detection |
|
||||
|
||||
Adapter is auto-selected based on URL. Override with `--adapter <name>`.
|
||||
|
||||
### YouTube
|
||||
|
||||
- Extracts transcripts/captions when available
|
||||
- Transcript format: `[MM:SS] Text segment` with chapter headings
|
||||
- Availability depends on YouTube exposing a caption track; videos with captions disabled or restricted playback may produce description-only output
|
||||
- Use `--wait-for force` if the page needs time to finish loading player metadata
|
||||
|
||||
### X/Twitter
|
||||
|
||||
- Extracts single tweets, threads, and X Articles
|
||||
- Auto-detects login state; if logged out and content requires auth, JSON output shows `"status": "needs_interaction"`
|
||||
- Use `--wait-for interaction` for login-protected content
|
||||
|
||||
### Hacker News
|
||||
|
||||
- Parses threaded comments with proper nesting and reply hierarchy
|
||||
- Includes story metadata (title, URL, author, score, comment count)
|
||||
- Shows comment deletion/dead status
|
||||
|
||||
## Media Download Workflow
|
||||
|
||||
Driven by `download_media` in EXTEND.md:
|
||||
|
||||
| Setting | Behavior |
|
||||
|---------|----------|
|
||||
| `1` (always) | Run CLI with `--download-media --output <path>` |
|
||||
| `0` (never) | Run CLI with `--output <path>` (no media download) |
|
||||
| `ask` (default) | Follow the ask-each-time flow below |
|
||||
|
||||
### Ask-Each-Time Flow
|
||||
|
||||
1. Run the CLI **without** `--download-media` with `--output <path>` → markdown saved
|
||||
2. Check the saved markdown for remote media URLs (`https://` in image/video links)
|
||||
3. **If no remote media found** → done, no prompt needed
|
||||
4. **If remote media found** → ask via `AskUserQuestion`:
|
||||
- header: "Media", question: "Download N images/videos to local files?"
|
||||
- "Yes" — Download to local directories
|
||||
- "No" — Keep remote URLs
|
||||
5. If the user confirms → run the CLI **again** with `--download-media --output <same-path>` (overwrites markdown with localized links)
|
||||
|
||||
### Media Layout
|
||||
|
||||
When `--download-media` is enabled:
|
||||
|
||||
- Images → `imgs/` next to the output file (or `--media-dir`)
|
||||
- Videos → `videos/` next to the output file (or `--media-dir`)
|
||||
- Markdown media links are rewritten to local relative paths
|
||||
|
||||
## Output Format
|
||||
|
||||
Markdown to stdout (or file with `--output`).
|
||||
|
||||
JSON output (`--format json`) returns structured data:
|
||||
|
||||
- `adapter` — which adapter handled the URL
|
||||
- `status` — `"ok"` or `"needs_interaction"`
|
||||
- `login` — login state detection (`logged_in`, `logged_out`, `unknown`)
|
||||
- `interaction` — interaction gate details (kind, provider, prompt)
|
||||
- `document` — structured content (url, title, author, publishedAt, content blocks, metadata)
|
||||
- `media` — collected media assets with url, kind, role
|
||||
- `markdown` — converted markdown text
|
||||
- `downloads` — media download results (when `--download-media` used)
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: first-time-setup
|
||||
description: First-time setup flow for baoyu-url-to-markdown preferences
|
||||
---
|
||||
|
||||
# First-Time Setup
|
||||
|
||||
## Overview
|
||||
|
||||
When no EXTEND.md is found, guide user through preference setup.
|
||||
|
||||
**BLOCKING OPERATION**: This setup MUST complete before ANY other workflow steps. Do NOT:
|
||||
- Start converting URLs
|
||||
- Ask about URLs or output paths
|
||||
- Proceed to any conversion
|
||||
|
||||
ONLY ask the questions in this setup flow, save EXTEND.md, then continue.
|
||||
|
||||
## Setup Flow
|
||||
|
||||
```
|
||||
No EXTEND.md found
|
||||
|
|
||||
v
|
||||
+---------------------+
|
||||
| AskUserQuestion |
|
||||
| (all questions) |
|
||||
+---------------------+
|
||||
|
|
||||
v
|
||||
+---------------------+
|
||||
| Create EXTEND.md |
|
||||
+---------------------+
|
||||
|
|
||||
v
|
||||
Continue conversion
|
||||
```
|
||||
|
||||
## Questions
|
||||
|
||||
**Language**: Use user's input language or saved language preference.
|
||||
|
||||
Use AskUserQuestion with ALL questions in ONE call:
|
||||
|
||||
### Question 1: Download Media
|
||||
|
||||
```yaml
|
||||
header: "Media"
|
||||
question: "How to handle images and videos in pages?"
|
||||
options:
|
||||
- label: "Ask each time (Recommended)"
|
||||
description: "After saving markdown, ask whether to download media"
|
||||
- label: "Always download"
|
||||
description: "Always download media to local imgs/ and videos/ directories"
|
||||
- label: "Never download"
|
||||
description: "Keep original remote URLs in markdown"
|
||||
```
|
||||
|
||||
### Question 2: Default Output Directory
|
||||
|
||||
```yaml
|
||||
header: "Output"
|
||||
question: "Default output directory?"
|
||||
options:
|
||||
- label: "url-to-markdown (Recommended)"
|
||||
description: "Save to ./url-to-markdown/{domain}/{slug}.md"
|
||||
```
|
||||
|
||||
Note: User will likely choose "Other" to type a custom path.
|
||||
|
||||
### Question 3: Save Location
|
||||
|
||||
```yaml
|
||||
header: "Save"
|
||||
question: "Where to save preferences?"
|
||||
options:
|
||||
- label: "User (Recommended)"
|
||||
description: "~/.baoyu-skills/ (all projects)"
|
||||
- label: "Project"
|
||||
description: ".baoyu-skills/ (this project only)"
|
||||
```
|
||||
|
||||
## Save Locations
|
||||
|
||||
| Choice | Path | Scope |
|
||||
|--------|------|-------|
|
||||
| User | `~/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | All projects |
|
||||
| Project | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Current project |
|
||||
|
||||
## After Setup
|
||||
|
||||
1. Create directory if needed
|
||||
2. Write EXTEND.md
|
||||
3. Confirm: "Preferences saved to [path]"
|
||||
4. Continue with conversion using saved preferences
|
||||
|
||||
## EXTEND.md Template
|
||||
|
||||
```md
|
||||
download_media: [ask/1/0]
|
||||
default_output_dir: [path or empty]
|
||||
```
|
||||
|
||||
## Modifying Preferences Later
|
||||
|
||||
Users can edit EXTEND.md directly or delete it to trigger setup again.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Quality Gate & Recovery
|
||||
|
||||
Headless Chrome can silently return low-quality content — layout shells, login walls, or framework payloads — without the CLI returning a non-zero exit code. Read this after every headless run so you can catch and recover from those cases.
|
||||
|
||||
## Checks the Agent Must Run
|
||||
|
||||
1. Confirm the markdown title matches the target page, not a generic site shell
|
||||
2. Confirm the body contains the expected article/page content, not just navigation, footer, or a generic error
|
||||
3. Watch for obvious failure signs:
|
||||
- `Application error`
|
||||
- `This page could not be found`
|
||||
- Login, signup, subscribe, or verification shells
|
||||
- Extremely short markdown for a page that should be long-form
|
||||
- Raw framework payloads or mostly boilerplate content
|
||||
4. Do NOT accept a run as successful just because the CLI exited `0`
|
||||
|
||||
**Tip**: run with `--format json` to get structured signals including `status`, `login.state`, and `interaction`. `"status": "needs_interaction"` means the page requires manual interaction.
|
||||
|
||||
## Recovery Workflow
|
||||
|
||||
1. Start headless (default) unless there is already a clear reason to use interaction mode
|
||||
2. Review markdown quality immediately after the run
|
||||
3. If the content is low quality or indicates login/CAPTCHA:
|
||||
- `--wait-for interaction` for auto-detected gates (login, CAPTCHA, Cloudflare)
|
||||
- `--wait-for force` when the page needs manual browsing, scroll loading, or complex interaction
|
||||
4. If `--wait-for` is used, tell the user exactly what to do:
|
||||
- Login required → sign in in the browser
|
||||
- CAPTCHA visible → solve it
|
||||
- Slow loading → wait until content is visible
|
||||
- `--wait-for force` → press Enter when ready
|
||||
5. If JSON output shows `"status": "needs_interaction"`, switch to `--wait-for interaction` automatically
|
||||
|
||||
## Capture Modes
|
||||
|
||||
| Mode | Behavior | Use When |
|
||||
|------|----------|----------|
|
||||
| Default | Headless Chrome, auto-extract on network idle | Public pages, static content |
|
||||
| `--headless` | Explicit headless (same as default) | Clarify intent |
|
||||
| `--wait-for interaction` | Opens visible Chrome, auto-detects login/CAPTCHA gates, waits for them to clear, then continues | Login-required, CAPTCHA-protected |
|
||||
| `--wait-for force` | Opens visible Chrome, auto-detects OR accepts Enter keypress to continue | Complex flows, lazy loading, paywalls |
|
||||
|
||||
**Interaction gate auto-detection**: Cloudflare Turnstile / "just a moment" pages, Google reCAPTCHA, hCaptcha, custom challenge / verification screens.
|
||||
Reference in New Issue
Block a user