sreweekly-digest: 脚本全文抓取升级 + scripts/README
This commit is contained in:
125
sreweekly-digest/scripts/README.md
Normal file
125
sreweekly-digest/scripts/README.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# sreweekly.py
|
||||
|
||||
SRE Weekly 抓取与文章提取脚本。单文件、无外部服务依赖，专为 n8n 定时任务设计：脚本自身**只负责抓取与落盘**，不发送邮件、不推送通知。
|
||||
|
||||
## 功能概览
|
||||
|
||||
- 通过 RSS (`https://sreweekly.com/feed/`) 扫描新一期 SRE Weekly
|
||||
- **回溯抓取历史期刊**（RSS 只暴露最近 ~10 期，`--backfill` 直接走站点 URL 抓任意老期）
|
||||
- 把每期 `content:encoded` 原样保存为 HTML
|
||||
- 解析出每期的所有文章条目（标题 / 链接 / 简介 / 引用 / 作者）
|
||||
- 用 [`trafilatura`](https://github.com/adbar/trafilatura) 抓文章原文并转 Markdown
|
||||
- 用 `manifest.json` 记录每期状态，避免重复处理
|
||||
- 每篇文章生成一个独立 `.md` 文件，含 feed 简介 + 抓取的正文
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
pip3 install feedparser trafilatura
|
||||
```
|
||||
|
||||
`trafilatura` 仅在抓文章正文时才必需；若加 `--skip-fetch` 只用 feed 简介，可省略。
|
||||
|
||||
## 目录结构
|
||||
|
||||
默认工作目录 `~/Workspace/nexus/sreweekly/`（可用 `--dir` 覆盖）：
|
||||
|
||||
```
|
||||
sreweekly/
|
||||
├── manifest.json # 状态记录
|
||||
├── html/<id>-<日期>.html # 每期原始 HTML（feed 版 或 backfill 站点整页）
|
||||
├── pages/<id>.html # backfill 抓到的站点整页缓存
|
||||
├── articles/<id>/ # 每篇文章原始 HTML 缓存 + index.json 抓取日志
|
||||
│ ├── 01-<slug>.html
|
||||
│ └── index.json
|
||||
└── markdown/<id>/ # 每期文章的 markdown 输出
|
||||
├── 01-<slug>.md
|
||||
└── 02-<slug>.md
|
||||
```
|
||||
|
||||
## 用法
|
||||
|
||||
```bash
|
||||
# 扫描 feed，新增期落盘并登记 manifest
|
||||
python3 sreweekly.py
|
||||
|
||||
# 只探测有无新期（供 n8n 分支判断），输出 NEW:533,532 或 NONE
|
||||
python3 sreweekly.py --check
|
||||
|
||||
# 提取指定期文章为 markdown（默认联网抓正文）
|
||||
python3 sreweekly.py --extract 533
|
||||
python3 sreweekly.py --extract latest # 最新一期
|
||||
python3 sreweekly.py --extract pending # 所有未提取的期（n8n 每日推荐）
|
||||
python3 sreweekly.py --extract all # 全部（配合 --force 覆盖）
|
||||
|
||||
# 查看状态
|
||||
python3 sreweekly.py --status
|
||||
python3 sreweekly.py --issues
|
||||
|
||||
# 回溯抓取历史期刊（绕过 RSS 只有最近 ~10 期的限制）
|
||||
python3 sreweekly.py --backfill 430..533 # 抓期号 430 到 533 之间所有未登记的期
|
||||
python3 sreweekly.py --backfill 1..533 # 全站回溯（500+ 期，约 12-15 分钟）
|
||||
python3 sreweekly.py --backfill 450 # 只抓单期
|
||||
# 回溯完再走标准提取流程：
|
||||
python3 sreweekly.py --extract pending
|
||||
```
|
||||
|
||||
## 选项
|
||||
|
||||
| 选项 | 说明 |
|
||||
| --- | --- |
|
||||
| `--feed URL` | RSS 地址，默认 `https://sreweekly.com/feed/` |
|
||||
| `--dir PATH` | 工作目录，默认 `~/Workspace/nexus/sreweekly/` |
|
||||
| `--force` | 重新提取已提取过的期，覆盖旧 markdown |
|
||||
| `--skip-fetch` | 不抓文章正文，只使用 feed 简介 |
|
||||
| `--force-fetch` | 忽略 `articles/` / `pages/` 本地缓存，重新联网抓 HTML |
|
||||
| `--sleep 秒数` | 每次抓取间隔（默认 `1.5`，`--backfill` 也遵守此参数） |
|
||||
|
||||
## 典型 n8n 流程
|
||||
|
||||
1. **Cron 触发** → `python3 sreweekly.py --check`
|
||||
2. 判断输出：`NEW:...` 走后续，`NONE` 结束
|
||||
3. `python3 sreweekly.py`（扫描落盘新期）
|
||||
4. `python3 sreweekly.py --extract pending`（抓正文 + 生成 markdown）
|
||||
5. 下游节点读取 `markdown/<id>/*.md` 汇总成邮件/推送
|
||||
|
||||
## Markdown 输出示例
|
||||
|
||||
每篇文章一个 `.md` 文件：
|
||||
|
||||
```markdown
|
||||
# 文章标题
|
||||
|
||||
- **期号**: SRE Weekly Issue #533（2024-05-20）
|
||||
- **作者**: Jane Doe
|
||||
- **链接**: https://example.com/post
|
||||
|
||||
## 简介
|
||||
（来自 feed 的原始描述与引用）
|
||||
|
||||
## 正文
|
||||
（trafilatura 抽取的原文 markdown；失败时显示错误信息）
|
||||
```
|
||||
|
||||
## Manifest 字段
|
||||
|
||||
`manifest.json` 中每期条目字段：
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `id` / `title` / `url` / `pub_date` | 期号、标题、原文链接、发布日期 |
|
||||
| `html_file` | 落盘 HTML 相对路径 |
|
||||
| `fetched_at` / `extracted_at` | 抓取与提取时间戳 |
|
||||
| `extracted` | 是否已提取为 markdown |
|
||||
| `article_count` | 该期文章数 |
|
||||
| `markdown_dir` | markdown 输出目录（相对） |
|
||||
| `articles_fetched` / `articles_failed` | 正文抓取成功/失败数 |
|
||||
| `source` | 数据来源标记：`backfill` 表示由 `--backfill` 从站点整页抓取（feed 抓取无此字段） |
|
||||
|
||||
## 故障排查
|
||||
|
||||
- **feed 抓取失败或为空**：检查 `--feed` 网络可达性，或 feed 结构变更
|
||||
- **`trafilatura returned empty`**：某些站点反爬/JS 渲染，正文段落会写入错误提示但不影响其它文章
|
||||
- **重跑某期**：`--extract <id> --force`；重新联网抓正文再加 `--force-fetch`
|
||||
- **回溯抓不到某期**：脚本会打印 `⚠️ #<id> 抓取失败：...` 并继续下一期，最后汇总失败期号；对失败期号单独 `--backfill <id> --force-fetch` 重试即可
|
||||
- **回溯抓到的期日期显示当天**：说明页面 `<a class="updated">` 结构变了，`extract_pub_date_from_page` 未命中，会 fallback 到今天——不影响提取，但排序会乱
|
||||
@@ -10,37 +10,58 @@ SRE Weekly 抓取与文章提取脚本
|
||||
|
||||
目录结构（默认工作目录 ~/Workspace/nexus/sreweekly/）:
|
||||
sreweekly/
|
||||
├── manifest.json # 状态记录（每期: html 是否保存、文章是否提取、文章数）
|
||||
├── manifest.json # 状态记录（每期: html 是否保存、文章是否提取、抓取统计）
|
||||
├── html/<id>-<日期>.html # 每期原始 HTML（feed 的 content:encoded 原样保存）
|
||||
└── markdown/<id>/ # 每期文章的 markdown 目录（每篇一个文件）
|
||||
├── 01-<标题slug>.md
|
||||
└── 02-<标题slug>.md
|
||||
├── articles/<id>/ # 每篇文章原始 HTML 缓存 + index.json 抓取日志
|
||||
│ ├── 01-<slug>.html
|
||||
│ └── index.json
|
||||
└── markdown/<id>/ # 每期文章的 markdown 目录（每篇一个文件，含正文）
|
||||
├── 01-<slug>.md
|
||||
└── 02-<slug>.md
|
||||
|
||||
用法:
|
||||
python3 sreweekly.py # 扫描 feed，新增期保存为 HTML 并登记 manifest
|
||||
python3 sreweekly.py --check # 只检查有无新期，输出 NEW:533,532 或 NONE（供 n8n 分支）
|
||||
python3 sreweekly.py --extract ID # 提取指定期（如 533）的文章 → markdown
|
||||
python3 sreweekly.py --extract ID # 提取指定期（如 533）的文章 → markdown（默认抓正文）
|
||||
python3 sreweekly.py --extract latest # 提取最新一期
|
||||
python3 sreweekly.py --extract pending # 提取所有尚未提取的期（n8n 每日流程推荐）
|
||||
python3 sreweekly.py --extract all # 提取全部（含已提取的，需配合 --force 覆盖）
|
||||
python3 sreweekly.py --status # 打印 manifest 状态表
|
||||
python3 sreweekly.py --issues # 列出已登记的所有期
|
||||
python3 sreweekly.py --backfill 400..500 # 回溯抓取指定期号区间（绕过 RSS 只有最近 ~10 期的限制）
|
||||
|
||||
选项:
|
||||
--feed URL RSS 地址（默认 https://sreweekly.com/feed/）
|
||||
--dir PATH 工作目录（默认 ~/Workspace/nexus/sreweekly/）
|
||||
--force 重新提取已提取过的期（覆盖旧文件）
|
||||
--force 重新提取已提取过的期（覆盖旧 markdown）
|
||||
--skip-fetch 不抓文章正文，只用 feed 简介（老行为）
|
||||
--force-fetch 忽略 articles/ 本地缓存，重新联网抓
|
||||
--sleep 秒数 每篇文章抓取之间的间隔（默认 1.5，避免打站过快）
|
||||
|
||||
依赖:
|
||||
pip3 install feedparser trafilatura
|
||||
"""
|
||||
from html.parser import HTMLParser
|
||||
import argparse, html, json, os, re, sys, time, urllib.request
|
||||
import argparse, html, json, os, re, sys, time, urllib.request, urllib.error
|
||||
|
||||
try:
|
||||
import feedparser
|
||||
except ImportError:
|
||||
sys.exit("缺少依赖: 请先执行 pip3 install feedparser")
|
||||
|
||||
try:
|
||||
import trafilatura
|
||||
except ImportError:
|
||||
trafilatura = None # 抓正文时才强制要求
|
||||
|
||||
DEFAULT_FEED = "https://sreweekly.com/feed/"
|
||||
DEFAULT_DIR = os.path.expanduser("~/Workspace/nexus/sreweekly")
|
||||
DEFAULT_UA = (
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36"
|
||||
)
|
||||
FETCH_TIMEOUT = 20
|
||||
NON_HTML_EXTS = (".pdf", ".zip", ".tar", ".gz", ".mp4", ".mp3", ".png", ".jpg", ".jpeg", ".gif", ".svg")
|
||||
|
||||
# ---------------------------------------------------------------- manifest
|
||||
|
||||
@@ -118,6 +139,93 @@ def cmd_scan(args, manifest, base_dir):
|
||||
print(f"📊 扫描完成：新增 {len(new_ids)} 期，共 {len(manifest['issues'])} 期", file=sys.stderr)
|
||||
return new_ids
|
||||
|
||||
# ---------------------------------------------------------------- 回溯抓取（老期刊）
|
||||
|
||||
MONTHS = {m: i for i, m in enumerate(
|
||||
["January","February","March","April","May","June",
|
||||
"July","August","September","October","November","December"], 1)}
|
||||
|
||||
def parse_backfill_range(spec):
|
||||
"""'400..500' → (400, 500)；'450' → (450, 450)。"""
|
||||
m = re.match(r"^\s*(\d+)\s*(?:\.\.\s*(\d+))?\s*$", spec or "")
|
||||
if not m:
|
||||
sys.exit(f"❌ --backfill 参数格式错误（示例 400..500 或 450）: {spec}")
|
||||
lo = int(m.group(1)); hi = int(m.group(2) or m.group(1))
|
||||
if lo > hi: lo, hi = hi, lo
|
||||
return lo, hi
|
||||
|
||||
def extract_pub_date_from_page(html_text):
|
||||
"""从 sreweekly 站点页面 HTML 里提取发布日期 → YYYY-MM-DD，失败返回 None。
|
||||
|
||||
结构: <a class="updated" href="...">June 23, 2024</a>
|
||||
"""
|
||||
if not html_text:
|
||||
return None
|
||||
m = re.search(r'class="updated"[^>]*>\s*([A-Za-z]+)\s+(\d{1,2}),\s*(\d{4})', html_text)
|
||||
if not m:
|
||||
return None
|
||||
mon = MONTHS.get(m.group(1))
|
||||
if not mon:
|
||||
return None
|
||||
return f"{int(m.group(3)):04d}-{mon:02d}-{int(m.group(2)):02d}"
|
||||
|
||||
def extract_title_from_page(html_text, iid):
|
||||
m = re.search(r"<title>\s*(.+?)\s*</title>", html_text or "", re.I | re.S)
|
||||
if not m:
|
||||
return f"SRE Weekly Issue #{iid}"
|
||||
t = html.unescape(m.group(1))
|
||||
# 去掉站点后缀 " – SRE WEEKLY" / " - SRE WEEKLY"
|
||||
t = re.sub(r"\s*[\u2013\u2014\-]\s*SRE WEEKLY\s*$", "", t, flags=re.I).strip()
|
||||
return t or f"SRE Weekly Issue #{iid}"
|
||||
|
||||
def cmd_backfill(args, manifest, base_dir):
|
||||
lo, hi = parse_backfill_range(args.backfill)
|
||||
html_dir = os.path.join(base_dir, "html")
|
||||
page_cache_dir = os.path.join(base_dir, "pages") # 存整页 HTML 作缓存
|
||||
os.makedirs(html_dir, exist_ok=True)
|
||||
added, skipped, missing = [], [], []
|
||||
for n in range(lo, hi + 1):
|
||||
iid = str(n)
|
||||
if iid in manifest["issues"]:
|
||||
skipped.append(iid)
|
||||
continue
|
||||
url = f"https://sreweekly.com/sre-weekly-issue-{n}/"
|
||||
cache_path = os.path.join(page_cache_dir, f"{n}.html")
|
||||
raw, err = fetch_article_html(url, cache_path, force=args.force_fetch)
|
||||
if not raw:
|
||||
print(f"⚠️ #{iid} 抓取失败：{err}（{url}）", file=sys.stderr)
|
||||
missing.append(iid)
|
||||
if args.sleep > 0:
|
||||
time.sleep(args.sleep)
|
||||
continue
|
||||
# 站点页 HTML 含 sreweekly-entry 结构，SREEntryParser 兼容；直接落到 html/ 供后续 --extract 使用
|
||||
pubdate = extract_pub_date_from_page(raw) or time.strftime("%Y-%m-%d")
|
||||
title = extract_title_from_page(raw, iid)
|
||||
fname = f"{iid}-{pubdate}.html"
|
||||
with open(os.path.join(html_dir, fname), "w", encoding="utf-8") as f:
|
||||
f.write(raw)
|
||||
manifest["issues"][iid] = {
|
||||
"id": iid,
|
||||
"title": title,
|
||||
"url": url,
|
||||
"pub_date": pubdate,
|
||||
"html_file": f"html/{fname}",
|
||||
"fetched_at": now_iso(),
|
||||
"source": "backfill",
|
||||
"extracted": False,
|
||||
"article_count": 0,
|
||||
"markdown_dir": None,
|
||||
}
|
||||
added.append(iid)
|
||||
print(f"🆕 回溯 #{iid} {title} → {fname}（{len(raw)} bytes）", file=sys.stderr)
|
||||
if args.sleep > 0 and n < hi:
|
||||
time.sleep(args.sleep)
|
||||
save_manifest(base_dir, manifest)
|
||||
print(f"📊 回溯完成：新增 {len(added)} 期 / 跳过已有 {len(skipped)} / 抓取失败 {len(missing)}", file=sys.stderr)
|
||||
if missing:
|
||||
print(f" 失败期号: {','.join(missing)}", file=sys.stderr)
|
||||
return added
|
||||
|
||||
def cmd_check(args, manifest):
|
||||
entries = fetch_feed(args.feed)
|
||||
known = set(manifest["issues"].keys())
|
||||
@@ -266,8 +374,72 @@ def slugify(title, max_len=70):
|
||||
s = re.sub(r"-{2,}", "-", s)
|
||||
return s[:max_len].strip("-") or "untitled"
|
||||
|
||||
def extract_one(iid, entry, base_dir, force=False):
|
||||
"""把某一期 HTML 解析为每篇文章一个 markdown 文件。返回 (文章数, 输出目录)。"""
|
||||
# ---------------------------------------------------------------- 正文抓取 & 转 markdown
|
||||
|
||||
def _looks_like_binary(url):
|
||||
u = (url or "").lower().split("?", 1)[0]
|
||||
return u.endswith(NON_HTML_EXTS)
|
||||
|
||||
def fetch_article_html(url, cache_path, force=False):
|
||||
"""抓文章原始 HTML，落到 cache_path。返回 (html_text, error)。"""
|
||||
if not url:
|
||||
return None, "empty url"
|
||||
if _looks_like_binary(url):
|
||||
return None, "non-html resource"
|
||||
if not force and os.path.exists(cache_path) and os.path.getsize(cache_path) > 0:
|
||||
with open(cache_path, "r", encoding="utf-8", errors="replace") as f:
|
||||
return f.read(), None
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": DEFAULT_UA,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=FETCH_TIMEOUT) as r:
|
||||
ctype = r.headers.get("Content-Type", "")
|
||||
if "html" not in ctype.lower() and "xml" not in ctype.lower():
|
||||
return None, f"non-html content-type: {ctype}"
|
||||
raw = r.read()
|
||||
charset = None
|
||||
m = re.search(r"charset=([\w-]+)", ctype, re.I)
|
||||
if m:
|
||||
charset = m.group(1)
|
||||
text = raw.decode(charset or "utf-8", errors="replace")
|
||||
except urllib.error.HTTPError as e:
|
||||
return None, f"HTTP {e.code}"
|
||||
except urllib.error.URLError as e:
|
||||
return None, f"URLError: {e.reason}"
|
||||
except (TimeoutError, OSError) as e:
|
||||
return None, f"{type(e).__name__}: {e}"
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
with open(cache_path, "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
return text, None
|
||||
|
||||
def html_to_markdown(html_text, url):
|
||||
"""用 trafilatura 抽正文并输出 markdown。失败返回 None。"""
|
||||
if not html_text or trafilatura is None:
|
||||
return None
|
||||
try:
|
||||
md = trafilatura.extract(
|
||||
html_text,
|
||||
output_format="markdown",
|
||||
url=url,
|
||||
include_links=True,
|
||||
include_images=True,
|
||||
include_tables=True,
|
||||
favor_recall=True,
|
||||
with_metadata=False,
|
||||
)
|
||||
if md:
|
||||
md = md.strip()
|
||||
return md or None
|
||||
except Exception as e:
|
||||
print(f"⚠️ trafilatura 抽取失败 {url}: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
def extract_one(iid, entry, base_dir, force=False, fetch=True, force_fetch=False, sleep=1.5):
|
||||
"""把某一期 HTML 解析为每篇文章一个 markdown 文件。返回 (文章数, 输出目录, changed)。"""
|
||||
md_root = os.path.join(base_dir, "markdown")
|
||||
os.makedirs(md_root, exist_ok=True)
|
||||
html_rel = entry.get("html_file")
|
||||
@@ -276,27 +448,57 @@ def extract_one(iid, entry, base_dir, force=False):
|
||||
html_path = os.path.join(base_dir, html_rel)
|
||||
if not os.path.exists(html_path):
|
||||
sys.exit(f"❌ HTML 文件不存在: {html_path}")
|
||||
if entry.get("extracted") and entry.get("article_count", 0) > 0 and not force:
|
||||
already_done = entry.get("extracted") and entry.get("article_count", 0) > 0
|
||||
if already_done and not force:
|
||||
print(f"⏭️ 期 #{iid} 已提取过（{entry['article_count']} 篇），跳过（--force 可覆盖）", file=sys.stderr)
|
||||
return entry.get("article_count", 0), entry.get("markdown_dir"), False
|
||||
|
||||
articles = parse_issue_html(html_path)
|
||||
out_dir = os.path.join(md_root, str(iid))
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
art_cache_dir = os.path.join(base_dir, "articles", str(iid))
|
||||
|
||||
fetched, failed = 0, 0
|
||||
fetch_log = []
|
||||
for idx, a in enumerate(articles, 1):
|
||||
fname = f"{idx:02d}-{slugify(a['title'])}.md"
|
||||
slug = slugify(a["title"])
|
||||
body_md, err = None, None
|
||||
if fetch and a.get("url"):
|
||||
cache_path = os.path.join(art_cache_dir, f"{idx:02d}-{slug}.html")
|
||||
need_net = force_fetch or not (os.path.exists(cache_path) and os.path.getsize(cache_path) > 0)
|
||||
raw, err = fetch_article_html(a["url"], cache_path, force=force_fetch)
|
||||
if raw:
|
||||
body_md = html_to_markdown(raw, a["url"])
|
||||
if not body_md:
|
||||
err = err or "trafilatura returned empty"
|
||||
if raw and not err:
|
||||
fetched += 1
|
||||
else:
|
||||
failed += 1
|
||||
fetch_log.append({"idx": idx, "url": a["url"], "ok": bool(body_md), "error": err})
|
||||
if need_net and sleep > 0 and idx < len(articles):
|
||||
time.sleep(sleep)
|
||||
fname = f"{idx:02d}-{slug}.md"
|
||||
fpath = os.path.join(out_dir, fname)
|
||||
content = render_markdown(a, entry)
|
||||
content = render_markdown(a, entry, body_md=body_md, fetch_error=err if fetch else None)
|
||||
with open(fpath, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
|
||||
if fetch and fetch_log:
|
||||
with open(os.path.join(art_cache_dir, "index.json"), "w", encoding="utf-8") as f:
|
||||
json.dump(fetch_log, f, ensure_ascii=False, indent=2)
|
||||
|
||||
entry["extracted"] = True
|
||||
entry["article_count"] = len(articles)
|
||||
entry["markdown_dir"] = f"markdown/{iid}"
|
||||
entry["extracted_at"] = now_iso()
|
||||
print(f"📄 期 #{iid} → {len(articles)} 篇文章 → {out_dir}", file=sys.stderr)
|
||||
if fetch:
|
||||
entry["articles_fetched"] = fetched
|
||||
entry["articles_failed"] = failed
|
||||
print(f"📄 期 #{iid} → {len(articles)} 篇文章 → {out_dir}（抓取 {fetched} 成功 / {failed} 失败）", file=sys.stderr)
|
||||
return len(articles), out_dir, True
|
||||
|
||||
def render_markdown(article, entry):
|
||||
def render_markdown(article, entry, body_md=None, fetch_error=None):
|
||||
"""文章 → markdown 文件内容。"""
|
||||
pub = entry.get("pub_date", "")
|
||||
issue_label = entry.get("title", f"SRE Weekly Issue #{entry.get('id')}")
|
||||
@@ -306,6 +508,10 @@ def render_markdown(article, entry):
|
||||
out.append(f"- **链接**: {article['url']}")
|
||||
if article["body"]:
|
||||
out += ["", "## 简介", "", article["body"]]
|
||||
if body_md:
|
||||
out += ["", "## 正文", "", body_md]
|
||||
elif fetch_error:
|
||||
out += ["", "## 正文", "", f"> ⚠️ 抓取失败：{fetch_error}"]
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
def resolve_targets(target, manifest):
|
||||
@@ -330,8 +536,15 @@ def cmd_extract(args, manifest, base_dir):
|
||||
if not ids:
|
||||
print("✅ 无待提取的期", file=sys.stderr)
|
||||
return
|
||||
fetch = not args.skip_fetch
|
||||
if fetch and trafilatura is None:
|
||||
sys.exit("❌ 需要抓取正文但缺少依赖：pip3 install --user --break-system-packages trafilatura（或加 --skip-fetch）")
|
||||
for iid in ids:
|
||||
extract_one(iid, manifest["issues"][str(iid)], base_dir, force=args.force)
|
||||
extract_one(
|
||||
iid, manifest["issues"][str(iid)], base_dir,
|
||||
force=args.force, fetch=fetch,
|
||||
force_fetch=args.force_fetch, sleep=args.sleep,
|
||||
)
|
||||
save_manifest(base_dir, manifest)
|
||||
|
||||
# ---------------------------------------------------------------- 展示
|
||||
@@ -362,16 +575,22 @@ def main():
|
||||
ap.add_argument("--dir", default=DEFAULT_DIR, help=f"工作目录（默认 {DEFAULT_DIR}）")
|
||||
ap.add_argument("--check", action="store_true", help="只检查有无新期，输出 NEW:id,id 或 NONE")
|
||||
ap.add_argument("--extract", metavar="ID|latest|pending|all", help="提取文章 → markdown")
|
||||
ap.add_argument("--backfill", metavar="FROM..TO", help="回溯抓取指定期号区间（绕过 RSS 限制，例：--backfill 400..500）")
|
||||
ap.add_argument("--status", action="store_true", help="打印 manifest 状态表")
|
||||
ap.add_argument("--issues", action="store_true", help="列出已登记的所有期")
|
||||
ap.add_argument("--force", action="store_true", help="强制重新提取已提取过的期")
|
||||
ap.add_argument("--skip-fetch", action="store_true", help="不抓文章正文（仅使用 feed 简介）")
|
||||
ap.add_argument("--force-fetch", action="store_true", help="忽略本地缓存重新抓文章 HTML")
|
||||
ap.add_argument("--sleep", type=float, default=1.5, help="每篇文章抓取之间的间隔秒数（默认 1.5）")
|
||||
args = ap.parse_args()
|
||||
|
||||
base_dir = os.path.abspath(os.path.expanduser(args.dir))
|
||||
os.makedirs(base_dir, exist_ok=True)
|
||||
|
||||
if args.extract or args.status or args.issues:
|
||||
if args.extract or args.status or args.issues or args.backfill:
|
||||
manifest = load_manifest(base_dir)
|
||||
if args.backfill:
|
||||
cmd_backfill(args, manifest, base_dir)
|
||||
if args.extract:
|
||||
cmd_extract(args, manifest, base_dir)
|
||||
if args.status:
|
||||
|
||||
Reference in New Issue
Block a user