Archived
refactor: worker 子系统全面异步化，拆掉同步转异步的桥接层
拆掉「同步调异步 / 异步桥同步」两类垫片的第二半（HTTP 线 B2 见前序提交）。 worker 侧此前是「起线程跑同步函数、函数体内部再用 asyncio.to_thread 把活塞给 另一条循环」——线程与协程互等，取消/暂停只能靠轮询。本轮把 worker 自身变成 协程，由 _start_task 直接挂到应用事件循环上，桥接层随之失去存在理由。 改动（按子系统）： - music_downloader：downloader / _aria2 / _concurrency / cover / _http / __main__ 全链路协程化。per-file 锁改 asyncio.Lock；计数闸改 「active 计数 + await asyncio.sleep(0.5)」的可取消等待（原阻塞式 with 会让 已取消任务仍要等全局名额腾出，卡死并拖后腿退款）；aria2p 的同步 RPC 仍经 to_thread，轮询改 asyncio.sleep；新增 core.http_client.iter_chunks，把**流式 读取期**的 aiohttp.ClientError 收敛成 UpstreamNetworkError（请求建立阶段早已 有这条路径，读取阶段此前没有，断流会绕过调用方的 UpstreamError 分类）。 删除 get_http_session / _SESSION_LOCAL。 models._RetryFromScratch 不再继承 requests.RequestException 而继承 Exception： 它是「续传起点失效、残片已清、从头重下」，既不是确定性失败 (DownloadException)，也不是残片已保留的网络失败(UpstreamError)，必须独立 成类且在 except 链中先于二者捕获。 - music_api：podcast.py 全部函数协程化（直链解析内部改 await NeteaseAPI.get_song_url）；compat.py 删除——16 个死掉的同步门面早已 清空，文件只剩空壳。 - services：_taskcore / _zipcore / batch / downloads / podcast / tasks 协程化。 threading.Thread → loop.create_task（任务句柄字段 worker_thread → worker_task），ThreadPoolExecutor → asyncio.gather + asyncio.Semaphore， 终态函数与结算回调全部改 await；监控线程里「worker 已死」判定改 task.done()。 删除 services/streaming.py（同步流垫片，唯一消费者已迁到 core.http_client.open）。zipfile 实例仍非线程安全，故「下载」并行、 「入包」严格串行。 - web：路由层 130+ 处 to_thread 拆除——被包住的目标自己已是协程，再包一层只是 把协程丢进线程里跑一圈。tasks_registry.start_task 直接 create_task。 web/app.py 关闭段去掉 async_bridge.shutdown()（不再有第二条约循环）。 web/routers/download.py 的播客端点补上 3 处漏掉的 await （extract_podcast_ref / get_radio_detail）——漏掉时不报错、只静默返回协程对象， 靠 pyright 抓出来后修正。 留在 to_thread 上的都是**正当的阻塞卸载**：同步 SQLite 落库与账本结算、 mutagen 全量重写标签、zipfile 整段拷贝、shutil.rmtree、aria2p 的同步 RPC、 PBKDF2 的 10 万次迭代。ruff 对残留的本地元数据操作（exists/stat/os.replace/ 往 .part 顺序写分块）按「微秒级、卸载的调度开销大于操作本身」放行，pyproject 逐条注明理由。 取消/暂停仍由 threading.Event 承载：置位方是请求线程/监控线程，读取方是 worker 协程，只有「置位」与「轮询读取」两种操作，既不需要跨循环唤醒也不会卡住事件 循环——比 asyncio.Event 更省心（ASYNC110 同理按此豁免）。 测试：10 个文件的替身边界同步迁移——网络替身从 core.sync_http 的 get_http_session 换成 downloader/服务模块的 _http.open + **真实** iter_chunks （顺带钉住读取期异常收敛）；`_run` 改为在独立线程里 asyncio.run(worker(...))； aria2 的假时钟从 time.sleep 换成 _POLL_INTERVAL=0；monitor 的线程句柄替身换成 done() 可控的轻量替身（asyncio.Task 的构造需要运行中的事件循环，而监控循环跑在 常驻线程里）；podcast / 封面 / 结算包装等替身一律改 async。 门禁：ruff 全过 / pyright 0-0 / pytest 383 项全绿（0 失败）。 下一步 B2：auth_system 12 个模块改 aiosqlite 背板、web 层剩余 to_thread 收尾； B3：删 core/sync_http.py、core/async_bridge.py、core/http_errors.py（现仅 sync_http 自用）。
This commit is contained in:
snowy-yang committed
1 parent
65adfc9089
commit
69a50c2c31
35 files changed
+1062
-1005
No files matched your search
+119
-125
@@ -13,25 +13,25 @@
|
||||
|
||||
术语：下文「条目」指一个下载单元——音乐歌单里是一首歌，播客整台里是一集。
|
||||
|
||||
线程模型（本轮最容易踩坏的地方，改动前请先读）：
|
||||
- 下载 I/O 在 ``ThreadPoolExecutor`` 里跑，窗口大小 = ``job.concurrency``；
|
||||
- **ZIP 写入与包内名去重只发生在调用线程**（``as_completed`` 消费结果时），
|
||||
:class:`zipfile.ZipFile` 不是线程安全的，且去重读改写 ``used_arcnames``
|
||||
需要串行——故下载与落包被拆成 ``download_one`` / ``write_entry`` 两个钩子，
|
||||
只有前者进线程池。
|
||||
并发模型（本轮最容易踩坏的地方，改动前请先读）：
|
||||
- 一个窗口 = ``job.concurrency`` 个条目，窗口内 ``asyncio.gather`` 并发下载；
|
||||
- **ZIP 写入与包内名去重只在引擎协程里串行发生**（按窗口顺序消费结果）——
|
||||
:class:`zipfile.ZipFile` 不是并发安全的，且去重读改写 ``used_arcnames``
|
||||
需要串行；因此 ``download_one``（并发）与 ``write_entry``（串行）是两个钩子，
|
||||
只有前者进 gather。``write_entry`` 里的重型落盘（几百 MB 的音频条目）
|
||||
由实现方自行经 ``asyncio.to_thread`` 卸载，避免卡住事件循环。
|
||||
|
||||
``run_zip_job`` 的四个终态（成功 / 全失败 / 取消 / 异常）原先各写一段 8~20 行的
|
||||
收尾内联在 try/except 里。此处把「一次运行」的状态收进 :class:`_ZipRun`，
|
||||
四个终态各自成函数——改账本口径或补一种终态时，不必再在 167 行里找落点。
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
import zipfile
|
||||
from collections.abc import Callable
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -80,7 +80,7 @@ class ZipContext:
|
||||
|
||||
task_id: str
|
||||
params: dict[str, Any]
|
||||
cancel_ev: threading.Event
|
||||
cancel_ev: Any
|
||||
work_dir: Path
|
||||
zf: zipfile.ZipFile
|
||||
progress: Callable[..., None]
|
||||
@@ -106,7 +106,7 @@ class ZipJob:
|
||||
|
||||
子类至少实现 :meth:`prepare` / :meth:`download_one` / :meth:`write_entry` /
|
||||
:meth:`zip_filename`，其余钩子按需覆写。**实例每次运行新建**——下载器信息
|
||||
预取池等运行期状态就挂在实例上。
|
||||
预取任务等运行期状态就挂在实例上。
|
||||
"""
|
||||
|
||||
tag = '批量打包' # 日志前缀
|
||||
@@ -119,7 +119,7 @@ class ZipJob:
|
||||
|
||||
# ---- 必须实现 ----
|
||||
|
||||
def prepare(self, params: dict[str, Any], task_id: str, failed: list[str]) -> list[Any]:
|
||||
async def prepare(self, params: dict[str, Any], task_id: str, failed: list[str]) -> list[Any]:
|
||||
"""组装条目清单。可抛 ``DownloadException`` 让任务按业务失败收敛。
|
||||
|
||||
``failed`` 可被直接追加——清单阶段就有条目已知失败（如按 ID 列表时个别
|
||||
@@ -127,8 +127,8 @@ class ZipJob:
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def download_one(self, item: Any, idx: int, ctx: ZipContext) -> Any:
|
||||
"""下载单个条目（**在下载线程池内执行**）。
|
||||
async def download_one(self, item: Any, idx: int, ctx: ZipContext) -> Any:
|
||||
"""下载单个条目（**在窗口内并发执行**）。
|
||||
|
||||
返回的 ``payload`` 会原样交给 :meth:`write_entry`（放临时文件名、
|
||||
已取到的元信息等）。失败请抛异常：``ItemFailed`` 用其自带文案，
|
||||
@@ -136,10 +136,12 @@ class ZipJob:
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def write_entry(self, item: Any, idx: int, payload: Any, ctx: ZipContext) -> None:
|
||||
"""把下载结果写入 ZIP（**在引擎调用线程内执行，天然串行**）。
|
||||
async def write_entry(self, item: Any, idx: int, payload: Any, ctx: ZipContext) -> None:
|
||||
"""把下载结果写入 ZIP（**由引擎按窗口顺序串行调用**）。
|
||||
|
||||
包内名去重在这里做——需要读写 ``ctx.used_arcnames``，必须串行。
|
||||
重型落盘请自行 ``await asyncio.to_thread(...)``：直接在协程里写
|
||||
几百 MB 会把事件循环卡住。
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@@ -149,7 +151,7 @@ class ZipJob:
|
||||
|
||||
# ---- 可选钩子 ----
|
||||
|
||||
def prefetch(self, window: list[Any], ctx: ZipContext) -> None:
|
||||
async def prefetch(self, window: list[Any], ctx: ZipContext) -> None:
|
||||
"""窗口级预热（每窗口一次、提交下载前调用）。默认无操作。"""
|
||||
|
||||
def fail_label(self, item: Any) -> str:
|
||||
@@ -174,14 +176,14 @@ class ZipJob:
|
||||
return f"部分{self.item_noun}已打包完成，可领取" + (
|
||||
f"，退回 {refunded} 次" if refunded else '')
|
||||
|
||||
def supplement(self, task_id: str, params: dict[str, Any], count: int) -> None:
|
||||
async def supplement(self, task_id: str, params: dict[str, Any], count: int) -> None:
|
||||
"""清单确定后的计费补扣（默认无操作）。"""
|
||||
|
||||
def on_delivered(self, items: list[Any], task_id: str, params: dict[str, Any]) -> None:
|
||||
async def on_delivered(self, items: list[Any], task_id: str, params: dict[str, Any]) -> None:
|
||||
"""交付落定后的操作记录（默认无操作）。"""
|
||||
|
||||
def cleanup(self) -> None:
|
||||
"""释放运行期资源（如信息预取池）。默认无操作。"""
|
||||
async def cleanup(self) -> None:
|
||||
"""释放运行期资源（如信息预取任务）。默认无操作。"""
|
||||
|
||||
|
||||
def _discard(path: str | None) -> None:
|
||||
@@ -206,12 +208,12 @@ class _ZipRun:
|
||||
|
||||
job: ZipJob
|
||||
task_id: str
|
||||
settle: Callable[[str, int, str], int]
|
||||
settle: Callable[[str, int, str], Awaitable[int]]
|
||||
tally: _Tally = field(default_factory=_Tally)
|
||||
task: dict[str, Any] = field(default_factory=dict)
|
||||
params: dict[str, Any] = field(default_factory=dict)
|
||||
cancel_ev: threading.Event | None = None
|
||||
pause_ev: threading.Event | None = None
|
||||
cancel_ev: Any = None
|
||||
pause_ev: Any = None
|
||||
items: list[Any] = field(default_factory=list)
|
||||
zip_tmp_path: str | None = None
|
||||
zip_valid: bool = False
|
||||
@@ -232,7 +234,7 @@ def _progress_of(run: _ZipRun) -> Callable[..., None]:
|
||||
return _progress
|
||||
|
||||
|
||||
def _start_zip_run(run: _ZipRun) -> None:
|
||||
async def _start_zip_run(run: _ZipRun) -> None:
|
||||
"""置 running、组装清单、写回 total、执行计费补扣。"""
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
task = DOWNLOAD_TASKS[run.task_id]
|
||||
@@ -245,13 +247,13 @@ def _start_zip_run(run: _ZipRun) -> None:
|
||||
# 计耗时只算「下载+打包」这一段（不含排队），故不用 task['start']
|
||||
run.t0 = time.time()
|
||||
|
||||
run.items = run.job.prepare(run.params, run.task_id, run.tally.failed)
|
||||
run.items = await run.job.prepare(run.params, run.task_id, run.tally.failed)
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
DOWNLOAD_TASKS[run.task_id]['total'] = len(run.items)
|
||||
run.job.supplement(run.task_id, run.params, len(run.items))
|
||||
await run.job.supplement(run.task_id, run.params, len(run.items))
|
||||
|
||||
|
||||
def _build_zip_archive(run: _ZipRun) -> None:
|
||||
async def _build_zip_archive(run: _ZipRun) -> None:
|
||||
"""建临时 ZIP 并逐窗口填充。返回时 ``zip_valid`` 才置位。
|
||||
|
||||
ZIP 构建放 downloads 卷（原实现落系统 %TEMP%——500 首无损可达 GB 级，
|
||||
@@ -269,7 +271,7 @@ def _build_zip_archive(run: _ZipRun) -> None:
|
||||
ctx = ZipContext(task_id=run.task_id, params=run.params,
|
||||
cancel_ev=run.cancel_ev, work_dir=work_dir, zf=zf,
|
||||
progress=_progress_of(run), total=len(run.items))
|
||||
_fill_zip(job, run.items, ctx, run.tally, run.task, run.pause_ev)
|
||||
await _fill_zip(job, run.items, ctx, run.tally, run.task, run.pause_ev)
|
||||
# zip_valid 只在 ZIP with 块正常退出（中央目录完整写出）后才置 True。
|
||||
# 此前两个 worker 在这之后还各留了一段 `if not zip_valid:` 失败处理——
|
||||
# 紧跟赋值、中间无任何可改zip_valid的语句，是**不可达**的复审残留，已删。
|
||||
@@ -280,7 +282,7 @@ def _build_zip_archive(run: _ZipRun) -> None:
|
||||
BATCH_DURATIONS.append(time.time() - run.t0)
|
||||
|
||||
|
||||
def _finish_all_failed(run: _ZipRun) -> None:
|
||||
async def _finish_all_failed(run: _ZipRun) -> None:
|
||||
"""全部失败（含清单非空但一条都没成）：不发布空 ZIP、不按交付结算，转 error 并全额退费。
|
||||
|
||||
原音乐批量用 ``ok==0 and failed``、播客用早抛 ``DownloadException``——这里
|
||||
@@ -289,8 +291,8 @@ def _finish_all_failed(run: _ZipRun) -> None:
|
||||
job, tally = run.job, run.tally
|
||||
_discard(run.zip_tmp_path)
|
||||
run.zip_tmp_path = None
|
||||
refunded = run.settle(run.task_id, 0, 'error')
|
||||
_mark_task_terminal(run.task_id, 'error', job.all_failed_msg(tally.failed, refunded))
|
||||
refunded = await run.settle(run.task_id, 0, 'error')
|
||||
await _mark_task_terminal(run.task_id, 'error', job.all_failed_msg(tally.failed, refunded))
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(run.task_id)
|
||||
if t is not None:
|
||||
@@ -300,7 +302,7 @@ def _finish_all_failed(run: _ZipRun) -> None:
|
||||
f"{job.tag}全部失败: {run.task_id} 失败{len(tally.failed)} 退回{refunded}次")
|
||||
|
||||
|
||||
def _publish_delivery(run: _ZipRun) -> None:
|
||||
async def _publish_delivery(run: _ZipRun) -> None:
|
||||
"""正常交付：按入包成功数结算，锁内复查取消/终态后发布结果。
|
||||
|
||||
``'settled'`` 是 ``_settle_batch_billing`` 的默认值，此处显式写出，让
|
||||
@@ -308,7 +310,7 @@ def _publish_delivery(run: _ZipRun) -> None:
|
||||
"""
|
||||
job, tally = run.job, run.tally
|
||||
assert run.cancel_ev is not None
|
||||
refunded = run.settle(run.task_id, tally.ok, 'settled')
|
||||
refunded = await run.settle(run.task_id, tally.ok, 'settled')
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(run.task_id)
|
||||
# 发布前锁内复查取消/失效状态——取消请求落在「检查后、拿锁前」的窗口
|
||||
@@ -329,24 +331,24 @@ def _publish_delivery(run: _ZipRun) -> None:
|
||||
'filename': job.zip_filename(tally.ok),
|
||||
'audio_files': [], # 缓存文件不归任务所有，不进入清理清单
|
||||
}
|
||||
job.on_delivered(run.items, run.task_id, run.params)
|
||||
await job.on_delivered(run.items, run.task_id, run.params)
|
||||
api_service.logger.info(
|
||||
f"{job.tag}完成: {job.log_label()}成功{tally.ok} 失败{len(tally.failed)} "
|
||||
f"退回{refunded}次 ({run.task_id})")
|
||||
|
||||
|
||||
def _finish_canceled(run: _ZipRun) -> None:
|
||||
async def _finish_canceled(run: _ZipRun) -> None:
|
||||
"""部分成功后取消——已入包部分照常交付（成果守恒），未完成部分按账本退费。
|
||||
|
||||
ZIP 未正常完成（写坏）时不按 ``ok_count`` 交付——按全额退费收敛。
|
||||
"""
|
||||
job, tally = run.job, run.tally
|
||||
delivered = tally.ok if run.zip_valid else 0
|
||||
refunded = run.settle(run.task_id, delivered, 'canceled')
|
||||
refunded = await run.settle(run.task_id, delivered, 'canceled')
|
||||
if delivered > 0 and run.zip_tmp_path:
|
||||
_mark_task_terminal(run.task_id, 'canceled',
|
||||
f"已取消（已打包 {tally.ok} {job.noun}可领取）" +
|
||||
(f"，退回 {refunded} 次" if refunded else ''))
|
||||
await _mark_task_terminal(run.task_id, 'canceled',
|
||||
f"已取消（已打包 {tally.ok} {job.noun}可领取）" +
|
||||
(f"，退回 {refunded} 次" if refunded else ''))
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(run.task_id)
|
||||
if t is not None:
|
||||
@@ -358,9 +360,9 @@ def _finish_canceled(run: _ZipRun) -> None:
|
||||
}
|
||||
else:
|
||||
_discard(run.zip_tmp_path)
|
||||
_mark_task_terminal(run.task_id, 'canceled',
|
||||
'任务已取消' + (
|
||||
f"（已退回 {refunded} 次下载次数）" if refunded else ''))
|
||||
await _mark_task_terminal(run.task_id, 'canceled',
|
||||
'任务已取消' + (
|
||||
f"（已退回 {refunded} 次下载次数）" if refunded else ''))
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(run.task_id)
|
||||
if t is not None:
|
||||
@@ -369,29 +371,29 @@ def _finish_canceled(run: _ZipRun) -> None:
|
||||
f"{job.tag}已取消: {run.task_id}（成功{tally.ok} 退回{refunded}次）")
|
||||
|
||||
|
||||
def _finish_business_error(run: _ZipRun, exc: DownloadException) -> None:
|
||||
async def _finish_business_error(run: _ZipRun, exc: DownloadException) -> None:
|
||||
"""业务失败（清单阶段：无可下载条目 / 补扣下载次数失败 / 上游不可用）。
|
||||
|
||||
文案是我们自己拼的用户可读文本，保留原文（同播客单集下载）。
|
||||
"""
|
||||
job, tally = run.job, run.tally
|
||||
delivered = tally.ok if run.zip_valid else 0
|
||||
refunded = run.settle(run.task_id, delivered, 'error')
|
||||
_apply_error(run.task_id, str(exc), refunded)
|
||||
refunded = await run.settle(run.task_id, delivered, 'error')
|
||||
await _apply_error(run.task_id, str(exc), refunded)
|
||||
_discard(run.zip_tmp_path)
|
||||
api_service.logger.error(
|
||||
f"{job.tag}失败: {run.task_id} {exc}（成功{tally.ok} 退回{refunded}次）")
|
||||
|
||||
|
||||
def _finish_unexpected(run: _ZipRun, exc: Exception) -> None:
|
||||
async def _finish_unexpected(run: _ZipRun, exc: Exception) -> None:
|
||||
"""非预期异常：原文 logger 先行（结算本身可能再抛，先落下 traceback），
|
||||
用户可见文案泛化；带上退款结果再补一条，避免排障时只看到「异常」。"""
|
||||
job, tally = run.job, run.tally
|
||||
api_service.logger.error(f"{job.tag}异常: {run.task_id} {exc}\n{traceback.format_exc()}")
|
||||
delivered = tally.ok if run.zip_valid else 0
|
||||
refunded = run.settle(run.task_id, delivered, 'error')
|
||||
refunded = await run.settle(run.task_id, delivered, 'error')
|
||||
if delivered > 0 and run.zip_tmp_path:
|
||||
_apply_error(run.task_id, job.partial_msg(refunded), refunded)
|
||||
await _apply_error(run.task_id, job.partial_msg(refunded), refunded)
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(run.task_id)
|
||||
if t is not None:
|
||||
@@ -401,16 +403,16 @@ def _finish_unexpected(run: _ZipRun, exc: Exception) -> None:
|
||||
'audio_files': [],
|
||||
}
|
||||
else:
|
||||
_apply_error(run.task_id, '打包失败，请稍后重试' + (
|
||||
await _apply_error(run.task_id, '打包失败，请稍后重试' + (
|
||||
f"（已退回 {refunded} 次下载次数）" if refunded else ''), refunded)
|
||||
_discard(run.zip_tmp_path)
|
||||
api_service.logger.error(
|
||||
f"{job.tag}异常: {run.task_id}（成功{tally.ok} 退回{refunded}次）")
|
||||
|
||||
|
||||
def run_zip_job(job: ZipJob, task_id: str, *,
|
||||
settle: Callable[[str, int, str], int] = _settle_batch_billing) -> None:
|
||||
"""执行一次批量打包任务（在 worker 线程内调用，自身不向外抛异常）。
|
||||
async def run_zip_job(job: ZipJob, task_id: str, *,
|
||||
settle: Callable[[str, int, str], Awaitable[int]] = _settle_batch_billing) -> None:
|
||||
"""执行一次批量打包任务（在 worker 协程内调用，自身不向外抛异常）。
|
||||
|
||||
``settle`` 由调用方注入，默认走 :func:`services._taskcore._settle_batch_billing`。
|
||||
做成参数是为了让测试能在**模块级**替换结算函数（包装层在调用时读取模块全局，
|
||||
@@ -424,99 +426,91 @@ def run_zip_job(job: ZipJob, task_id: str, *,
|
||||
"""
|
||||
run = _ZipRun(job=job, task_id=task_id, settle=settle)
|
||||
try:
|
||||
_start_zip_run(run)
|
||||
_build_zip_archive(run)
|
||||
await _start_zip_run(run)
|
||||
await _build_zip_archive(run)
|
||||
|
||||
if run.tally.ok == 0:
|
||||
_finish_all_failed(run)
|
||||
await _finish_all_failed(run)
|
||||
return
|
||||
_publish_delivery(run)
|
||||
await _publish_delivery(run)
|
||||
except DownloadCanceled:
|
||||
_finish_canceled(run)
|
||||
await _finish_canceled(run)
|
||||
except DownloadException as e:
|
||||
_finish_business_error(run, e)
|
||||
await _finish_business_error(run, e)
|
||||
except Exception as e:
|
||||
_finish_unexpected(run, e)
|
||||
await _finish_unexpected(run, e)
|
||||
finally:
|
||||
# 回收信息预取线程池，避免线程泄漏（未启动的预取任务一并取消）
|
||||
job.cleanup()
|
||||
# 回收信息预取任务，避免任务泄漏（未启动的预取任务一并取消）
|
||||
await job.cleanup()
|
||||
|
||||
|
||||
def _apply_error(task_id: str, msg: str, refunded: int) -> None:
|
||||
async def _apply_error(task_id: str, msg: str, refunded: int) -> None:
|
||||
"""异常/业务失败的终态收敛：置 error（终态先到先得）、记账退款额。
|
||||
|
||||
走 :func:`_mark_task_terminal` 而非就地写任务字典——后者是播客侧的旧写法，
|
||||
除了重复一遍「已终态不覆盖」判断外，还漏掉了 worker 线程已死时按账本补结算
|
||||
的兜底（``/F-E``）。
|
||||
除了重复一遍「已终态不覆盖」判断外，还漏掉了 worker 已结束未结算时按账本
|
||||
补结算的兜底（``/F-E``）。
|
||||
"""
|
||||
_mark_task_terminal(task_id, 'error', msg)
|
||||
await _mark_task_terminal(task_id, 'error', msg)
|
||||
with DOWNLOAD_TASKS_LOCK:
|
||||
t = DOWNLOAD_TASKS.get(task_id)
|
||||
if t is not None:
|
||||
t['refunded'] = refunded
|
||||
|
||||
|
||||
def _fill_zip(job: ZipJob, items: list[Any], ctx: ZipContext, tally: _Tally,
|
||||
task: dict[str, Any], pause_ev: threading.Event) -> None:
|
||||
async def _fill_zip(job: ZipJob, items: list[Any], ctx: ZipContext, tally: _Tally,
|
||||
task: dict[str, Any], pause_ev) -> None:
|
||||
"""逐窗口下载并写入 ZIP（返回时中央目录尚未写出，由调用方的 with 退出时落盘）。
|
||||
|
||||
窗口 = ``job.concurrency`` 个条目：下载并行、落包串行。窗口大小取 1 时
|
||||
窗口 = ``job.concurrency`` 个条目：下载并发、落包串行。窗口大小取 1 时
|
||||
退化为顺序下载（播客整台即如此）。
|
||||
|
||||
取消语义与旧的线程池版本对齐：等待名额期间可被取消；窗口内某条抛
|
||||
DownloadCanceled 时整批中止（在途下载因每次分块都查取消位而很快退出）。
|
||||
"""
|
||||
concurrency = max(1, int(job.concurrency))
|
||||
pool: ThreadPoolExecutor | None = None
|
||||
try:
|
||||
# 不用 with 管理下载池——取消时 with 退出会 wait=True 等在途 future
|
||||
# （其内部阻塞在预取 HTTP 上最长 30s），改为不等待 + 取消排队任务
|
||||
pool = ThreadPoolExecutor(max_workers=concurrency)
|
||||
offset = 0
|
||||
while offset < len(items):
|
||||
if ctx.cancel_ev.is_set():
|
||||
raise DownloadCanceled("任务已取消")
|
||||
# 断线暂停：释放名额等待恢复/取消，恢复后重新进入排队
|
||||
_wait_if_paused(task, pause_ev, ctx.cancel_ev, tally.ok + len(tally.failed))
|
||||
# 每个窗口持有 BATCH_ZIP_SEM（同一时刻全局最多 N 个批量任务在下载）
|
||||
# 等待名额期间可被取消，不再无限阻塞线程与退款
|
||||
if not BATCH_ZIP_SEM.acquire_cancellable(ctx.cancel_ev):
|
||||
raise DownloadCanceled("任务已取消")
|
||||
try:
|
||||
# 暂停/取消落在「等待恢复返回」与「拿到名额」之间：立刻放名额回到
|
||||
# 循环顶部重新处理暂停（原播客侧缺这一步，会多下完这一集才停）
|
||||
if (ctx.cancel_ev.is_set() or pause_ev.is_set()
|
||||
or task['status'] == 'paused'):
|
||||
continue
|
||||
window = items[offset:offset + concurrency]
|
||||
start_idx = offset + 1 # 条目序号从 1 起（进度文案用）
|
||||
offset += len(window)
|
||||
job.prefetch(window, ctx) # 只预取本窗口
|
||||
futures = {
|
||||
pool.submit(job.download_one, item, start_idx + i, ctx): (item, start_idx + i)
|
||||
for i, item in enumerate(window)
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
if ctx.cancel_ev.is_set():
|
||||
raise DownloadCanceled("任务已取消")
|
||||
item, idx = futures[future]
|
||||
try:
|
||||
payload = future.result()
|
||||
# 落包在主线程：ZipFile 非线程安全 + 包内名去重要串行
|
||||
job.write_entry(item, idx, payload, ctx)
|
||||
except DownloadCanceled:
|
||||
raise
|
||||
except ItemFailed as f:
|
||||
tally.failed.append(f.label)
|
||||
except Exception as e:
|
||||
tally.failed.append(f"{job.fail_label(item)}({e})")
|
||||
else:
|
||||
tally.ok += 1
|
||||
ctx.progress(job.done_msg(item, idx, ctx))
|
||||
finally:
|
||||
BATCH_ZIP_SEM.release()
|
||||
finally:
|
||||
if pool is not None:
|
||||
try:
|
||||
# 取消/异常时不等待在途下载线程（其可能阻塞在预取 HTTP 上），
|
||||
# 排队任务一并取消；在途线程自行收尾（.part 残片由续传/清扫兜底）
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
except TypeError:
|
||||
pool.shutdown(wait=False)
|
||||
offset = 0
|
||||
while offset < len(items):
|
||||
if ctx.cancel_ev.is_set():
|
||||
raise DownloadCanceled("任务已取消")
|
||||
# 断线暂停：释放名额等待恢复/取消，恢复后重新进入排队
|
||||
await _wait_if_paused(task, pause_ev, ctx.cancel_ev, tally.ok + len(tally.failed))
|
||||
# 每个窗口持有 BATCH_ZIP_SEM（同一时刻全局最多 N 个批量任务在下载）
|
||||
# 等待名额期间可被取消，不再无限等待与延迟退款
|
||||
if not await BATCH_ZIP_SEM.acquire_cancellable(ctx.cancel_ev):
|
||||
raise DownloadCanceled("任务已取消")
|
||||
try:
|
||||
# 暂停/取消落在「等待恢复返回」与「拿到名额」之间：立刻放名额回到
|
||||
# 循环顶部重新处理暂停（原播客侧缺这一步，会多下完这一集才停）
|
||||
if (ctx.cancel_ev.is_set() or pause_ev.is_set()
|
||||
or task['status'] == 'paused'):
|
||||
continue
|
||||
window = items[offset:offset + concurrency]
|
||||
start_idx = offset + 1 # 条目序号从 1 起（进度文案用）
|
||||
offset += len(window)
|
||||
await job.prefetch(window, ctx) # 只预取本窗口
|
||||
results = await asyncio.gather(
|
||||
*(job.download_one(item, start_idx + i, ctx)
|
||||
for i, item in enumerate(window)),
|
||||
return_exceptions=True,
|
||||
)
|
||||
for i, result in enumerate(results):
|
||||
if ctx.cancel_ev.is_set():
|
||||
raise DownloadCanceled("任务已取消")
|
||||
item, idx = window[i], start_idx + i
|
||||
try:
|
||||
if isinstance(result, BaseException):
|
||||
raise result
|
||||
# 落包串行：ZipFile 非并发安全 + 包内名去重要按序
|
||||
await job.write_entry(item, idx, result, ctx)
|
||||
except DownloadCanceled:
|
||||
raise
|
||||
except ItemFailed as f:
|
||||
tally.failed.append(f.label)
|
||||
except Exception as e:
|
||||
tally.failed.append(f"{job.fail_label(item)}({e})")
|
||||
else:
|
||||
tally.ok += 1
|
||||
ctx.progress(job.done_msg(item, idx, ctx))
|
||||
finally:
|
||||
BATCH_ZIP_SEM.release()
|
||||
Reference in new issue
Block a user