refactor: worker 子系统全面异步化,拆掉同步转异步的桥接层

拆掉「同步调异步 / 异步桥同步」两类垫片的第二半(HTTP 线 B2 见前序提交)。
worker 侧此前是「起线程跑同步函数、函数体内部再用 asyncio.to_thread 把活塞给
另一条循环」——线程与协程互等,取消/暂停只能靠轮询。本轮把 worker 自身变成
协程,由 _start_task 直接挂到应用事件循环上,桥接层随之失去存在理由。

改动(按子系统):
- music_downloader:downloader / _aria2 / _concurrency / cover / _http /
  __main__ 全链路协程化。per-file 锁改 asyncio.Lock;计数闸改
  「active 计数 + await asyncio.sleep(0.5)」的可取消等待(原阻塞式 with 会让
  已取消任务仍要等全局名额腾出,卡死并拖后腿退款);aria2p 的同步 RPC 仍经
  to_thread,轮询改 asyncio.sleep;新增 core.http_client.iter_chunks,把**流式
  读取期**的 aiohttp.ClientError 收敛成 UpstreamNetworkError(请求建立阶段早已
  有这条路径,读取阶段此前没有,断流会绕过调用方的 UpstreamError 分类)。
  删除 get_http_session / _SESSION_LOCAL。
  models._RetryFromScratch 不再继承 requests.RequestException 而继承 Exception:
  它是「续传起点失效、残片已清、从头重下」,既不是确定性失败
  (DownloadException),也不是残片已保留的网络失败(UpstreamError),必须独立
  成类且在 except 链中先于二者捕获。
- music_api:podcast.py 全部函数协程化(直链解析内部改
  await NeteaseAPI.get_song_url);compat.py 删除——16 个死掉的同步门面早已
  清空,文件只剩空壳。
- services:_taskcore / _zipcore / batch / downloads / podcast / tasks 协程化。
  threading.Thread → loop.create_task(任务句柄字段 worker_thread →
  worker_task),ThreadPoolExecutor → asyncio.gather + asyncio.Semaphore,
  终态函数与结算回调全部改 await;监控线程里「worker 已死」判定改 task.done()。
  删除 services/streaming.py(同步流垫片,唯一消费者已迁到
  core.http_client.open)。zipfile 实例仍非线程安全,故「下载」并行、
  「入包」严格串行。
- web:路由层 130+ 处 to_thread 拆除——被包住的目标自己已是协程,再包一层只是
  把协程丢进线程里跑一圈。tasks_registry.start_task 直接 create_task。
  web/app.py 关闭段去掉 async_bridge.shutdown()(不再有第二条约循环)。
  web/routers/download.py 的播客端点补上 3 处漏掉的 await
  (extract_podcast_ref / get_radio_detail)——漏掉时不报错、只静默返回协程对象,
  靠 pyright 抓出来后修正。

留在 to_thread 上的都是**正当的阻塞卸载**:同步 SQLite 落库与账本结算、
mutagen 全量重写标签、zipfile 整段拷贝、shutil.rmtree、aria2p 的同步 RPC、
PBKDF2 的 10 万次迭代。ruff 对残留的本地元数据操作(exists/stat/os.replace/
往 .part 顺序写分块)按「微秒级、卸载的调度开销大于操作本身」放行,pyproject
逐条注明理由。

取消/暂停仍由 threading.Event 承载:置位方是请求线程/监控线程,读取方是 worker
协程,只有「置位」与「轮询读取」两种操作,既不需要跨循环唤醒也不会卡住事件
循环——比 asyncio.Event 更省心(ASYNC110 同理按此豁免)。

测试:10 个文件的替身边界同步迁移——网络替身从 core.sync_http 的
get_http_session 换成 downloader/服务模块的 _http.open + **真实** iter_chunks
(顺带钉住读取期异常收敛);`_run` 改为在独立线程里 asyncio.run(worker(...));
aria2 的假时钟从 time.sleep 换成 _POLL_INTERVAL=0;monitor 的线程句柄替身换成
done() 可控的轻量替身(asyncio.Task 的构造需要运行中的事件循环,而监控循环跑在
常驻线程里);podcast / 封面 / 结算包装等替身一律改 async。

门禁:ruff 全过 / pyright 0-0 / pytest 383 项全绿(0 失败)。

下一步 B2:auth_system 12 个模块改 aiosqlite 背板、web 层剩余 to_thread 收尾;
B3:删 core/sync_http.py、core/async_bridge.py、core/http_errors.py(现仅
sync_http 自用)。
This commit is contained in:
snowy-yang committed 2026-09-28 19:36:56 +08:00
1 parent 65adfc9089
commit 69a50c2c31
35 files changed
+1062 -1005

No files matched your search

+92 -94
View File
@@ -1,19 +1,24 @@
# -*- coding: utf-8 -*-
"""下载主类:落盘、元数据写入与 ZIP 打包。
"""下载主类:落盘、元数据写入与交付。
(自 music_downloader.py 拆出,行为与原实现一致。)
(自 music_downloader.py 拆出。)
异步口径:整个下载链路是协程。网络走 core.http_client(进程级连接池),
文件落盘/标签写入是阻塞 IO 与纯 Python CPU 工作,经 ``asyncio.to_thread`` 卸载;
互斥改 asyncio 锁(见 _concurrency),取消仍由 threading.Event 承载——
取消位由请求线程/监控线程置位,只有轮询读取,跨线程安全且无需跨循环唤醒。
"""
from __future__ import annotations
import asyncio
import os
import re
import threading
import time
from pathlib import Path
from core import sync_http as requests # 同步门面(底层 aiohttp,替代 requests)
from core.async_bridge import run_sync
from core.http_client import UpstreamError, UpstreamHTTPError, iter_chunks
from core.http_client import client as _http
from music_api import APIException, NeteaseAPI, load_cookies
from ._aria2 import Aria2Backend, get_backend
@@ -24,7 +29,6 @@ from ._concurrency import (
from ._http import (
DL_CHUNK_SIZE,
_sniff_audio_header,
get_http_session,
)
from ._log import (
logger,
@@ -52,7 +56,7 @@ class MusicDownloader:
用法:
downloader = MusicDownloader(download_dir="downloads")
result = downloader.download_music_file(music_id, quality="lossless")
result = await downloader.download_music_file(music_id, quality="lossless")
"""
def __init__(
@@ -66,7 +70,7 @@ class MusicDownloader:
download_dir: 下载目录
max_concurrent: 单个批量任务内的最大并发下载数
global_max_concurrent: 整个服务同时下载的文件数(含标签写入)。
所有请求共享该信号量,防止大量用户同时下载导致内存/CPU 耗尽。
所有请求共享该闸,防止大量用户同时下载导致内存/CPU 耗尽。
"""
self.download_dir = Path(download_dir)
self.download_dir.mkdir(exist_ok=True)
@@ -74,14 +78,14 @@ class MusicDownloader:
self.api = NeteaseAPI()
# 全局并发信号量:所有请求共用,限制同时进行的下载数
# 全局并发闸:所有请求共用,限制同时进行的下载数
# (带计数,供前端实时展示排队情况)
self._download_semaphore = CountingSemaphore(global_max_concurrent)
# 标签写入专用信号量:写标签会重写整个文件(IO 密集),
# 标签写入专用闸:写标签会重写整个文件(IO 密集),
# 与下载分离限流 —— 写满时只阻塞后来者,不占下载名额、不拖慢其它文件的传输
cpu_count = os.cpu_count() or 2
self._tag_semaphore = threading.BoundedSemaphore(max(1, min(4, cpu_count)))
self._tag_semaphore = asyncio.Semaphore(max(1, min(4, cpu_count)))
# ----------- 工具方法 -----------
@@ -145,24 +149,20 @@ class MusicDownloader:
# ----------- 信息获取 -----------
def get_music_info(self, music_id: int, quality: str = "standard",
cookies: dict[str, str] | None = None) -> MusicInfo:
"""获取音乐详细信息(同步,URL/详情/歌词 3 个接口并行请求,加快单曲/批量启动)"""
async def get_music_info(self, music_id: int, quality: str = "standard",
cookies: dict[str, str] | None = None) -> MusicInfo:
"""获取音乐详细信息(URL/详情/歌词 3 个接口并发请求,加快单曲/批量启动)"""
try:
cookies = cookies if cookies is not None else load_cookies()
# 3 个接口并行请求,单曲下载/批量预取都能明显提速
# URL/详情拿到后立即取消未完成的歌词请求,已知失败不再被拖着等完
# 异步化:改用 asyncio.gather 并发(api 方法已异步),失败语义与旧线程池一致
import asyncio
async def _gather_info():
url_t = self.api.get_song_url(music_id, quality, cookies)
detail_t = self.api.get_song_detail(music_id)
lyric_t = self.api.get_lyric(music_id, cookies)
return await asyncio.gather(url_t, detail_t, lyric_t, return_exceptions=True)
url_result, detail_result, lyric_result = run_sync(_gather_info())
# 3 个接口并发请求,单曲下载/批量预取都能明显提速
# URL/详情拿到后不再等待失败的歌词请求,已知失败不拖着等完
url_result, detail_result, lyric_result = await asyncio.gather(
self.api.get_song_url(music_id, quality, cookies),
self.api.get_song_detail(music_id),
self.api.get_lyric(music_id, cookies),
return_exceptions=True,
)
if isinstance(url_result, BaseException):
raise url_result
if isinstance(detail_result, BaseException):
@@ -255,9 +255,9 @@ class MusicDownloader:
safe_tag = self._sanitize_filename(tag, max_bytes=tag_budget)
return self.download_dir / f"{safe_base}{safe_tag}{ext}"
# ----------- 下载(同步) -----------
# ----------- 下载 -----------
def download_music_file(
async def download_music_file(
self,
music_id: int,
quality: str = "standard",
@@ -269,7 +269,7 @@ class MusicDownloader:
music_info: MusicInfo | None = None,
progress_cb=None,
) -> DownloadResult:
"""同步下载音乐到本地
"""下载音乐到本地
Args:
music_id: 音乐ID
@@ -288,15 +288,15 @@ class MusicDownloader:
- 流式下载到 .part 临时文件,完整下载后再原子改名,半截文件不会被当作已下载
- 断点续传:失败保留 .part,重试/下次下载从已写字节续传(服务器支持 Range 时)
- per-file 锁:同名歌曲并发下载互斥,防止 .part 被交叉写入损坏
- 连接复用:线程级 Session + keep-alive,避免每首歌重建 TCP/TLS 连接
- 网络传输与 ffmpeg 元数据写入分离限流,互不抢占对方资源
- 连接复用:core.http_client 的进程级连接池 + keep-alive
- 网络传输与标签写入分离限流,互不抢占对方资源
本函数只保留编排与重试策略:缓存校验、续传起点、传输通道、完整性判定、
元数据限流各自见对应私有方法的 docstring(此前是 205 行 / 圈复杂度 47
的单体,任何一处改动都要通读全部步骤)。
"""
if music_info is None:
music_info = self.get_music_info(music_id, quality, cookies)
music_info = await self.get_music_info(music_id, quality, cookies)
file_path = self._build_file_path(music_info)
part_path0 = file_path.with_name(file_path.name + ".part")
@@ -304,9 +304,9 @@ class MusicDownloader:
next_wait = 0.0
for attempt in range(retries + 1):
# 重试退避睡眠在文件锁外执行——同名并发任务不再陪睡
# 重试退避等待在文件锁外执行——同名并发任务不再陪睡
if next_wait > 0:
time.sleep(next_wait)
await asyncio.sleep(next_wait)
next_wait = 0.0
# 重试轮次顶部的取消检查也要清理 .part(原实现在 try 外抛出会绕过清理)
if cancel_event is not None and cancel_event.is_set():
@@ -319,24 +319,24 @@ class MusicDownloader:
part_path = part_path0
# 同一目标文件的下载者串行化(不同文件之间仍然并行)
with get_file_lock(file_path):
async with get_file_lock(file_path):
try:
# 其它线程可能刚好完成此文件,锁内复查(命中即返回,不再下载)
cached = self._cached_result(file_path, music_info,
write_meta, meta_fields)
# 其它任务可能刚好完成此文件,锁内复查(命中即返回,不再下载)
cached = await self._cached_result(file_path, music_info,
write_meta, meta_fields)
if cached is not None:
return cached
# 上次已下完但未改名的 .part:直接转正,不发起网络请求
promoted = self._promote_complete_part(
promoted = await self._promote_complete_part(
part_path, file_path, music_info, write_meta, meta_fields)
if promoted is not None:
return promoted
# part_path 已在文件锁外绑定为 part_path0(同一路径)
self._transfer(music_info, part_path,
self._resume_offset(part_path, music_info),
cancel_event, progress_cb)
await self._transfer(music_info, part_path,
self._resume_offset(part_path, music_info),
cancel_event, progress_cb)
# 完整性校验(已知期望大小时必须精确相等;未知期望时至少非空)
mismatch = self._size_mismatch_error(part_path, music_info)
@@ -347,8 +347,8 @@ class MusicDownloader:
continue
# 完整性通过 → 嗅探 → 原子转正 → 写元数据
return self._publish(part_path, file_path, music_info,
cancel_event, write_meta, meta_fields)
return await self._publish(part_path, file_path, music_info,
cancel_event, write_meta, meta_fields)
except DownloadCanceled:
# 用户明确放弃:清理半截临时文件
@@ -363,7 +363,7 @@ class MusicDownloader:
last_error = "续传起点失效,重新下载"
next_wait = self._backoff_wait(attempt, retries)
continue
except (requests.RequestException, OSError) as e:
except (UpstreamError, OSError) as e:
# 网络错误:保留 .part 支持断点续传;
# 但若任务已被取消,.part 没有任何续传价值,一并清理
if cancel_event is not None and cancel_event.is_set():
@@ -385,7 +385,7 @@ class MusicDownloader:
# ----------- 下载分步实现(download_music_file 的组成件) -----------
def _cached_result(
async def _cached_result(
self,
file_path: Path,
music_info: MusicInfo,
@@ -406,9 +406,9 @@ class MusicDownloader:
if cache_bad:
self._unlink_quiet(file_path)
return None
# 缓存命中也尊重本次的元数据请求(重跑一次 ffmpeg,幂等)
# 缓存命中也尊重本次的元数据请求(重跑一次写标签,幂等)
if write_meta:
self._write_metadata_locked(file_path, music_info, meta_fields)
await self._write_metadata_locked(file_path, music_info, meta_fields)
return DownloadResult(
success=True,
file_path=str(file_path),
@@ -416,7 +416,7 @@ class MusicDownloader:
music_info=music_info,
)
def _promote_complete_part(
async def _promote_complete_part(
self,
part_path: Path,
file_path: Path,
@@ -437,7 +437,7 @@ class MusicDownloader:
return None
os.replace(part_path, file_path)
if write_meta:
self._write_metadata_locked(file_path, music_info, meta_fields)
await self._write_metadata_locked(file_path, music_info, meta_fields)
return DownloadResult(
success=True,
file_path=str(file_path),
@@ -460,7 +460,7 @@ class MusicDownloader:
return 0
return existing
def _transfer(
async def _transfer(
self,
music_info: MusicInfo,
part_path: Path,
@@ -474,18 +474,20 @@ class MusicDownloader:
"""
# 排队用可取消等待:原阻塞式 with 使取消后仍要等全局 10 名额腾出,
# 任务卡死、退款延迟
if not self._download_semaphore.acquire_cancellable(cancel_event, poll=0.5):
if not await self._download_semaphore.acquire_cancellable(cancel_event, poll=0.5):
raise DownloadCanceled("下载已取消")
try:
aria2_backend = get_backend()
# get_backend 首次调用要拉起 aria2c 并等 RPC 就绪(最长 8s),
# 经 to_thread 卸载,不在事件循环上原地等
aria2_backend = await asyncio.to_thread(get_backend)
if aria2_backend is not None:
# aria2c 通道:多连接 + 原生断点续传
self._fetch_via_aria2(aria2_backend, music_info, part_path,
cancel_event, progress_cb)
await self._fetch_via_aria2(aria2_backend, music_info, part_path,
cancel_event, progress_cb)
else:
# 内置通道:aiohttp 流式落盘 + Range 起点校验
self._fetch_via_http(music_info, part_path, resume_from,
cancel_event, progress_cb)
await self._fetch_via_http(music_info, part_path, resume_from,
cancel_event, progress_cb)
finally:
# 与 acquire_cancellable 配对释放(原 with 语义)
self._download_semaphore.release()
@@ -506,7 +508,7 @@ class MusicDownloader:
return (f"下载不完整: 期望 {music_info.file_size or '未知'} 字节, "
f"实际 {actual_size} 字节")
def _publish(
async def _publish(
self,
part_path: Path,
file_path: Path,
@@ -527,14 +529,14 @@ class MusicDownloader:
os.replace(part_path, file_path)
# 元数据写入移出下载信号量(CPU 操作不占网络名额),
# 元数据写入移出下载闸(CPU 操作不占网络名额),
# 但仍在文件锁内执行,保证其它请求看到的是已写入标签的最终文件
if write_meta:
# 元数据段入口补取消检查点——已取消的任务不再
# 进入信号量排队/ffmpeg 等待(有界但可达 120s),收尾不再延迟
# 进入标签闸排队/写盘等待(有界但可达数十秒),收尾不再延迟
if cancel_event is not None and cancel_event.is_set():
raise DownloadCanceled("下载已取消")
self._write_metadata_locked(file_path, music_info, meta_fields)
await self._write_metadata_locked(file_path, music_info, meta_fields)
return DownloadResult(
success=True,
@@ -543,26 +545,26 @@ class MusicDownloader:
music_info=music_info,
)
def _write_metadata_locked(
async def _write_metadata_locked(
self,
file_path: Path,
music_info: MusicInfo,
meta_fields: list[str] | None,
) -> None:
"""在标签信号量内写元数据:写标签会重写整个文件(IO 密集),与下载分离
"""在标签闸内写元数据:写标签会重写整个文件(IO 密集),与下载分离
限流——写满时只阻塞后来者,不占下载名额、不拖慢其它文件的传输。"""
with self._tag_semaphore:
self._write_metadata(file_path, music_info, meta_fields)
async with self._tag_semaphore:
await self._write_metadata(file_path, music_info, meta_fields)
@staticmethod
def _backoff_wait(attempt: int, retries: int) -> float:
"""失败后的退避秒数;已是最后一轮则不等待。锁外睡眠——同名并发任务不陪睡。"""
"""失败后的退避秒数;已是最后一轮则不等待。锁外等待——同名并发任务不陪睡。"""
return 0.5 * (attempt + 1) if attempt < retries else 0.0
# ----------- 传输通道(二选一,产物都是 part_path) -----------
def _fetch_via_http(
async def _fetch_via_http(
self,
music_info: MusicInfo,
part_path: Path,
@@ -570,7 +572,7 @@ class MusicDownloader:
cancel_event: threading.Event | None,
progress_cb,
) -> None:
"""内置流式通道:core.sync_http 逐块落盘,支持 Range 断点续传。
"""内置流式通道:core.http_client 逐块落盘,支持 Range 断点续传。
四步:构造 Range 头 → 发请求 → 定追加/重写并校验分片起点 → 流式落盘
(含体积上限与取消)。HTTP 错误在此分类成「确定性失败」或交外层重试。
@@ -579,23 +581,22 @@ class MusicDownloader:
DownloadCanceled: 取消事件置位
DownloadException: 体积超限、CDN 4xx 等确定性失败
_RetryFromScratch: 续传起点失效(残片已清,外层从头重下)
UpstreamError: 网络/5xx 等可重试失败(保留残片续传)
"""
expected_size = music_info.file_size or 0
self._safe_progress(progress_cb, resume_from, expected_size)
resp: requests.SyncResponse | None = None
resp = None
try:
resp = get_http_session().get(
music_info.download_url,
stream=True,
timeout=(10, 60),
headers=self._range_headers(resume_from),
)
headers = self._range_headers(resume_from)
resp = await _http.open('GET', music_info.download_url,
headers=headers or None, timeout=60)
mode = self._resolve_write_mode(resp, resume_from, part_path)
resp.raise_for_status()
self._stream_body(resp, part_path, mode, resume_from,
expected_size, cancel_event, progress_cb)
except requests.HTTPError as e:
status = self._http_status(e)
if resp.status >= 400:
raise UpstreamHTTPError(resp.status, resp.reason or '')
await self._stream_body(resp, part_path, mode, resume_from,
expected_size, cancel_event, progress_cb)
except UpstreamHTTPError as e:
status = e.status
if status == 416:
# Range 起点无效(如 CDN 环节变化):清除旧 part,从头重新下载
self._unlink_quiet(part_path)
@@ -622,7 +623,7 @@ class MusicDownloader:
服务器忽略 Range 返回 200 时若继续追加,会把整段拼在残片后面;206 还须
校验分片起点(见 _verify_content_range)。
"""
if resume_from <= 0 or resp.status_code != 206:
if resume_from <= 0 or resp.status != 206:
return "wb"
self._verify_content_range(resp, resume_from, part_path)
return "ab"
@@ -643,7 +644,7 @@ class MusicDownloader:
raise _RetryFromScratch(
f"Content-Range 起点错位({m.group(1)} != {resume_from}),已从头重新下载")
def _stream_body(
async def _stream_body(
self,
resp,
part_path: Path,
@@ -662,7 +663,7 @@ class MusicDownloader:
written = resume_from if mode == "ab" else 0
over_limit = False
with open(part_path, mode) as f:
for chunk in resp.iter_content(chunk_size=DL_CHUNK_SIZE):
async for chunk in iter_chunks(resp, DL_CHUNK_SIZE):
if cancel_event is not None and cancel_event.is_set():
raise DownloadCanceled("下载已取消")
if not chunk:
@@ -688,11 +689,6 @@ class MusicDownloader:
except Exception:
pass
@staticmethod
def _http_status(exc: BaseException) -> int | None:
"""从 HTTPError 上取响应状态码(无响应对象时返回 None)。"""
return getattr(getattr(exc, "response", None), "status_code", None)
@staticmethod
def _is_deterministic_client_error(status: int | None) -> bool:
"""确定性 4xx(403 防盗链 / 429 风控等):对同一失效直链重试只会加剧风控、
@@ -700,7 +696,7 @@ class MusicDownloader:
"""
return status is not None and 400 <= status < 500 and status != 408
def _fetch_via_aria2(
async def _fetch_via_aria2(
self,
backend: Aria2Backend,
music_info: MusicInfo,
@@ -711,10 +707,10 @@ class MusicDownloader:
"""aria2c 通道:多连接加速 + 原生断点续传。
异常语义与内置通道对齐:取消抛 DownloadCanceled、体积超限抛
DownloadException(均清残片),网络类失败抛 RequestException
DownloadException(均清残片),网络类失败抛 UpstreamNetworkError
(保留残片,由外层退避重试续传)。
"""
backend.fetch(
await backend.fetch(
music_info.download_url,
part_path,
cancel_event=cancel_event,
@@ -728,7 +724,7 @@ class MusicDownloader:
# 三个方法已删除——全服务端与验证脚本零调用(CLI 遗留),且 to_memory 不接受
# cookies 参数、真被启用会有行为差异;如需内存下载/异步批量再按当前接口重写
def _write_metadata(
async def _write_metadata(
self,
file_path: Path,
music_info: MusicInfo,
@@ -743,11 +739,13 @@ class MusicDownloader:
封面走共享缓存(与 ZIP 打包共用一次网络请求);各容器的落盘细节见
tagging.py——按扩展名分派到 ID3v2.3 / Vorbis comment / iTunes ilst。
写盘是阻塞 IO + 纯 Python CPU(mutagen 全量重写文件),经 to_thread 卸载。
"""
include_cover = fields is None or "cover" in fields
cover: bytes | None = None
if include_cover and music_info.pic_url:
cover = fetch_cover_bytes(music_info.pic_url)
cover = await fetch_cover_bytes(music_info.pic_url)
if not cover:
logger.warning("封面下载失败,跳过封面写入")
return write_tags(file_path, build_tags(music_info, fields), cover)
return await asyncio.to_thread(
write_tags, file_path, build_tags(music_info, fields), cover)