Archived
refactor: aria2 改手写原生异步 JSON-RPC，aria2p 与同步 HTTP 栈从依赖树清零
为什么：aria2p 内部是 requests + websocket-client 的同步实现，每个 RPC 都是一次 「同步网络 IO 藏在 to_thread 里」——线程池跑得再满也不产生真正的并发，每次轮询 还要额外付一次线程跳转。这是本项目最后一处此类残留，删掉后整个进程的网络 IO 都走同一个 aiohttp 连接池。 改动： - music_downloader/_aria2.py 整体重写：新增 _Aria2Rpc，直接向 aria2 的 JSON-RPC 2.0 端点发 getVersion / addUri / tellStatus / forceRemove / removeDownloadResult，一律走 core.http_client.request_once（不重试，与 aria2p 口径一致）；新增 Aria2RpcError 承载 HTTP 200 + error 对象的协议错误。 - _abort 逐条复刻 aria2p.remove(force=True, files=?, clean=True) 语义：终态走 removeDownloadResult；在途先 forceRemove 再清结果表；控制文件 .aria2 无条件 删除（clean=True 默认）；仅当 files 且主移除成功才删数据文件。 - get_backend / _try_start 改原生异步：asyncio.Lock 单飞 + await rpc.get_version() 探活，不再用 to_thread 包 subprocess.Popen；超时路径改 to_thread(_reap) 回收。 - 新增 Protocol _Rpc / _ProcessHandle（补 returncode 声明）——只声明本模块真正 调用的方法，测试据此用轻量替身驱动状态机，不必依赖真实 aria2 进程。 - pyproject.toml 删 aria2p 依赖并同步更新过时注释；uv lock 连带移除 8 个传递 依赖：requests / urllib3 / websocket-client / charset-normalizer / loguru / platformdirs / win32-setctime。至此全仓无任何 requests 栈。 - tests/test_aria2.py 重写为 21 项用例（_FakeRpc / _FakeProc / _LiveProc / _FakeHTTP 替身），新增终态体积守卫、_try_start 三分支（就绪重试 / 超时不就绪 杀进程 / 进程提前退出）与 JSON-RPC 信封四例。 - tests/test_downloader.py 的 get_backend 替身由同步 lambda 改为协程——调用点 已直接 await，同步替身会被 await 成 None 而报 TypeError。 验证（在 aria2p 已物理移除的环境上复核，故能真正证明无残留依赖）： - uv run ruff check . → All checks passed! - uv run pyright → 0 errors, 0 warnings, 0 informations - uv run pytest → 392 项，0 失败 / 0 error / 0 skipped（基线 382，净增 10） - data/site.db sha256 前后一致（a19f5dbf…6487），无 -wal/-shm 残留
This commit is contained in:
snowy-yang committed
1 parent
6c7f8f589e
commit
046225e38d
6 files changed
+516
-338
No files matched your search
+180
-45
@@ -12,15 +12,18 @@
|
||||
由调用方回落到内置的 core.http_client 流式通道；
|
||||
- aria2 没有「单文件体积上限」选项，故用轮询 tellStatus 实现体积守卫与取消响应。
|
||||
|
||||
异步口径：aria2p 客户端本身是同步阻塞实现（内部走 urllib），故每次 RPC 调用
|
||||
（提交/查询/移除）都经 ``asyncio.to_thread`` 卸载；轮询间隔改 ``asyncio.sleep``
|
||||
让出事件循环。进程启动（``_try_start``）最长阻塞 8s，调用方同样经 to_thread 触发。
|
||||
异步口径：RPC 由本模块直接打 aria2 的 JSON-RPC（HTTP POST + JSON-RPC 2.0），
|
||||
全程 await——原先依赖的 aria2p 内部是 requests + websocket 的同步实现，每次
|
||||
RPC 都要经 asyncio.to_thread 卸载，是本项目最后一处「把同步 HTTP 藏在 to_thread
|
||||
里」的地方。轮询间隔用 asyncio.sleep 让出事件循环。剩下的 to_thread 只有两处，
|
||||
都是真正的阻塞卸载：进程创建（subprocess.Popen）与强杀回收（kill + wait）。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import atexit
|
||||
import contextlib
|
||||
import secrets
|
||||
import shutil
|
||||
import socket
|
||||
@@ -32,6 +35,7 @@ from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from core.http_client import UpstreamNetworkError
|
||||
from core.http_client import client as _http
|
||||
|
||||
from ._http import DEFAULT_UA
|
||||
from ._log import logger
|
||||
@@ -45,6 +49,8 @@ class _ProcessHandle(Protocol):
|
||||
也让测试可以用轻量替身驱动生命周期逻辑。
|
||||
"""
|
||||
|
||||
returncode: int | None
|
||||
|
||||
def poll(self) -> int | None: ...
|
||||
def terminate(self) -> None: ...
|
||||
def kill(self) -> None: ...
|
||||
@@ -64,11 +70,30 @@ _DEFAULT_MAX_BYTES = 500 * 1024 * 1024
|
||||
_SPLIT = 5
|
||||
_MIN_SPLIT_SIZE = '1M'
|
||||
|
||||
_lock = threading.Lock()
|
||||
# 单次 RPC 超时：本机回环，正常在毫秒级；给足余量以免大库时误判网络失败
|
||||
_RPC_TIMEOUT = 10.0
|
||||
|
||||
# tellStatus 返回的**终态**：任务已不在下载队列，只能清结果表
|
||||
_TERMINAL_STATUSES = frozenset({'complete', 'error', 'removed'})
|
||||
|
||||
_backend: Aria2Backend | None = None
|
||||
_unavailable = False # 启动失败后置位，不再反复重试
|
||||
|
||||
|
||||
# 启动单飞锁按事件循环隔离：asyncio.Lock 绑定创建它的 loop，应用主 loop 与测试
|
||||
# 各自的 loop 并存，跨 loop 复用会抛 "bound to a different event loop"。
|
||||
_START_LOCKS: dict[int, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def _start_lock() -> asyncio.Lock:
|
||||
key = id(asyncio.get_running_loop())
|
||||
lock = _START_LOCKS.get(key)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
_START_LOCKS[key] = lock
|
||||
return lock
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
"""向内核要一个空闲端口（bind 后立即释放，交给 aria2c 监听）。"""
|
||||
with socket.socket() as s:
|
||||
@@ -76,11 +101,86 @@ def _free_port() -> int:
|
||||
return int(s.getsockname()[1])
|
||||
|
||||
|
||||
class Aria2RpcError(RuntimeError):
|
||||
"""aria2 在 JSON-RPC 响应体里回报的 error 对象（协议层失败，非 HTTP 层）。"""
|
||||
|
||||
|
||||
class _Aria2Rpc:
|
||||
"""aria2 JSON-RPC 极简客户端（只覆盖本模块用到的 5 个方法）。
|
||||
|
||||
替代 aria2p：后者内部是 requests + websocket 的同步实现，每个 RPC 都得
|
||||
to_thread 卸载；而 aria2 的 RPC 就是朴素的 HTTP POST + JSON-RPC 2.0，
|
||||
直接复用 core.http_client 的进程级连接池即可，且连接池本就为异步而生。
|
||||
|
||||
一律走 ``request_once``（**不重试**），与 aria2p 口径一致：addUri /
|
||||
forceRemove 这类变更调用若被重发可能重复提交，RPC 抖动由上层 ``fetch``
|
||||
收敛成 UpstreamNetworkError、交给下载器的退避重试统一处理。
|
||||
"""
|
||||
|
||||
def __init__(self, port: int, secret: str) -> None:
|
||||
self._url = f'http://127.0.0.1:{port}/jsonrpc'
|
||||
self._token = f'token:{secret}'
|
||||
self._seq = 0
|
||||
|
||||
async def _call(self, method: str, *params: Any) -> Any:
|
||||
self._seq += 1
|
||||
body = {
|
||||
'jsonrpc': '2.0',
|
||||
'id': str(self._seq),
|
||||
'method': f'aria2.{method}',
|
||||
'params': [self._token, *params],
|
||||
}
|
||||
resp = await _http.request_once(
|
||||
'POST', self._url, json_body=body, timeout=_RPC_TIMEOUT)
|
||||
data = resp.json()
|
||||
if not isinstance(data, dict):
|
||||
raise Aria2RpcError(f'aria2 RPC 响应不是 JSON 对象: {resp.text[:200]!r}')
|
||||
err = data.get('error')
|
||||
if err:
|
||||
raise Aria2RpcError(f'aria2 RPC {method} 失败: {err}')
|
||||
return data.get('result')
|
||||
|
||||
async def get_version(self) -> str:
|
||||
"""探活（启动时就绪判定）。"""
|
||||
return str(await self._call('getVersion'))
|
||||
|
||||
async def add_uri(self, url: str, options: dict[str, str]) -> str:
|
||||
"""提交下载，返回 GID。"""
|
||||
return str(await self._call('addUri', [url], options))
|
||||
|
||||
async def tell_status(self, gid: str) -> dict[str, Any]:
|
||||
"""查询任务状态（不传 keys，取全量字段）。"""
|
||||
res = await self._call('tellStatus', gid)
|
||||
return res if isinstance(res, dict) else {}
|
||||
|
||||
async def force_remove(self, gid: str) -> str:
|
||||
"""停掉在途任务（不做联系 tracker 等耗时动作）。"""
|
||||
return str(await self._call('forceRemove', gid))
|
||||
|
||||
async def remove_download_result(self, gid: str) -> str:
|
||||
"""清掉已终结任务在结果表里的条目（否则结果表只增不减）。"""
|
||||
return str(await self._call('removeDownloadResult', gid))
|
||||
|
||||
|
||||
class _Rpc(Protocol):
|
||||
"""aria2 RPC 的最小接口。
|
||||
|
||||
与 _ProcessHandle 同理：只声明本模块真正调用的方法，既不绑死具体实现，
|
||||
也让测试能用轻量替身驱动整套状态机（不必起 HTTP 假服务器）。
|
||||
"""
|
||||
|
||||
async def get_version(self) -> str: ...
|
||||
async def add_uri(self, url: str, options: dict[str, str]) -> str: ...
|
||||
async def tell_status(self, gid: str) -> dict[str, Any]: ...
|
||||
async def force_remove(self, gid: str) -> str: ...
|
||||
async def remove_download_result(self, gid: str) -> str: ...
|
||||
|
||||
|
||||
class Aria2Backend:
|
||||
"""aria2c RPC 后端。请通过 get_backend() 获取单例，不要直接构造。"""
|
||||
|
||||
def __init__(self, api: Any, proc: _ProcessHandle, port: int) -> None:
|
||||
self._api = api
|
||||
def __init__(self, rpc: _Rpc, proc: _ProcessHandle, port: int) -> None:
|
||||
self._rpc = rpc
|
||||
self._proc = proc
|
||||
self.port = port
|
||||
self._closed = False
|
||||
@@ -88,14 +188,15 @@ class Aria2Backend:
|
||||
# ---------- 生命周期 ----------
|
||||
|
||||
def close(self) -> None:
|
||||
"""关闭 RPC 客户端并终止 aria2c 进程（幂等）。"""
|
||||
"""终止 aria2c 进程（幂等）。
|
||||
|
||||
**同步**：由 atexit 调用，那时已经没有事件循环，且进程回收本就是阻塞操作。
|
||||
这里不关 RPC 连接——它走 core.http_client 的**进程级共享连接池**，关掉会
|
||||
连带影响内置流式通道（原先 aria2p 关的是它自己私有的 requests 会话）。
|
||||
"""
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
try:
|
||||
self._api.client.close()
|
||||
except Exception:
|
||||
pass # 客户端关闭失败不影响后续进程回收
|
||||
if self._proc.poll() is None:
|
||||
self._proc.terminate()
|
||||
try:
|
||||
@@ -142,52 +243,79 @@ class Aria2Backend:
|
||||
options['referer'] = referer
|
||||
|
||||
try:
|
||||
download = await asyncio.to_thread(self._api.add_uris, [url], options=options)
|
||||
gid = await self._rpc.add_uri(url, options)
|
||||
except Exception as e:
|
||||
raise UpstreamNetworkError(f'aria2 提交任务失败: {e}') from e
|
||||
|
||||
deadline = time.monotonic() + timeout
|
||||
status = ''
|
||||
while True:
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
await asyncio.to_thread(self._abort, download, drop_files=True)
|
||||
await self._abort(gid, target, drop_files=True, status=status)
|
||||
raise DownloadCanceled('下载已取消')
|
||||
try:
|
||||
await asyncio.to_thread(download.update)
|
||||
st = await self._rpc.tell_status(gid)
|
||||
except Exception as e:
|
||||
# RPC 抖动按可重试的网络失败上报，交给外层退避重试
|
||||
raise UpstreamNetworkError(f'aria2 状态查询失败: {e}') from e
|
||||
|
||||
done = int(download.completed_length or 0)
|
||||
total = int(download.total_length or 0)
|
||||
status = str(st.get('status') or '')
|
||||
done = int(st.get('completedLength') or 0)
|
||||
total = int(st.get('totalLength') or 0)
|
||||
if progress_cb is not None:
|
||||
try:
|
||||
progress_cb(done, total)
|
||||
except Exception:
|
||||
pass # 进度回调绝不影响下载主流程
|
||||
if done > max_bytes:
|
||||
await asyncio.to_thread(self._abort, download, drop_files=True)
|
||||
await self._abort(gid, target, drop_files=True, status=status)
|
||||
raise DownloadException(
|
||||
f'文件超过大小上限 {max_bytes // (1024 * 1024)}MB，已中止')
|
||||
if download.is_complete:
|
||||
if status == 'complete':
|
||||
self._drop_control_file(target)
|
||||
return
|
||||
if download.has_failed:
|
||||
# 保留残片与控制文件，等待外层重试时续传
|
||||
if status == 'error':
|
||||
# 保留残片与控制文件，等待外层重试时续传（**不调 _abort**）
|
||||
raise UpstreamNetworkError(
|
||||
f'aria2 下载失败: {download.error_message or download.status}')
|
||||
f'aria2 下载失败: {st.get("errorMessage") or status}')
|
||||
if time.monotonic() > deadline:
|
||||
await asyncio.to_thread(self._abort, download, drop_files=False)
|
||||
await self._abort(gid, target, drop_files=False, status=status)
|
||||
raise UpstreamNetworkError('aria2 下载超时')
|
||||
await asyncio.sleep(_POLL_INTERVAL)
|
||||
|
||||
# ---------- 内部 ----------
|
||||
|
||||
def _abort(self, download: Any, *, drop_files: bool) -> None:
|
||||
"""强制移除任务。drop_files=True 时连同已下载字节一起删除。"""
|
||||
async def _abort(
|
||||
self, gid: str, target: Path, *, drop_files: bool, status: str,
|
||||
) -> None:
|
||||
"""移除任务。与 ``aria2p.remove(force=True, files=drop_files, clean=True)``
|
||||
逐条对齐（该库的默认行为就是还要清控制文件）：
|
||||
|
||||
- 已终结（complete/error/removed）→ 只能清结果表（removeDownloadResult）；
|
||||
在途 → forceRemove 先停掉，再顺手清结果表；
|
||||
- 控制文件（target.aria2）**总是**删除——这是 aria2p 的 clean=True 默认值，
|
||||
也是「取消 / 体积超限」后下轮必须重新下载的依据；
|
||||
- 只有 drop_files 才删已落地的数据文件（超时场景保留残片供续传）。
|
||||
任一步失败只告警、不影响主流程（与 aria2p 的 logger.exception 同口径），
|
||||
且与 aria2p 一致：主移除动作失败时**不**删数据文件。
|
||||
"""
|
||||
removed = True
|
||||
try:
|
||||
self._api.remove([download], force=True, files=drop_files)
|
||||
if status in _TERMINAL_STATUSES:
|
||||
await self._rpc.remove_download_result(gid)
|
||||
else:
|
||||
await self._rpc.force_remove(gid)
|
||||
with contextlib.suppress(Exception):
|
||||
await self._rpc.remove_download_result(gid)
|
||||
except Exception as e:
|
||||
removed = False
|
||||
logger.warning(f'终止 aria2 任务失败（不影响主流程）: {e}')
|
||||
self._drop_control_file(target)
|
||||
if drop_files and removed:
|
||||
try:
|
||||
target.unlink(missing_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning(f'删除 aria2 残片失败（不影响主流程）: {e}')
|
||||
|
||||
@staticmethod
|
||||
def _drop_control_file(target: Path) -> None:
|
||||
@@ -199,39 +327,45 @@ class Aria2Backend:
|
||||
pass
|
||||
|
||||
|
||||
def get_backend() -> Aria2Backend | None:
|
||||
async def get_backend() -> Aria2Backend | None:
|
||||
"""返回 aria2 后端单例；aria2c 不可用/启动失败时返回 None（调用方降级）。
|
||||
|
||||
首次调用尝试拉起常驻进程（最长阻塞 _RPC_READY_TIMEOUT 秒），之后命中缓存；
|
||||
失败结果进程内记忆，不反复重试。**同步阻塞**——异步调用方须经
|
||||
``asyncio.to_thread(get_backend)`` 触发，避免首次启动期间冻住事件循环。
|
||||
首次调用尝试拉起常驻进程并等 RPC 就绪（最长 ``_RPC_READY_TIMEOUT`` 秒，
|
||||
等待用 ``await asyncio.sleep`` 让出事件循环，故本函数是**原生异步**、
|
||||
不需要调用方再包 to_thread）；之后命中缓存。失败结果进程内记忆，
|
||||
不反复重试。并发首调由启动单飞锁收敛，只拉起一个进程。
|
||||
"""
|
||||
global _backend, _unavailable
|
||||
if _unavailable:
|
||||
return None
|
||||
if _backend is not None:
|
||||
return _backend
|
||||
with _lock:
|
||||
async with _start_lock():
|
||||
if _unavailable:
|
||||
return None
|
||||
if _backend is None:
|
||||
_backend = _try_start()
|
||||
_backend = await _try_start()
|
||||
if _backend is None:
|
||||
_unavailable = True
|
||||
return _backend
|
||||
|
||||
|
||||
def _try_start() -> Aria2Backend | None:
|
||||
def _reap(proc: _ProcessHandle) -> None:
|
||||
"""强杀并回收子进程（阻塞）。只应由 asyncio.to_thread 调用。"""
|
||||
try:
|
||||
proc.kill()
|
||||
except OSError:
|
||||
return
|
||||
with contextlib.suppress(Exception):
|
||||
proc.wait(timeout=5)
|
||||
|
||||
|
||||
async def _try_start() -> Aria2Backend | None:
|
||||
"""拉起 aria2c --enable-rpc 并等待 RPC 就绪；任何失败都返回 None。"""
|
||||
exe = shutil.which(ARIA2_BIN)
|
||||
if not exe:
|
||||
logger.info('未检测到 aria2c，下载走内置流式通道')
|
||||
return None
|
||||
try:
|
||||
import aria2p
|
||||
except ImportError as e:
|
||||
logger.info(f'aria2p 不可用，下载走内置流式通道: {e}')
|
||||
return None
|
||||
|
||||
port = _free_port()
|
||||
secret = secrets.token_urlsafe(24)
|
||||
@@ -246,33 +380,34 @@ def _try_start() -> Aria2Backend | None:
|
||||
'--quiet=true',
|
||||
]
|
||||
try:
|
||||
# 进程创建是真阻塞调用（Windows CreateProcess 数十毫秒），
|
||||
# 且本函数在事件循环内被 await，故经 to_thread 卸载（真正的阻塞卸载）。
|
||||
# exe 来自 shutil.which，参数全部由本模块构造，不经任何用户输入
|
||||
proc = subprocess.Popen(
|
||||
cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
proc = await asyncio.to_thread(
|
||||
subprocess.Popen, cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
except OSError as e:
|
||||
logger.warning(f'aria2c 启动失败，降级内置下载: {e}')
|
||||
return None
|
||||
|
||||
client = aria2p.Client(host='http://127.0.0.1', port=port, secret=secret)
|
||||
rpc = _Aria2Rpc(port, secret)
|
||||
deadline = time.monotonic() + _RPC_READY_TIMEOUT
|
||||
while True:
|
||||
if proc.poll() is not None:
|
||||
logger.warning(f'aria2c 进程提前退出（code={proc.returncode}），降级内置下载')
|
||||
return None
|
||||
try:
|
||||
client.get_version()
|
||||
await rpc.get_version()
|
||||
break
|
||||
except Exception:
|
||||
if time.monotonic() > deadline:
|
||||
logger.warning(f'aria2c RPC {_RPC_READY_TIMEOUT:.0f}s 内未就绪，降级内置下载')
|
||||
proc.kill()
|
||||
proc.wait(timeout=5)
|
||||
await asyncio.to_thread(_reap, proc)
|
||||
return None
|
||||
time.sleep(0.2)
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
atexit.register(_shutdown)
|
||||
logger.info(f'aria2c 常驻 RPC 已就绪（127.0.0.1:{port}，{_SPLIT} 连接/文件）')
|
||||
return Aria2Backend(aria2p.API(client), proc, port)
|
||||
return Aria2Backend(rpc, proc, port)
|
||||
|
||||
|
||||
def _shutdown() -> None:
|
||||
|
||||
Reference in new issue
Block a user