refactor: worker 子系统全面异步化,拆掉同步转异步的桥接层

拆掉「同步调异步 / 异步桥同步」两类垫片的第二半(HTTP 线 B2 见前序提交)。
worker 侧此前是「起线程跑同步函数、函数体内部再用 asyncio.to_thread 把活塞给
另一条循环」——线程与协程互等,取消/暂停只能靠轮询。本轮把 worker 自身变成
协程,由 _start_task 直接挂到应用事件循环上,桥接层随之失去存在理由。

改动(按子系统):
- music_downloader:downloader / _aria2 / _concurrency / cover / _http /
  __main__ 全链路协程化。per-file 锁改 asyncio.Lock;计数闸改
  「active 计数 + await asyncio.sleep(0.5)」的可取消等待(原阻塞式 with 会让
  已取消任务仍要等全局名额腾出,卡死并拖后腿退款);aria2p 的同步 RPC 仍经
  to_thread,轮询改 asyncio.sleep;新增 core.http_client.iter_chunks,把**流式
  读取期**的 aiohttp.ClientError 收敛成 UpstreamNetworkError(请求建立阶段早已
  有这条路径,读取阶段此前没有,断流会绕过调用方的 UpstreamError 分类)。
  删除 get_http_session / _SESSION_LOCAL。
  models._RetryFromScratch 不再继承 requests.RequestException 而继承 Exception:
  它是「续传起点失效、残片已清、从头重下」,既不是确定性失败
  (DownloadException),也不是残片已保留的网络失败(UpstreamError),必须独立
  成类且在 except 链中先于二者捕获。
- music_api:podcast.py 全部函数协程化(直链解析内部改
  await NeteaseAPI.get_song_url);compat.py 删除——16 个死掉的同步门面早已
  清空,文件只剩空壳。
- services:_taskcore / _zipcore / batch / downloads / podcast / tasks 协程化。
  threading.Thread → loop.create_task(任务句柄字段 worker_thread →
  worker_task),ThreadPoolExecutor → asyncio.gather + asyncio.Semaphore,
  终态函数与结算回调全部改 await;监控线程里「worker 已死」判定改 task.done()。
  删除 services/streaming.py(同步流垫片,唯一消费者已迁到
  core.http_client.open)。zipfile 实例仍非线程安全,故「下载」并行、
  「入包」严格串行。
- web:路由层 130+ 处 to_thread 拆除——被包住的目标自己已是协程,再包一层只是
  把协程丢进线程里跑一圈。tasks_registry.start_task 直接 create_task。
  web/app.py 关闭段去掉 async_bridge.shutdown()(不再有第二条约循环)。
  web/routers/download.py 的播客端点补上 3 处漏掉的 await
  (extract_podcast_ref / get_radio_detail)——漏掉时不报错、只静默返回协程对象,
  靠 pyright 抓出来后修正。

留在 to_thread 上的都是**正当的阻塞卸载**:同步 SQLite 落库与账本结算、
mutagen 全量重写标签、zipfile 整段拷贝、shutil.rmtree、aria2p 的同步 RPC、
PBKDF2 的 10 万次迭代。ruff 对残留的本地元数据操作(exists/stat/os.replace/
往 .part 顺序写分块)按「微秒级、卸载的调度开销大于操作本身」放行,pyproject
逐条注明理由。

取消/暂停仍由 threading.Event 承载:置位方是请求线程/监控线程,读取方是 worker
协程,只有「置位」与「轮询读取」两种操作,既不需要跨循环唤醒也不会卡住事件
循环——比 asyncio.Event 更省心(ASYNC110 同理按此豁免)。

测试:10 个文件的替身边界同步迁移——网络替身从 core.sync_http 的
get_http_session 换成 downloader/服务模块的 _http.open + **真实** iter_chunks
(顺带钉住读取期异常收敛);`_run` 改为在独立线程里 asyncio.run(worker(...));
aria2 的假时钟从 time.sleep 换成 _POLL_INTERVAL=0;monitor 的线程句柄替身换成
done() 可控的轻量替身(asyncio.Task 的构造需要运行中的事件循环,而监控循环跑在
常驻线程里);podcast / 封面 / 结算包装等替身一律改 async。

门禁:ruff 全过 / pyright 0-0 / pytest 383 项全绿(0 失败)。

下一步 B2:auth_system 12 个模块改 aiosqlite 背板、web 层剩余 to_thread 收尾;
B3:删 core/sync_http.py、core/async_bridge.py、core/http_errors.py(现仅
sync_http 自用)。
This commit is contained in:
snowy-yang committed 2026-09-28 19:36:56 +08:00
1 parent 65adfc9089
commit 69a50c2c31
35 files changed
+1062 -1005

No files matched your search

-4
View File
@@ -22,9 +22,6 @@ from .client import (
_as_dict,
_ensure_dict,
)
from .compat import (
url_v1,
)
from .constants import (
APIConstants,
QualityLevel,
@@ -59,5 +56,4 @@ __all__ = [
'_ensure_dict',
'load_cookies',
'parse_cookie_string',
'url_v1',
]
-25
View File
@@ -1,25 +0,0 @@
# -*- coding: utf-8 -*-
"""同步兼容函数接口(内部经 run_sync 桥接异步实现)。
(自 music_api.py 拆出,行为与原实现一致。)
⚠️ 迁移期残留:本模块的定位是「同步代码调用异步实现」的桥接门面。调用方陆续
改为原生异步(`await NeteaseAPI().xxx()`)后,这里的方法即成为死代码,应删除而
非新增。当前仅 `url_v1` 仍有唯一调用方(`music_api.podcast` 的同步解析路径),
故本文件只保留它——其余 16 个同步门面已随调用方异步化一并移除。
"""
from typing import Any
from core.async_bridge import run_sync # 模块级同步兼容接口用的同步/异步桥
from .api import (
NeteaseAPI,
)
# 向后兼容的函数接口
def url_v1(song_id: int, level: str, cookies: dict[str, str]) -> dict[str, Any]:
"""获取歌曲URL(向后兼容)"""
api = NeteaseAPI()
return run_sync(api.get_song_url(song_id, level, cookies))
+62 -69
View File
@@ -10,10 +10,12 @@
说明:
- 播客单集的音频本质上是网易云曲库中的 song(program.mainSong),
因此直链复用 music_api.url_v1(eapi song/enhance/player/url/v1)获取。
因此直链复用 NeteaseAPI.get_song_url(eapi song/enhance/player/url/v1)获取。
- 支持格式:mp3 / m4a / aac 等(以接口返回 type 为准)。
- 支持链接协议:music.163.com/djradio?id=(电台)、/program?id= 与 /sound?id=(单集)、
以及 163cn.tv 短链。
- 本模块**整体异步**:HTTP 统一走 core.http_client(进程级连接池 + 有界重试 +
8MB 响应体上限),调用方直接 await,不再需要同步门面/线程包装。
"""
import json
@@ -21,9 +23,10 @@ import re
import time
from typing import Any
from core import sync_http as requests # 同步门面(底层 aiohttp)
from core.http_client import UpstreamError
from core.http_client import client as _http
from .compat import url_v1
from .api import NeteaseAPI
from .constants import APIConstants
from .errors import APIException
@@ -51,26 +54,19 @@ class PodcastConstants:
DEFAULT_HOT_CATE_IDS = [6, 3, 8, 10001]
# 与 music_api 同口径的上游 JSON 响应体上限
_MAX_UPSTREAM_JSON_BYTES = 8 * 1024 * 1024
def _post_json(url: str, data: dict[str, Any], cookies: dict[str, str] | None = None) -> dict[str, Any]:
"""发送表单 POST 请求并返回 JSON"""
async def _post_json(url: str, data: dict[str, Any], cookies: dict[str, str] | None = None) -> dict[str, Any]:
"""发送表单 POST 请求并返回 JSON(响应体上限由 core.http_client 统一把关)"""
headers = {
'User-Agent': APIConstants.USER_AGENT,
'Referer': APIConstants.REFERER,
}
try:
response = requests.post(url, data=data, headers=headers,
cookies=cookies or {}, timeout=30)
response.raise_for_status()
if len(response.content) > _MAX_UPSTREAM_JSON_BYTES:
raise APIException('播客接口响应体异常(超过大小上限)')
response = await _http.request('POST', url, data=data, headers=headers,
cookies=cookies or {}, timeout=30)
result = response.json()
if not isinstance(result, dict):
raise APIException('播客接口响应异常(非标准 JSON 对象)')
except requests.RequestException as e:
except UpstreamError as e:
raise APIException(f"播客接口请求失败: {e}") from e
except json.JSONDecodeError as e:
raise APIException(f"解析播客接口响应失败: {e}") from e
@@ -180,7 +176,7 @@ def _norm_program(raw: dict[str, Any]) -> dict[str, Any]:
_SHORTLINK_CACHE: dict[str, str] = {}
def _resolve_short_link(url: str) -> str:
async def _resolve_short_link(url: str) -> str:
"""解析 163cn.tv 短链,返回跳转后的真实链接(仅接受整串短链,子串不触发请求)"""
if not re.match(r'^https?://(www\.)?163cn\.tv/\S+$', str(url or '').strip(), re.IGNORECASE):
return url
@@ -192,10 +188,10 @@ def _resolve_short_link(url: str) -> str:
return cached
resp = None
try:
resp = requests.get(url, allow_redirects=False, timeout=10,
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'})
resp = await _http.open('GET', url, allow_redirects=False, timeout=10,
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'})
real = resp.headers.get('Location', url)
except requests.RequestException:
except UpstreamError:
return url
finally:
if resp is not None:
@@ -210,7 +206,7 @@ def _resolve_short_link(url: str) -> str:
return real
def extract_podcast_ref(raw: str) -> dict[str, str]:
async def extract_podcast_ref(raw: str) -> dict[str, str]:
"""从用户输入中提取播客引用(正规 URL 解析,替代子串匹配与非数字剔除)
Returns:
@@ -223,7 +219,7 @@ def extract_podcast_ref(raw: str) -> dict[str, str]:
text = str(raw or '').strip()
# 短链先还原(仅当整串就是一个短链 URL)
if '163cn.tv' in text.lower():
text = _resolve_short_link(text)
text = await _resolve_short_link(text)
# 有 URL 结构:按路径判断类型,query 正规解析 id(参数顺序无关)
if '://' in text or text.startswith('//'):
@@ -256,9 +252,9 @@ def extract_podcast_ref(raw: str) -> dict[str, str]:
# ============ 核心 API ============
def search_podcast(keywords: str, cookies: dict[str, str] | None = None,
limit: int = 20, offset: int = 0,
search_type: str = 'all') -> dict[str, Any]:
async def search_podcast(keywords: str, cookies: dict[str, str] | None = None,
limit: int = 20, offset: int = 0,
search_type: str = 'all') -> dict[str, Any]:
"""搜索播客
Args:
@@ -278,18 +274,17 @@ def search_podcast(keywords: str, cookies: dict[str, str] | None = None,
result: dict[str, Any] = {'radios': [], 'voices': []}
def _do_search(stype: int) -> dict[str, Any]:
return _post_json(PodcastConstants.SEARCH_API,
{'s': keywords, 'type': stype, 'limit': limit, 'offset': offset},
cookies)
if search_type in ('all', 'radio'):
res = _do_search(1009)
res = await _post_json(PodcastConstants.SEARCH_API,
{'s': keywords, 'type': 1009, 'limit': limit, 'offset': offset},
cookies)
radios = (res.get('result') or {}).get('djRadios') or []
result['radios'] = [_norm_radio(r) for r in radios]
if search_type in ('all', 'voice'):
res = _do_search(2000)
res = await _post_json(PodcastConstants.SEARCH_API,
{'s': keywords, 'type': 2000, 'limit': limit, 'offset': offset},
cookies)
rdata = res.get('result') or {}
voices = rdata.get('songs') or rdata.get('voices') or rdata.get('programs') or []
result['voices'] = [_norm_program(v) for v in voices]
@@ -297,13 +292,13 @@ def search_podcast(keywords: str, cookies: dict[str, str] | None = None,
return result
def get_radio_detail(radio_id: Any, cookies: dict[str, str] | None = None) -> dict[str, Any]:
async def get_radio_detail(radio_id: Any, cookies: dict[str, str] | None = None) -> dict[str, Any]:
"""获取播客电台(djradio)详情"""
try:
rid = int(str(radio_id).strip())
except (TypeError, ValueError):
raise APIException('无效的电台ID') from None
result = _post_json(PodcastConstants.DJRADIO_GET_API, {'id': rid}, cookies)
result = await _post_json(PodcastConstants.DJRADIO_GET_API, {'id': rid}, cookies)
# 正常返回键为 djRadio;不存在时 code=404(_post_json 已抛错)或 data 为空
data = result.get('djRadio') or result.get('data') or {}
if not data:
@@ -311,8 +306,8 @@ def get_radio_detail(radio_id: Any, cookies: dict[str, str] | None = None) -> di
return _norm_radio(data)
def get_radio_programs(radio_id: Any, cookies: dict[str, str] | None = None,
limit: int = 30, offset: int = 0, asc: bool = True) -> dict[str, Any]:
async def get_radio_programs(radio_id: Any, cookies: dict[str, str] | None = None,
limit: int = 30, offset: int = 0, asc: bool = True) -> dict[str, Any]:
"""获取电台下的单集列表(分页)
Args:
@@ -329,10 +324,10 @@ def get_radio_programs(radio_id: Any, cookies: dict[str, str] | None = None,
except (TypeError, ValueError):
raise APIException('无效的电台ID') from None
limit = max(1, min(int(limit or 30), 100))
result = _post_json(PodcastConstants.DJ_PROGRAM_BYRADIO_API,
{'radioId': rid, 'limit': limit, 'offset': max(0, int(offset or 0)),
'asc': 'true' if asc else 'false'},
cookies)
result = await _post_json(PodcastConstants.DJ_PROGRAM_BYRADIO_API,
{'radioId': rid, 'limit': limit, 'offset': max(0, int(offset or 0)),
'asc': 'true' if asc else 'false'},
cookies)
programs = [_norm_program(p) for p in (result.get('programs') or [])]
count = _to_int(result.get('count'))
offset_val = max(0, int(offset or 0))
@@ -343,20 +338,20 @@ def get_radio_programs(radio_id: Any, cookies: dict[str, str] | None = None,
}
def get_program_detail(program_id: Any, cookies: dict[str, str] | None = None) -> dict[str, Any]:
async def get_program_detail(program_id: Any, cookies: dict[str, str] | None = None) -> dict[str, Any]:
"""获取单集(program)详情"""
try:
pid = int(str(program_id).strip())
except (TypeError, ValueError):
raise APIException('无效的单集ID') from None
result = _post_json(PodcastConstants.DJ_PROGRAM_DETAIL_API, {'id': pid}, cookies)
result = await _post_json(PodcastConstants.DJ_PROGRAM_DETAIL_API, {'id': pid}, cookies)
program = result.get('program') or {}
if not program:
raise APIException('未找到该播客单集(可能已下架或ID错误)')
return _norm_program(program)
def get_categories() -> list[dict[str, Any]]:
async def get_categories() -> list[dict[str, Any]]:
"""获取播客分类列表(GET 接口)
Returns:
@@ -367,13 +362,10 @@ def get_categories() -> list[dict[str, Any]]:
'Referer': APIConstants.REFERER,
}
try:
response = requests.get(PodcastConstants.DJRADIO_CATEGORY_API,
headers=headers, timeout=30)
response.raise_for_status()
if len(response.content) > _MAX_UPSTREAM_JSON_BYTES:
raise APIException('播客接口响应体异常(超过大小上限)')
response = await _http.request('GET', PodcastConstants.DJRADIO_CATEGORY_API,
headers=headers, timeout=30)
result = response.json()
except requests.RequestException as e:
except UpstreamError as e:
raise APIException(f"获取播客分类失败: {e}") from e
except json.JSONDecodeError as e:
raise APIException(f"解析播客分类响应失败: {e}") from e
@@ -396,9 +388,9 @@ def get_categories() -> list[dict[str, Any]]:
return cats
def get_hot_radios(cookies: dict[str, str] | None = None,
limit: int = 30, offset: int = 0,
cate_id: int | None = None) -> list[dict[str, Any]]:
async def get_hot_radios(cookies: dict[str, str] | None = None,
limit: int = 30, offset: int = 0,
cate_id: int | None = None) -> list[dict[str, Any]]:
"""获取分类下热门播客
Args:
@@ -423,9 +415,9 @@ def get_hot_radios(cookies: dict[str, str] | None = None,
cate_ids.append(cid)
for cid in cate_ids:
result = _post_json(PodcastConstants.DJRADIO_HOT_API,
{'limit': limit, 'offset': max(0, int(offset or 0)), 'cateId': cid},
cookies)
result = await _post_json(PodcastConstants.DJRADIO_HOT_API,
{'limit': limit, 'offset': max(0, int(offset or 0)), 'cateId': cid},
cookies)
radios = [_norm_radio(r) for r in (result.get('djRadios') or [])]
if radios:
return radios
@@ -455,8 +447,8 @@ def _falldown_order(level: str) -> list[str]:
return list(dict.fromkeys(order)) # 去重保序
def resolve_episode_url(song_id: Any, cookies: dict[str, str] | None = None,
level: str = 'auto') -> dict[str, Any] | None:
async def resolve_episode_url(song_id: Any, cookies: dict[str, str] | None = None,
level: str = 'auto') -> dict[str, Any] | None:
"""解析单集音频直链(带音质降级)
Args:
@@ -473,9 +465,10 @@ def resolve_episode_url(song_id: Any, cookies: dict[str, str] | None = None,
sid = int(str(song_id).strip())
except (TypeError, ValueError):
return None
api = NeteaseAPI()
for lv in order:
try:
res = url_v1(sid, lv, cookies or {})
res = await api.get_song_url(sid, lv, cookies or {})
except (APIException, ValueError):
continue
data = (res or {}).get('data') or []
@@ -495,36 +488,36 @@ def resolve_episode_url(song_id: Any, cookies: dict[str, str] | None = None,
# ============ 组合能力 ============
def parse_podcast_ref(raw: str, cookies: dict[str, str] | None = None,
page_size: int = 30) -> dict[str, Any]:
async def parse_podcast_ref(raw: str, cookies: dict[str, str] | None = None,
page_size: int = 30) -> dict[str, Any]:
"""智能解析用户输入:电台链接 → 电台详情+单集列表;单集链接 → 单集详情
Returns:
{'kind': 'radio', 'radio': {...}, 'programs': [...], 'count': n, 'has_more': bool}
或 {'kind': 'program', 'program': {...}}
"""
ref = extract_podcast_ref(raw)
ref = await extract_podcast_ref(raw)
if ref['kind'] == 'radio':
radio = get_radio_detail(ref['id'], cookies)
page = get_radio_programs(ref['id'], cookies, limit=page_size, offset=0, asc=True)
radio = await get_radio_detail(ref['id'], cookies)
page = await get_radio_programs(ref['id'], cookies, limit=page_size, offset=0, asc=True)
return {'kind': 'radio', 'radio': radio, **page}
if ref['kind'] == 'program':
return {'kind': 'program', 'program': get_program_detail(ref['id'], cookies)}
return {'kind': 'program', 'program': await get_program_detail(ref['id'], cookies)}
# 纯数字:先试单集,再试电台
try:
return {'kind': 'program', 'program': get_program_detail(ref['id'], cookies)}
return {'kind': 'program', 'program': await get_program_detail(ref['id'], cookies)}
except APIException:
pass
radio = get_radio_detail(ref['id'], cookies)
page = get_radio_programs(ref['id'], cookies, limit=page_size, offset=0, asc=True)
radio = await get_radio_detail(ref['id'], cookies)
page = await get_radio_programs(ref['id'], cookies, limit=page_size, offset=0, asc=True)
return {'kind': 'radio', 'radio': radio, **page}
def list_radio_all_program_ids(radio_id: Any, cookies: dict[str, str] | None = None,
max_count: int = 200) -> list[dict[str, Any]]:
async def list_radio_all_program_ids(radio_id: Any, cookies: dict[str, str] | None = None,
max_count: int = 200) -> list[dict[str, Any]]:
"""拉取电台全部单集(用于整台打包,正序),返回 [{'id','name','main_song_id','serial_num','cover_url'}]
最多拉取 max_count 集,防止超大电台拖垮磁盘/任务。
@@ -537,7 +530,7 @@ def list_radio_all_program_ids(radio_id: Any, cookies: dict[str, str] | None = N
page_size = 100
max_pages = 30 # 有界保护:即使上游 has_more 恒真也最多翻 30 页
for _ in range(max_pages):
page = get_radio_programs(radio_id, cookies, limit=page_size, offset=offset, asc=True)
page = await get_radio_programs(radio_id, cookies, limit=page_size, offset=offset, asc=True)
new_count = 0
for p in page['programs']:
if p['id'] in seen_ids: