This repository has been archived on 2026-10-02. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
snowy-yang 69a50c2c31 refactor: worker 子系统全面异步化,拆掉同步转异步的桥接层
拆掉「同步调异步 / 异步桥同步」两类垫片的第二半(HTTP 线 B2 见前序提交)。
worker 侧此前是「起线程跑同步函数、函数体内部再用 asyncio.to_thread 把活塞给
另一条循环」——线程与协程互等,取消/暂停只能靠轮询。本轮把 worker 自身变成
协程,由 _start_task 直接挂到应用事件循环上,桥接层随之失去存在理由。

改动(按子系统):
- music_downloader:downloader / _aria2 / _concurrency / cover / _http /
  __main__ 全链路协程化。per-file 锁改 asyncio.Lock;计数闸改
  「active 计数 + await asyncio.sleep(0.5)」的可取消等待(原阻塞式 with 会让
  已取消任务仍要等全局名额腾出,卡死并拖后腿退款);aria2p 的同步 RPC 仍经
  to_thread,轮询改 asyncio.sleep;新增 core.http_client.iter_chunks,把**流式
  读取期**的 aiohttp.ClientError 收敛成 UpstreamNetworkError(请求建立阶段早已
  有这条路径,读取阶段此前没有,断流会绕过调用方的 UpstreamError 分类)。
  删除 get_http_session / _SESSION_LOCAL。
  models._RetryFromScratch 不再继承 requests.RequestException 而继承 Exception:
  它是「续传起点失效、残片已清、从头重下」,既不是确定性失败
  (DownloadException),也不是残片已保留的网络失败(UpstreamError),必须独立
  成类且在 except 链中先于二者捕获。
- music_api:podcast.py 全部函数协程化(直链解析内部改
  await NeteaseAPI.get_song_url);compat.py 删除——16 个死掉的同步门面早已
  清空,文件只剩空壳。
- services:_taskcore / _zipcore / batch / downloads / podcast / tasks 协程化。
  threading.Thread → loop.create_task(任务句柄字段 worker_thread →
  worker_task),ThreadPoolExecutor → asyncio.gather + asyncio.Semaphore,
  终态函数与结算回调全部改 await;监控线程里「worker 已死」判定改 task.done()。
  删除 services/streaming.py(同步流垫片,唯一消费者已迁到
  core.http_client.open)。zipfile 实例仍非线程安全,故「下载」并行、
  「入包」严格串行。
- web:路由层 130+ 处 to_thread 拆除——被包住的目标自己已是协程,再包一层只是
  把协程丢进线程里跑一圈。tasks_registry.start_task 直接 create_task。
  web/app.py 关闭段去掉 async_bridge.shutdown()(不再有第二条约循环)。
  web/routers/download.py 的播客端点补上 3 处漏掉的 await
  (extract_podcast_ref / get_radio_detail)——漏掉时不报错、只静默返回协程对象,
  靠 pyright 抓出来后修正。

留在 to_thread 上的都是**正当的阻塞卸载**:同步 SQLite 落库与账本结算、
mutagen 全量重写标签、zipfile 整段拷贝、shutil.rmtree、aria2p 的同步 RPC、
PBKDF2 的 10 万次迭代。ruff 对残留的本地元数据操作(exists/stat/os.replace/
往 .part 顺序写分块)按「微秒级、卸载的调度开销大于操作本身」放行,pyproject
逐条注明理由。

取消/暂停仍由 threading.Event 承载:置位方是请求线程/监控线程,读取方是 worker
协程,只有「置位」与「轮询读取」两种操作,既不需要跨循环唤醒也不会卡住事件
循环——比 asyncio.Event 更省心(ASYNC110 同理按此豁免)。

测试:10 个文件的替身边界同步迁移——网络替身从 core.sync_http 的
get_http_session 换成 downloader/服务模块的 _http.open + **真实** iter_chunks
(顺带钉住读取期异常收敛);`_run` 改为在独立线程里 asyncio.run(worker(...));
aria2 的假时钟从 time.sleep 换成 _POLL_INTERVAL=0;monitor 的线程句柄替身换成
done() 可控的轻量替身(asyncio.Task 的构造需要运行中的事件循环,而监控循环跑在
常驻线程里);podcast / 封面 / 结算包装等替身一律改 async。

门禁:ruff 全过 / pyright 0-0 / pytest 383 项全绿(0 失败)。

下一步 B2:auth_system 12 个模块改 aiosqlite 背板、web 层剩余 to_thread 收尾;
B3:删 core/sync_http.py、core/async_bridge.py、core/http_errors.py(现仅
sync_http 自用)。
2026-09-28 19:36:56 +08:00

453 lines
20 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# -*- coding: utf-8 -*-
"""下载提交端点。
端点:
- POST /api/v1/download ↔ Flask /download(计费 1 次;file/json 两模式)
- POST /api/v1/download/batch_zip ↔ Flask /download/batch_zip(按去重歌曲数计费)
- POST /api/v1/podcast/download ↔ Flask /podcast/download(计费 1 次)
- POST /api/v1/podcast/download/batch_zip ↔ Flask /podcast/download/batch_zip
计费:经 web.billing.require_billing(预扣+开账同事务)+ billing.scope()
(「任何失败路径必须退款」不变量收敛在作用域退出:自动退仍 open 的行,
settle/rename 消费过的行天然豁免);任务登记与启动由 web.tasks_registry 承担;
worker 复用 services 层既有实现,账本与交付语义不变。
"""
import logging
import threading
import time
from typing import Annotated, Any
from fastapi import APIRouter, Depends, Request
from core import media
from core.config import VALID_QUALITY_LEVELS
from music_api import APIException
from web import ratelimit
from web.billing import require_billing
from web.deps import SiteUser, current_user, require_login
from web.request_utils import cookies as _cookies
from web.request_utils import extract_id as _extract_id
from web.request_utils import request_data as _request_data
from web.request_utils import safe_int as _safe_int
from web.response import error, success
from web.serializers import record_req_log as _record_log
from web.tasks_registry import register_download_task, start_task
router = APIRouter()
LoginUser = Annotated[SiteUser, Depends(require_login)]
logger = logging.getLogger('web.download')
def _parse_meta_fields(raw: Any):
from core.utils import parse_meta_fields
return parse_meta_fields(raw)
def _parse_download_options(data: dict[str, Any]) -> tuple:
"""file/json 两模式共用的可选参数解析。
返回 (filename_format, write_meta, meta_fields, lyric_format, need_zip);
lyric_format 非法时静默回退 lrc(历史口径)。
"""
filename_format = data.get('filename_format', 'artist_song')
write_meta = str(data.get('write_meta', 'true')).lower() in ('1', 'true', 'yes')
meta_fields = _parse_meta_fields(data.get('meta_fields'))
lyric_format = data.get('lyric_format', 'lrc')
if lyric_format not in ('lrc', 'srt'):
lyric_format = 'lrc'
need_zip = str(data.get('zip', 'false')).lower() in ('1', 'true', 'yes')
return filename_format, write_meta, meta_fields, lyric_format, need_zip
async def _sync_download_json(downloader, music_id: str, quality: str, cookies: dict,
filename_format: str, write_meta: bool, meta_fields,
billing, ip: str):
"""format=json 的同步下载分支:取信息 → 下载 → 组响应 → 结算。
必须运行在 billing.scope() 内:失败退款由作用域退出统一完成,本函数
只负责在成功路径 settle(1, 'delivered')(结算后自动退自然豁免)。
"""
try:
info = await downloader.get_music_info(int(music_id), quality, cookies)
except Exception as e:
logger.error('获取音乐信息失败: %s', e)
return error('获取失败:歌曲不存在或暂无版权', 404)
from services._shared import _build_download_base
base = _build_download_base(
filename_format, {'name': info.name, 'artist_string': info.artists})
filename = f'{base}.{info.file_type}'
try:
dl = await downloader.download_music_file(
int(music_id), quality,
write_meta=write_meta, meta_fields=meta_fields,
cookies=cookies, music_info=info)
if not dl.success:
logger.error('下载失败: %s', dl.error_message)
return error('下载失败,请稍后重试或更换音质', 500)
except Exception as e:
logger.error('下载异常: %s', e)
return error('服务暂时不可用,请稍后再试', 500)
try:
_record_log('下载', ip, info.name)
except Exception:
pass
response_data = {
'music_id': music_id,
'name': info.name,
'artist': info.artists.replace('/', ', '),
'album': info.album,
'quality': quality,
'quality_name': media.quality_display_name(quality),
'file_type': info.file_type,
'file_size': info.file_size,
'file_size_formatted': media.format_file_size(info.file_size),
'filename': filename,
'duration': info.duration * 1000,
}
await billing.settle(1, 'delivered')
return success(response_data, '下载完成')
def _dedup_ids(raw_ids: Any) -> list[str]:
if isinstance(raw_ids, str):
raw_list = raw_ids.split(',')
else:
raw_list = [str(x) for x in (raw_ids or [])]
out, seen = [], set()
for x in raw_list:
mid = _extract_id(x)
s = str(mid).strip()
if s and s not in seen:
seen.add(s)
out.append(s)
return out
async def _resolve_personal(data: dict[str, Any], user: SiteUser | None, request: Request):
from web.routers.music import _resolve_user_cookies
return await _resolve_user_cookies(data, user, request)
# ============ 单曲下载 ============
@router.post('/download')
async def download(user: Annotated[SiteUser | None, Depends(current_user)], request: Request):
"""下载单曲(file=后台任务 / json=同步返回)。
顺序:限流 → 预扣计费 → billing.scope()(失败自动退款)→ 四道参数闸 →
账号 Cookie → 可选参数 → json 同步分支 / file 后台任务分支。
"""
await ratelimit.guard(request, bucket='dl', limit=20, window=60)
data = await _request_data(request)
music_id = data.get('id')
quality = data.get('quality', 'lossless')
return_format = data.get('format', 'file')
ip = ratelimit.client_ip(request)
billing = await require_billing(user, action='音乐下载', ip=ip)
async with billing.scope():
try:
if not music_id:
return error("参数 'music_id' 不能为空", 400)
music_id = str(music_id).strip()
if not music_id.isdigit():
return error('无效的歌曲 ID')
if quality not in VALID_QUALITY_LEVELS:
return error(f"无效的音质参数,支持: {', '.join(VALID_QUALITY_LEVELS)}")
if return_format not in ('file', 'json'):
return error("返回格式只支持 'file' 或 'json'")
music_id = _extract_id(music_id)
account = data.get('account', 'admin')
if account == 'user':
cookies = await _resolve_personal(data, user, request)
if not cookies:
return error('请先扫码登录个人账号后再使用个人解析', 401)
else:
cookies = _cookies()
(filename_format, write_meta, meta_fields,
lyric_format, need_zip) = _parse_download_options(data)
from services.api_service import api_service
downloader = api_service.downloader
if return_format == 'json':
return await _sync_download_json(
downloader, music_id, quality, cookies,
filename_format, write_meta, meta_fields, billing, ip)
# file 模式:注册后台任务
from services import downloads as services_downloads
from services import tasks as services_tasks
device_id = str(data.get('device_id') or '').strip()[:64]
services_tasks._touch_device(device_id)
task = _single_task({
'music_id': music_id, 'quality': quality,
'filename_format': filename_format, 'write_meta': write_meta,
'meta_fields': meta_fields, 'lyric_format': lyric_format,
'need_zip': need_zip, 'cookies': cookies, 'remote_ip': ip,
}, device_id)
reg_err = await register_download_task(billing, task, str(data.get('task_id') or '').strip())
if reg_err:
return error(reg_err, 429)
task_id = task['task_id']
start_err = await start_task(task_id, services_downloads._single_download_worker, (task_id,))
if start_err:
return error(start_err, 500)
return success({'task_id': task_id, 'status': 'queued'}, '下载任务已提交')
except APIException:
return error('服务暂时不可用,请稍后再试', 500)
except Exception as e:
logger.error('下载音乐异常: %s', e)
return error('服务暂时不可用,请稍后再试', 500)
def _single_task(params: dict[str, Any], device_id: str) -> dict[str, Any]:
"""组装单曲下载任务记录(params 原样透传给 worker)。
改成「一个字典 + device_id」而非 11 个位置参数:原先调用点要按固定顺序
传 11 个几乎全是字符串的参数,错位不报错却会静默串味(quality 与
filename_format 互换这类),而本函数只是把它们原样放进 params。
"""
return {
'type': 'single',
'total': 1, 'done': 0, 'failed': 0,
'status': 'queued', 'start': time.time(), 'msg': '排队中...',
'cancel': threading.Event(), 'pause': threading.Event(),
'device_id': device_id, 'result': None,
'params': params,
}
# ============ 批量 ZIP ============
@router.post('/download/batch_zip')
async def download_batch_zip(user: Annotated[SiteUser | None, Depends(current_user)], request: Request):
"""批量打包 ZIP(按去重歌曲数计费)。"""
await ratelimit.guard(request, bucket='dl', limit=20, window=60)
data = await _request_data(request)
ip = ratelimit.client_ip(request)
def _cost() -> int:
return max(1, min(len(_dedup_ids(data.get('ids') or data.get('id') or [])), 500))
billing = await require_billing(user, action='批量ZIP下载', cost=_cost, ip=ip)
async with billing.scope():
try:
raw_ids = data.get('ids') or data.get('id')
ids = _dedup_ids(raw_ids)
if not ids:
return error('缺少 ids 参数')
non_digit = [x for x in ids if not x.isdigit()]
if non_digit:
return error('ids 中含无效歌曲 ID')
if len(ids) > 500:
return error('单个批量任务最多 500 首,请分批提交')
quality = data.get('quality', 'lossless')
if quality not in VALID_QUALITY_LEVELS:
return error(f"无效的音质参数,支持: {', '.join(VALID_QUALITY_LEVELS)}")
filename_format = data.get('filename_format', 'artist_song')
write_meta = str(data.get('write_meta', 'true')).lower() in ('1', 'true', 'yes')
meta_fields = _parse_meta_fields(data.get('meta_fields'))
lyric_format = data.get('lyric_format', 'lrc')
if lyric_format not in ('lrc', 'srt'):
lyric_format = 'lrc'
try:
concurrency = int(data.get('concurrency', 8) or 8)
except (TypeError, ValueError):
concurrency = 8
concurrency = max(1, min(10, concurrency))
cookies = _cookies()
device_id = str(data.get('device_id') or '').strip()[:64]
from services import batch as services_batch
from services import tasks as services_tasks
services_tasks._touch_device(device_id)
task = {
'type': 'batch',
'total': len(ids), 'done': 0, 'failed': 0,
'status': 'queued', 'start': time.time(), 'msg': '排队中...',
'cancel': threading.Event(), 'pause': threading.Event(),
'device_id': device_id, 'result': None,
'params': {
'ids': ids, 'quality': quality, 'filename_format': filename_format,
'write_meta': write_meta, 'meta_fields': meta_fields,
'lyric_format': lyric_format, 'concurrency': concurrency,
'cookies': cookies, 'remote_ip': ip,
'want_zip': True,
},
}
reg_err = await register_download_task(billing, task, str(data.get('task_id') or '').strip())
if reg_err:
return error(reg_err, 429)
task_id = task['task_id']
start_err = await start_task(task_id, services_batch._batch_zip_worker, (task_id,))
if start_err:
return error(start_err, 500)
return success({'task_id': task_id, 'status': 'queued', 'total': len(ids)},
'批量打包任务已提交')
except Exception as e:
logger.error('批量打包提交异常: %s', e)
return error('服务暂时不可用,请稍后再试', 500)
# ============ 播客单集下载 ============
def _podcast_billing_count(data: dict[str, Any]) -> int:
"""播客打包计费。"""
if data.get('radio_id'):
return 1
limit = _safe_int(data.get('limit', 200) or 200, 200)
limit = max(1, min(limit, 500))
ids = _dedup_ids(data.get('ids'))
return max(1, min(len(ids), limit))
def _podcast_quality(data: dict[str, Any]) -> str:
q = str(data.get('quality') or 'auto')
return q if q in ('auto', 'standard', 'exhigh', 'lossless', 'hires') else 'auto'
@router.post('/podcast/download')
async def podcast_download(user: Annotated[SiteUser | None, Depends(current_user)], request: Request):
"""播客单集下载(计费 1 次)。"""
await ratelimit.guard(request, bucket='dl', limit=20, window=60)
data = await _request_data(request)
raw = data.get('id')
ip = ratelimit.client_ip(request)
billing = await require_billing(user, action='播客单集下载', ip=ip)
async with billing.scope():
try:
if not raw:
return error("参数 'id' 不能为空", 400)
from music_api import podcast
quality = _podcast_quality(data)
need_zip = str(data.get('zip', 'false')).lower() in ('1', 'true', 'yes')
ref = await podcast.extract_podcast_ref(raw)
if ref['kind'] == 'radio':
return error('这是电台链接,请先解析电台后选择单集下载,或使用「打包整个播客」')
account = data.get('account', 'admin')
if account == 'user':
cookies = await _resolve_personal(data, user, request)
if not cookies:
return error('请先扫码登录个人账号后再使用个人解析', 401)
else:
cookies = _cookies()
device_id = str(data.get('device_id') or '').strip()[:64]
from services import podcast as services_podcast
from services import tasks as services_tasks
services_tasks._touch_device(device_id)
task = {
'type': 'single',
'total': 1, 'done': 0, 'failed': 0,
'status': 'queued', 'start': time.time(), 'msg': '排队中...',
'cancel': threading.Event(), 'pause': threading.Event(),
'device_id': device_id, 'result': None,
'params': {'program_id': ref['id'], 'quality': quality,
'need_zip': need_zip, 'cookies': cookies, 'remote_ip': ip},
}
reg_err = await register_download_task(billing, task, str(data.get('task_id') or '').strip())
if reg_err:
return error(reg_err, 429)
task_id = task['task_id']
start_err = await start_task(task_id, services_podcast._podcast_download_worker, (task_id,))
if start_err:
return error(start_err, 500)
return success({'task_id': task_id, 'status': 'queued'}, '播客单集下载任务已提交')
except APIException as e:
return error(str(e), 404)
except Exception as e:
logger.error('播客单集下载提交异常: %s', e)
return error('服务暂时不可用,请稍后再试', 500)
@router.post('/podcast/download/batch_zip')
async def podcast_batch_zip(user: Annotated[SiteUser | None, Depends(current_user)], request: Request):
"""整台播客打包(先扣 1(电台模式)或按去重单集数扣,worker 按实际集数补扣差额)。"""
await ratelimit.guard(request, bucket='dl', limit=20, window=60)
data = await _request_data(request)
ip = ratelimit.client_ip(request)
def _cost() -> int:
return _podcast_billing_count(data)
billing = await require_billing(user, action='播客整台打包', cost=_cost, ip=ip)
async with billing.scope():
try:
raw_radio = data.get('radio_id')
raw_ids = data.get('ids')
if not raw_radio and not raw_ids:
return error('缺少 radio_id 或 ids 参数')
limit = max(1, min(_safe_int(data.get('limit', 200) or 200, 200), 500))
quality = _podcast_quality(data)
cookies = _cookies()
# id_list 只在「按单集 ID 列表」分支赋值,但下面组装任务参数时两个分支都会
# 引用它(radio_id 为假则取 id_list)——先绑定,避免异常/类型检查下的未绑定名
id_list: list[str] = []
from music_api import podcast
if raw_radio:
ref = await podcast.extract_podcast_ref(raw_radio)
if ref['kind'] == 'program':
return error('这是单集链接,请使用「下载单集」或直接输入电台链接/ID')
radio_id = ref['id']
radio_name = ''
try:
radio_name = (await podcast.get_radio_detail(radio_id, cookies)).get('name', '')
except APIException:
pass
else:
id_list = _dedup_ids(raw_ids)[:limit]
if not id_list:
return error('缺少 ids 参数')
radio_id = None
radio_name = str(data.get('name') or '播客单集')
billed = _podcast_billing_count(data)
device_id = str(data.get('device_id') or '').strip()[:64]
from services import podcast as services_podcast
from services import tasks as services_tasks
services_tasks._touch_device(device_id)
task = {
'type': 'batch',
'total': 0, 'done': 0, 'failed': 0,
'status': 'queued', 'start': time.time(), 'msg': '排队中...',
'cancel': threading.Event(), 'pause': threading.Event(),
'device_id': device_id, 'result': None,
'params': {
'radio_id': radio_id,
'ids': None if radio_id else id_list,
'limit': limit, 'quality': quality, 'radio_name': radio_name,
'cookies': cookies, 'remote_ip': ip,
# 整台模式:worker 解析完实际集数后按差额补扣(一首 = 1 次)
'billing_uid': user.id if user else None,
'billed': billed,
},
}
reg_err = await register_download_task(billing, task, str(data.get('task_id') or '').strip())
if reg_err:
return error(reg_err, 429)
task_id = task['task_id']
start_err = await start_task(task_id, services_podcast._podcast_zip_worker, (task_id,))
if start_err:
return error(start_err, 500)
return success({'task_id': task_id, 'status': 'queued'}, '播客打包任务已提交')
except APIException as e:
return error(str(e), 404)
except Exception as e:
logger.error('播客打包提交异常: %s', e)
return error('服务暂时不可用,请稍后再试', 500)