Archived
refactor: worker 子系统全面异步化,拆掉同步转异步的桥接层
拆掉「同步调异步 / 异步桥同步」两类垫片的第二半(HTTP 线 B2 见前序提交)。 worker 侧此前是「起线程跑同步函数、函数体内部再用 asyncio.to_thread 把活塞给 另一条循环」——线程与协程互等,取消/暂停只能靠轮询。本轮把 worker 自身变成 协程,由 _start_task 直接挂到应用事件循环上,桥接层随之失去存在理由。 改动(按子系统): - music_downloader:downloader / _aria2 / _concurrency / cover / _http / __main__ 全链路协程化。per-file 锁改 asyncio.Lock;计数闸改 「active 计数 + await asyncio.sleep(0.5)」的可取消等待(原阻塞式 with 会让 已取消任务仍要等全局名额腾出,卡死并拖后腿退款);aria2p 的同步 RPC 仍经 to_thread,轮询改 asyncio.sleep;新增 core.http_client.iter_chunks,把**流式 读取期**的 aiohttp.ClientError 收敛成 UpstreamNetworkError(请求建立阶段早已 有这条路径,读取阶段此前没有,断流会绕过调用方的 UpstreamError 分类)。 删除 get_http_session / _SESSION_LOCAL。 models._RetryFromScratch 不再继承 requests.RequestException 而继承 Exception: 它是「续传起点失效、残片已清、从头重下」,既不是确定性失败 (DownloadException),也不是残片已保留的网络失败(UpstreamError),必须独立 成类且在 except 链中先于二者捕获。 - music_api:podcast.py 全部函数协程化(直链解析内部改 await NeteaseAPI.get_song_url);compat.py 删除——16 个死掉的同步门面早已 清空,文件只剩空壳。 - services:_taskcore / _zipcore / batch / downloads / podcast / tasks 协程化。 threading.Thread → loop.create_task(任务句柄字段 worker_thread → worker_task),ThreadPoolExecutor → asyncio.gather + asyncio.Semaphore, 终态函数与结算回调全部改 await;监控线程里「worker 已死」判定改 task.done()。 删除 services/streaming.py(同步流垫片,唯一消费者已迁到 core.http_client.open)。zipfile 实例仍非线程安全,故「下载」并行、 「入包」严格串行。 - web:路由层 130+ 处 to_thread 拆除——被包住的目标自己已是协程,再包一层只是 把协程丢进线程里跑一圈。tasks_registry.start_task 直接 create_task。 web/app.py 关闭段去掉 async_bridge.shutdown()(不再有第二条约循环)。 web/routers/download.py 的播客端点补上 3 处漏掉的 await (extract_podcast_ref / get_radio_detail)——漏掉时不报错、只静默返回协程对象, 靠 pyright 抓出来后修正。 留在 to_thread 上的都是**正当的阻塞卸载**:同步 SQLite 落库与账本结算、 mutagen 全量重写标签、zipfile 整段拷贝、shutil.rmtree、aria2p 的同步 RPC、 PBKDF2 的 10 万次迭代。ruff 对残留的本地元数据操作(exists/stat/os.replace/ 往 .part 顺序写分块)按「微秒级、卸载的调度开销大于操作本身」放行,pyproject 逐条注明理由。 取消/暂停仍由 threading.Event 承载:置位方是请求线程/监控线程,读取方是 worker 协程,只有「置位」与「轮询读取」两种操作,既不需要跨循环唤醒也不会卡住事件 循环——比 asyncio.Event 更省心(ASYNC110 同理按此豁免)。 测试:10 个文件的替身边界同步迁移——网络替身从 core.sync_http 的 get_http_session 换成 downloader/服务模块的 _http.open + **真实** iter_chunks (顺带钉住读取期异常收敛);`_run` 改为在独立线程里 asyncio.run(worker(...)); aria2 的假时钟从 time.sleep 换成 _POLL_INTERVAL=0;monitor 的线程句柄替身换成 done() 可控的轻量替身(asyncio.Task 的构造需要运行中的事件循环,而监控循环跑在 常驻线程里);podcast / 封面 / 结算包装等替身一律改 async。 门禁:ruff 全过 / pyright 0-0 / pytest 383 项全绿(0 失败)。 下一步 B2:auth_system 12 个模块改 aiosqlite 背板、web 层剩余 to_thread 收尾; B3:删 core/sync_http.py、core/async_bridge.py、core/http_errors.py(现仅 sync_http 自用)。
This commit is contained in:
1 parent
65adfc9089
commit
69a50c2c31
35 files changed
+1062
-1005
No files matched your search
@@ -1,19 +1,24 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""下载主类:落盘、元数据写入与 ZIP 打包。
|
||||
"""下载主类:落盘、元数据写入与交付。
|
||||
|
||||
(自 music_downloader.py 拆出,行为与原实现一致。)
|
||||
(自 music_downloader.py 拆出。)
|
||||
|
||||
异步口径:整个下载链路是协程。网络走 core.http_client(进程级连接池),
|
||||
文件落盘/标签写入是阻塞 IO 与纯 Python CPU 工作,经 ``asyncio.to_thread`` 卸载;
|
||||
互斥改 asyncio 锁(见 _concurrency),取消仍由 threading.Event 承载——
|
||||
取消位由请求线程/监控线程置位,只有轮询读取,跨线程安全且无需跨循环唤醒。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from core import sync_http as requests # 同步门面(底层 aiohttp,替代 requests)
|
||||
from core.async_bridge import run_sync
|
||||
from core.http_client import UpstreamError, UpstreamHTTPError, iter_chunks
|
||||
from core.http_client import client as _http
|
||||
from music_api import APIException, NeteaseAPI, load_cookies
|
||||
|
||||
from ._aria2 import Aria2Backend, get_backend
|
||||
@@ -24,7 +29,6 @@ from ._concurrency import (
|
||||
from ._http import (
|
||||
DL_CHUNK_SIZE,
|
||||
_sniff_audio_header,
|
||||
get_http_session,
|
||||
)
|
||||
from ._log import (
|
||||
logger,
|
||||
@@ -52,7 +56,7 @@ class MusicDownloader:
|
||||
|
||||
用法:
|
||||
downloader = MusicDownloader(download_dir="downloads")
|
||||
result = downloader.download_music_file(music_id, quality="lossless")
|
||||
result = await downloader.download_music_file(music_id, quality="lossless")
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -66,7 +70,7 @@ class MusicDownloader:
|
||||
download_dir: 下载目录
|
||||
max_concurrent: 单个批量任务内的最大并发下载数
|
||||
global_max_concurrent: 整个服务同时下载的文件数(含标签写入)。
|
||||
所有请求共享该信号量,防止大量用户同时下载导致内存/CPU 耗尽。
|
||||
所有请求共享该闸,防止大量用户同时下载导致内存/CPU 耗尽。
|
||||
"""
|
||||
self.download_dir = Path(download_dir)
|
||||
self.download_dir.mkdir(exist_ok=True)
|
||||
@@ -74,14 +78,14 @@ class MusicDownloader:
|
||||
|
||||
self.api = NeteaseAPI()
|
||||
|
||||
# 全局并发信号量:所有请求共用,限制同时进行的下载数
|
||||
# 全局并发闸:所有请求共用,限制同时进行的下载数
|
||||
# (带计数,供前端实时展示排队情况)
|
||||
self._download_semaphore = CountingSemaphore(global_max_concurrent)
|
||||
|
||||
# 标签写入专用信号量:写标签会重写整个文件(IO 密集),
|
||||
# 标签写入专用闸:写标签会重写整个文件(IO 密集),
|
||||
# 与下载分离限流 —— 写满时只阻塞后来者,不占下载名额、不拖慢其它文件的传输
|
||||
cpu_count = os.cpu_count() or 2
|
||||
self._tag_semaphore = threading.BoundedSemaphore(max(1, min(4, cpu_count)))
|
||||
self._tag_semaphore = asyncio.Semaphore(max(1, min(4, cpu_count)))
|
||||
|
||||
# ----------- 工具方法 -----------
|
||||
|
||||
@@ -145,24 +149,20 @@ class MusicDownloader:
|
||||
|
||||
# ----------- 信息获取 -----------
|
||||
|
||||
def get_music_info(self, music_id: int, quality: str = "standard",
|
||||
cookies: dict[str, str] | None = None) -> MusicInfo:
|
||||
"""获取音乐详细信息(同步,URL/详情/歌词 3 个接口并行请求,加快单曲/批量启动)"""
|
||||
async def get_music_info(self, music_id: int, quality: str = "standard",
|
||||
cookies: dict[str, str] | None = None) -> MusicInfo:
|
||||
"""获取音乐详细信息(URL/详情/歌词 3 个接口并发请求,加快单曲/批量启动)"""
|
||||
try:
|
||||
cookies = cookies if cookies is not None else load_cookies()
|
||||
|
||||
# 3 个接口并行请求,单曲下载/批量预取都能明显提速
|
||||
# URL/详情拿到后立即取消未完成的歌词请求,已知失败不再被拖着等完
|
||||
# 异步化:改用 asyncio.gather 并发(api 方法已异步),失败语义与旧线程池一致
|
||||
import asyncio
|
||||
|
||||
async def _gather_info():
|
||||
url_t = self.api.get_song_url(music_id, quality, cookies)
|
||||
detail_t = self.api.get_song_detail(music_id)
|
||||
lyric_t = self.api.get_lyric(music_id, cookies)
|
||||
return await asyncio.gather(url_t, detail_t, lyric_t, return_exceptions=True)
|
||||
|
||||
url_result, detail_result, lyric_result = run_sync(_gather_info())
|
||||
# 3 个接口并发请求,单曲下载/批量预取都能明显提速
|
||||
# URL/详情拿到后不再等待失败的歌词请求,已知失败不拖着等完
|
||||
url_result, detail_result, lyric_result = await asyncio.gather(
|
||||
self.api.get_song_url(music_id, quality, cookies),
|
||||
self.api.get_song_detail(music_id),
|
||||
self.api.get_lyric(music_id, cookies),
|
||||
return_exceptions=True,
|
||||
)
|
||||
if isinstance(url_result, BaseException):
|
||||
raise url_result
|
||||
if isinstance(detail_result, BaseException):
|
||||
@@ -255,9 +255,9 @@ class MusicDownloader:
|
||||
safe_tag = self._sanitize_filename(tag, max_bytes=tag_budget)
|
||||
return self.download_dir / f"{safe_base}{safe_tag}{ext}"
|
||||
|
||||
# ----------- 下载(同步) -----------
|
||||
# ----------- 下载 -----------
|
||||
|
||||
def download_music_file(
|
||||
async def download_music_file(
|
||||
self,
|
||||
music_id: int,
|
||||
quality: str = "standard",
|
||||
@@ -269,7 +269,7 @@ class MusicDownloader:
|
||||
music_info: MusicInfo | None = None,
|
||||
progress_cb=None,
|
||||
) -> DownloadResult:
|
||||
"""同步下载音乐到本地
|
||||
"""下载音乐到本地
|
||||
|
||||
Args:
|
||||
music_id: 音乐ID
|
||||
@@ -288,15 +288,15 @@ class MusicDownloader:
|
||||
- 流式下载到 .part 临时文件,完整下载后再原子改名,半截文件不会被当作已下载
|
||||
- 断点续传:失败保留 .part,重试/下次下载从已写字节续传(服务器支持 Range 时)
|
||||
- per-file 锁:同名歌曲并发下载互斥,防止 .part 被交叉写入损坏
|
||||
- 连接复用:线程级 Session + keep-alive,避免每首歌重建 TCP/TLS 连接
|
||||
- 网络传输与 ffmpeg 元数据写入分离限流,互不抢占对方资源
|
||||
- 连接复用:core.http_client 的进程级连接池 + keep-alive
|
||||
- 网络传输与标签写入分离限流,互不抢占对方资源
|
||||
|
||||
本函数只保留编排与重试策略:缓存校验、续传起点、传输通道、完整性判定、
|
||||
元数据限流各自见对应私有方法的 docstring(此前是 205 行 / 圈复杂度 47
|
||||
的单体,任何一处改动都要通读全部步骤)。
|
||||
"""
|
||||
if music_info is None:
|
||||
music_info = self.get_music_info(music_id, quality, cookies)
|
||||
music_info = await self.get_music_info(music_id, quality, cookies)
|
||||
|
||||
file_path = self._build_file_path(music_info)
|
||||
part_path0 = file_path.with_name(file_path.name + ".part")
|
||||
@@ -304,9 +304,9 @@ class MusicDownloader:
|
||||
|
||||
next_wait = 0.0
|
||||
for attempt in range(retries + 1):
|
||||
# 重试退避睡眠在文件锁外执行——同名并发任务不再陪睡
|
||||
# 重试退避等待在文件锁外执行——同名并发任务不再陪睡
|
||||
if next_wait > 0:
|
||||
time.sleep(next_wait)
|
||||
await asyncio.sleep(next_wait)
|
||||
next_wait = 0.0
|
||||
# 重试轮次顶部的取消检查也要清理 .part(原实现在 try 外抛出会绕过清理)
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
@@ -319,24 +319,24 @@ class MusicDownloader:
|
||||
part_path = part_path0
|
||||
|
||||
# 同一目标文件的下载者串行化(不同文件之间仍然并行)
|
||||
with get_file_lock(file_path):
|
||||
async with get_file_lock(file_path):
|
||||
try:
|
||||
# 其它线程可能刚好完成此文件,锁内复查(命中即返回,不再下载)
|
||||
cached = self._cached_result(file_path, music_info,
|
||||
write_meta, meta_fields)
|
||||
# 其它任务可能刚好完成此文件,锁内复查(命中即返回,不再下载)
|
||||
cached = await self._cached_result(file_path, music_info,
|
||||
write_meta, meta_fields)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
# 上次已下完但未改名的 .part:直接转正,不发起网络请求
|
||||
promoted = self._promote_complete_part(
|
||||
promoted = await self._promote_complete_part(
|
||||
part_path, file_path, music_info, write_meta, meta_fields)
|
||||
if promoted is not None:
|
||||
return promoted
|
||||
|
||||
# part_path 已在文件锁外绑定为 part_path0(同一路径)
|
||||
self._transfer(music_info, part_path,
|
||||
self._resume_offset(part_path, music_info),
|
||||
cancel_event, progress_cb)
|
||||
await self._transfer(music_info, part_path,
|
||||
self._resume_offset(part_path, music_info),
|
||||
cancel_event, progress_cb)
|
||||
|
||||
# 完整性校验(已知期望大小时必须精确相等;未知期望时至少非空)
|
||||
mismatch = self._size_mismatch_error(part_path, music_info)
|
||||
@@ -347,8 +347,8 @@ class MusicDownloader:
|
||||
continue
|
||||
|
||||
# 完整性通过 → 嗅探 → 原子转正 → 写元数据
|
||||
return self._publish(part_path, file_path, music_info,
|
||||
cancel_event, write_meta, meta_fields)
|
||||
return await self._publish(part_path, file_path, music_info,
|
||||
cancel_event, write_meta, meta_fields)
|
||||
|
||||
except DownloadCanceled:
|
||||
# 用户明确放弃:清理半截临时文件
|
||||
@@ -363,7 +363,7 @@ class MusicDownloader:
|
||||
last_error = "续传起点失效,重新下载"
|
||||
next_wait = self._backoff_wait(attempt, retries)
|
||||
continue
|
||||
except (requests.RequestException, OSError) as e:
|
||||
except (UpstreamError, OSError) as e:
|
||||
# 网络错误:保留 .part 支持断点续传;
|
||||
# 但若任务已被取消,.part 没有任何续传价值,一并清理
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
@@ -385,7 +385,7 @@ class MusicDownloader:
|
||||
|
||||
# ----------- 下载分步实现(download_music_file 的组成件) -----------
|
||||
|
||||
def _cached_result(
|
||||
async def _cached_result(
|
||||
self,
|
||||
file_path: Path,
|
||||
music_info: MusicInfo,
|
||||
@@ -406,9 +406,9 @@ class MusicDownloader:
|
||||
if cache_bad:
|
||||
self._unlink_quiet(file_path)
|
||||
return None
|
||||
# 缓存命中也尊重本次的元数据请求(重跑一次 ffmpeg,幂等)
|
||||
# 缓存命中也尊重本次的元数据请求(重跑一次写标签,幂等)
|
||||
if write_meta:
|
||||
self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
await self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
return DownloadResult(
|
||||
success=True,
|
||||
file_path=str(file_path),
|
||||
@@ -416,7 +416,7 @@ class MusicDownloader:
|
||||
music_info=music_info,
|
||||
)
|
||||
|
||||
def _promote_complete_part(
|
||||
async def _promote_complete_part(
|
||||
self,
|
||||
part_path: Path,
|
||||
file_path: Path,
|
||||
@@ -437,7 +437,7 @@ class MusicDownloader:
|
||||
return None
|
||||
os.replace(part_path, file_path)
|
||||
if write_meta:
|
||||
self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
await self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
return DownloadResult(
|
||||
success=True,
|
||||
file_path=str(file_path),
|
||||
@@ -460,7 +460,7 @@ class MusicDownloader:
|
||||
return 0
|
||||
return existing
|
||||
|
||||
def _transfer(
|
||||
async def _transfer(
|
||||
self,
|
||||
music_info: MusicInfo,
|
||||
part_path: Path,
|
||||
@@ -474,18 +474,20 @@ class MusicDownloader:
|
||||
"""
|
||||
# 排队用可取消等待:原阻塞式 with 使取消后仍要等全局 10 名额腾出,
|
||||
# 任务卡死、退款延迟
|
||||
if not self._download_semaphore.acquire_cancellable(cancel_event, poll=0.5):
|
||||
if not await self._download_semaphore.acquire_cancellable(cancel_event, poll=0.5):
|
||||
raise DownloadCanceled("下载已取消")
|
||||
try:
|
||||
aria2_backend = get_backend()
|
||||
# get_backend 首次调用要拉起 aria2c 并等 RPC 就绪(最长 8s),
|
||||
# 经 to_thread 卸载,不在事件循环上原地等
|
||||
aria2_backend = await asyncio.to_thread(get_backend)
|
||||
if aria2_backend is not None:
|
||||
# aria2c 通道:多连接 + 原生断点续传
|
||||
self._fetch_via_aria2(aria2_backend, music_info, part_path,
|
||||
cancel_event, progress_cb)
|
||||
await self._fetch_via_aria2(aria2_backend, music_info, part_path,
|
||||
cancel_event, progress_cb)
|
||||
else:
|
||||
# 内置通道:aiohttp 流式落盘 + Range 起点校验
|
||||
self._fetch_via_http(music_info, part_path, resume_from,
|
||||
cancel_event, progress_cb)
|
||||
await self._fetch_via_http(music_info, part_path, resume_from,
|
||||
cancel_event, progress_cb)
|
||||
finally:
|
||||
# 与 acquire_cancellable 配对释放(原 with 语义)
|
||||
self._download_semaphore.release()
|
||||
@@ -506,7 +508,7 @@ class MusicDownloader:
|
||||
return (f"下载不完整: 期望 {music_info.file_size or '未知'} 字节, "
|
||||
f"实际 {actual_size} 字节")
|
||||
|
||||
def _publish(
|
||||
async def _publish(
|
||||
self,
|
||||
part_path: Path,
|
||||
file_path: Path,
|
||||
@@ -527,14 +529,14 @@ class MusicDownloader:
|
||||
|
||||
os.replace(part_path, file_path)
|
||||
|
||||
# 元数据写入移出下载信号量(CPU 操作不占网络名额),
|
||||
# 元数据写入移出下载闸(CPU 操作不占网络名额),
|
||||
# 但仍在文件锁内执行,保证其它请求看到的是已写入标签的最终文件
|
||||
if write_meta:
|
||||
# 元数据段入口补取消检查点——已取消的任务不再
|
||||
# 进入信号量排队/ffmpeg 等待(有界但可达 120s),收尾不再延迟
|
||||
# 进入标签闸排队/写盘等待(有界但可达数十秒),收尾不再延迟
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
raise DownloadCanceled("下载已取消")
|
||||
self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
await self._write_metadata_locked(file_path, music_info, meta_fields)
|
||||
|
||||
return DownloadResult(
|
||||
success=True,
|
||||
@@ -543,26 +545,26 @@ class MusicDownloader:
|
||||
music_info=music_info,
|
||||
)
|
||||
|
||||
def _write_metadata_locked(
|
||||
async def _write_metadata_locked(
|
||||
self,
|
||||
file_path: Path,
|
||||
music_info: MusicInfo,
|
||||
meta_fields: list[str] | None,
|
||||
) -> None:
|
||||
"""在标签信号量内写元数据:写标签会重写整个文件(IO 密集),与下载分离
|
||||
"""在标签闸内写元数据:写标签会重写整个文件(IO 密集),与下载分离
|
||||
限流——写满时只阻塞后来者,不占下载名额、不拖慢其它文件的传输。"""
|
||||
with self._tag_semaphore:
|
||||
self._write_metadata(file_path, music_info, meta_fields)
|
||||
async with self._tag_semaphore:
|
||||
await self._write_metadata(file_path, music_info, meta_fields)
|
||||
|
||||
@staticmethod
|
||||
def _backoff_wait(attempt: int, retries: int) -> float:
|
||||
"""失败后的退避秒数;已是最后一轮则不等待。锁外睡眠——同名并发任务不陪睡。"""
|
||||
"""失败后的退避秒数;已是最后一轮则不等待。锁外等待——同名并发任务不陪睡。"""
|
||||
return 0.5 * (attempt + 1) if attempt < retries else 0.0
|
||||
|
||||
|
||||
# ----------- 传输通道(二选一,产物都是 part_path) -----------
|
||||
|
||||
def _fetch_via_http(
|
||||
async def _fetch_via_http(
|
||||
self,
|
||||
music_info: MusicInfo,
|
||||
part_path: Path,
|
||||
@@ -570,7 +572,7 @@ class MusicDownloader:
|
||||
cancel_event: threading.Event | None,
|
||||
progress_cb,
|
||||
) -> None:
|
||||
"""内置流式通道:core.sync_http 逐块落盘,支持 Range 断点续传。
|
||||
"""内置流式通道:core.http_client 逐块落盘,支持 Range 断点续传。
|
||||
|
||||
四步:构造 Range 头 → 发请求 → 定追加/重写并校验分片起点 → 流式落盘
|
||||
(含体积上限与取消)。HTTP 错误在此分类成「确定性失败」或交外层重试。
|
||||
@@ -579,23 +581,22 @@ class MusicDownloader:
|
||||
DownloadCanceled: 取消事件置位
|
||||
DownloadException: 体积超限、CDN 4xx 等确定性失败
|
||||
_RetryFromScratch: 续传起点失效(残片已清,外层从头重下)
|
||||
UpstreamError: 网络/5xx 等可重试失败(保留残片续传)
|
||||
"""
|
||||
expected_size = music_info.file_size or 0
|
||||
self._safe_progress(progress_cb, resume_from, expected_size)
|
||||
resp: requests.SyncResponse | None = None
|
||||
resp = None
|
||||
try:
|
||||
resp = get_http_session().get(
|
||||
music_info.download_url,
|
||||
stream=True,
|
||||
timeout=(10, 60),
|
||||
headers=self._range_headers(resume_from),
|
||||
)
|
||||
headers = self._range_headers(resume_from)
|
||||
resp = await _http.open('GET', music_info.download_url,
|
||||
headers=headers or None, timeout=60)
|
||||
mode = self._resolve_write_mode(resp, resume_from, part_path)
|
||||
resp.raise_for_status()
|
||||
self._stream_body(resp, part_path, mode, resume_from,
|
||||
expected_size, cancel_event, progress_cb)
|
||||
except requests.HTTPError as e:
|
||||
status = self._http_status(e)
|
||||
if resp.status >= 400:
|
||||
raise UpstreamHTTPError(resp.status, resp.reason or '')
|
||||
await self._stream_body(resp, part_path, mode, resume_from,
|
||||
expected_size, cancel_event, progress_cb)
|
||||
except UpstreamHTTPError as e:
|
||||
status = e.status
|
||||
if status == 416:
|
||||
# Range 起点无效(如 CDN 环节变化):清除旧 part,从头重新下载
|
||||
self._unlink_quiet(part_path)
|
||||
@@ -622,7 +623,7 @@ class MusicDownloader:
|
||||
服务器忽略 Range 返回 200 时若继续追加,会把整段拼在残片后面;206 还须
|
||||
校验分片起点(见 _verify_content_range)。
|
||||
"""
|
||||
if resume_from <= 0 or resp.status_code != 206:
|
||||
if resume_from <= 0 or resp.status != 206:
|
||||
return "wb"
|
||||
self._verify_content_range(resp, resume_from, part_path)
|
||||
return "ab"
|
||||
@@ -643,7 +644,7 @@ class MusicDownloader:
|
||||
raise _RetryFromScratch(
|
||||
f"Content-Range 起点错位({m.group(1)} != {resume_from}),已从头重新下载")
|
||||
|
||||
def _stream_body(
|
||||
async def _stream_body(
|
||||
self,
|
||||
resp,
|
||||
part_path: Path,
|
||||
@@ -662,7 +663,7 @@ class MusicDownloader:
|
||||
written = resume_from if mode == "ab" else 0
|
||||
over_limit = False
|
||||
with open(part_path, mode) as f:
|
||||
for chunk in resp.iter_content(chunk_size=DL_CHUNK_SIZE):
|
||||
async for chunk in iter_chunks(resp, DL_CHUNK_SIZE):
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
raise DownloadCanceled("下载已取消")
|
||||
if not chunk:
|
||||
@@ -688,11 +689,6 @@ class MusicDownloader:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _http_status(exc: BaseException) -> int | None:
|
||||
"""从 HTTPError 上取响应状态码(无响应对象时返回 None)。"""
|
||||
return getattr(getattr(exc, "response", None), "status_code", None)
|
||||
|
||||
@staticmethod
|
||||
def _is_deterministic_client_error(status: int | None) -> bool:
|
||||
"""确定性 4xx(403 防盗链 / 429 风控等):对同一失效直链重试只会加剧风控、
|
||||
@@ -700,7 +696,7 @@ class MusicDownloader:
|
||||
"""
|
||||
return status is not None and 400 <= status < 500 and status != 408
|
||||
|
||||
def _fetch_via_aria2(
|
||||
async def _fetch_via_aria2(
|
||||
self,
|
||||
backend: Aria2Backend,
|
||||
music_info: MusicInfo,
|
||||
@@ -711,10 +707,10 @@ class MusicDownloader:
|
||||
"""aria2c 通道:多连接加速 + 原生断点续传。
|
||||
|
||||
异常语义与内置通道对齐:取消抛 DownloadCanceled、体积超限抛
|
||||
DownloadException(均清残片),网络类失败抛 RequestException
|
||||
DownloadException(均清残片),网络类失败抛 UpstreamNetworkError
|
||||
(保留残片,由外层退避重试续传)。
|
||||
"""
|
||||
backend.fetch(
|
||||
await backend.fetch(
|
||||
music_info.download_url,
|
||||
part_path,
|
||||
cancel_event=cancel_event,
|
||||
@@ -728,7 +724,7 @@ class MusicDownloader:
|
||||
# 三个方法已删除——全服务端与验证脚本零调用(CLI 遗留),且 to_memory 不接受
|
||||
# cookies 参数、真被启用会有行为差异;如需内存下载/异步批量再按当前接口重写
|
||||
|
||||
def _write_metadata(
|
||||
async def _write_metadata(
|
||||
self,
|
||||
file_path: Path,
|
||||
music_info: MusicInfo,
|
||||
@@ -743,11 +739,13 @@ class MusicDownloader:
|
||||
|
||||
封面走共享缓存(与 ZIP 打包共用一次网络请求);各容器的落盘细节见
|
||||
tagging.py——按扩展名分派到 ID3v2.3 / Vorbis comment / iTunes ilst。
|
||||
写盘是阻塞 IO + 纯 Python CPU(mutagen 全量重写文件),经 to_thread 卸载。
|
||||
"""
|
||||
include_cover = fields is None or "cover" in fields
|
||||
cover: bytes | None = None
|
||||
if include_cover and music_info.pic_url:
|
||||
cover = fetch_cover_bytes(music_info.pic_url)
|
||||
cover = await fetch_cover_bytes(music_info.pic_url)
|
||||
if not cover:
|
||||
logger.warning("封面下载失败,跳过封面写入")
|
||||
return write_tags(file_path, build_tags(music_info, fields), cover)
|
||||
return await asyncio.to_thread(
|
||||
write_tags, file_path, build_tags(music_info, fields), cover)
|
||||
Reference in new issue
Block a user