Archived
refactor: _fetch_via_http 拆解(94 行 / 圈 21 → 46 行 / 圈 5)
为什么:_fetch_via_http 是全仓内置下载通道的落盘实现,也是本文件此前最大的
圈复杂度所在。94 行的单体里叠着三段互不相关的职责——Range 起点校验、流式写盘
与体积上限、HTTP 错误分类——还共享一个 try/except/finally。上一轮拆
download_music_file 时它是唯一没动的重函数。
改法:拆成 7 个具名成员,主函数只留四步编排(构造 Range 头 → 发请求 →
定追加/重写并校验分片起点 → 流式落盘)。
_range_headers 续传时请求「起点之后的全部字节」
_resolve_write_mode 续传且服务器确实给 206 才追加,否则从头重写
_verify_content_range 校验 206 分片起点;无法确认就清残片从头下
_stream_body 逐块写盘 + 取消检查 + 体积硬上限(500MB)
_safe_progress 上报进度,回调异常吞掉(原两处 try/except 收敛)
_http_status 从 HTTPError 取状态码
_is_deterministic_client_error 4xx 判定(408 例外,唯一保留可重试的 4xx)
复杂度(同口径 AST 实测,HEAD 与工作区用同一脚本):
_fetch_via_http 94 行 / 圈 21 / 嵌套 7 → 46 行 / 圈 5 / 嵌套 3
文件内最大圈复杂度 21(当时就是它) → 16(余下最大是 download_music_file)
文件行数 710 → 753(+43,全是新签名与 docstring)· 函数数 19 → 26
口径说明:上一轮口头报的「圈 35」用的是另一脚本(另计入布尔运算与比较),
本轮 HEAD 同口径实测为 21。以本轮为准——HEAD 与重构后由同一脚本测出,可比。
等价性证据(两道,互相独立):
1. 反向注入验证:把 HEAD 版 downloader 注入成 music_downloader.downloader,用同一套
tests/test_downloader.py 跑 → 30 例**全过**。证明这批测试忠实编码了旧行为,
因而对新实现构成等价性验证(含本次新增的 5xx 用例,它在旧实现上同样通过)。
2. AST 字面量多重集比对 + 逐条位置定位:数值/布尔 26 种、字符串 80 种,
差异全部归因,且**没有一条用户可见文案或阈值被改动**:
206 2→1:旧的两段独立 if(`!= 206` 归零 mode、`== 206` 校验起点)合并进
_resolve_write_mode 的一个判断
0 26→24:两处 progress 的 `or 0` 收敛成 expected_size(-1);
两处 206 判断各含一个 `> 0` 合并成一个 `<= 0`(-1)
'wb' 2→1:初始赋值与条件改写两处合并成一个 return
500MB 阈值、各条错误文案原样保留。
重写条件表达式必须补分支穷举(字面量审计不管这一层),已逐路核对:
resume_from <= 0 → 'wb'(旧:不进任何 if,保持初始 'wb')
resume_from > 0 且 status != 206 → 'wb'(旧:第一个 if 命中)
resume_from > 0 且 status == 206 → 校验起点 + 'ab'(旧:第二个 if 命中)
三路与旧实现一致。
刻意保留的语义(是决定,不是疏漏):
- 超限清理仍**延后到 with 退出后**——Windows 无法删除仍被打开的文件,原实现在
打开中 unlink 被静默吞掉,残片会留在磁盘上。
- 进度回调异常一律吞掉,绝不影响下载主流程。
- 4xx 里 408 是唯一例外(保留可重试);416 与「206 头被剥离解析不出起点」同口径
清残片从头下(盲追加拼出的是坏文件)。
- 非 416/4xx 的错误走**裸 raise**(不用 `raise e`),保持原 traceback 不被改动。
测试:新增 1 例 test_5xx_is_retried_and_keeps_part —— 5xx 是服务端瞬时故障而非
确定性失败,须可重试且保留 .part 供续传。这条路径此前无覆盖,且它正是
_is_deterministic_client_error 与 4xx 的分界所在。
顺带记录一个验证工具自身的坑(第一次跑反向注入时 30 例全红,看似「测试不忠实」,
实为注入没生效、测试静默走了真实网络,报错为 Cannot connect to host cdn.test):
只覆盖 sys.modules['music_downloader.downloader'] 不够——pytest 的 monkeypatch
解析目标字符串走 _pytest.monkeypatch.resolve(),是「逐级 getattr 父包」而非纯
sys.modules 查找,包的 downloader 属性仍指向真模块。必须同时 setattr 父包属性。
教训:等价性验证的**工具本身**也要能证伪,否则结论会 180 度反转。
验证:ruff 全过 · pyright 0/0 · pytest 275 passed(274→275)· 行尾 i/lf w/lf
与仓库一致 · 生产 data/ 未被触碰(全部用例走 tmp_path 沙箱)。
This commit is contained in:
1 parent
8716617d5d
commit
9910e4dbec
2 files changed
+123
-60
No files matched your search
+103
-60
@@ -572,83 +572,35 @@ class MusicDownloader:
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""内置流式通道:core.sync_http 逐块落盘,支持 Range 断点续传。
|
"""内置流式通道:core.sync_http 逐块落盘,支持 Range 断点续传。
|
||||||
|
|
||||||
|
四步:构造 Range 头 → 发请求 → 定追加/重写并校验分片起点 → 流式落盘
|
||||||
|
(含体积上限与取消)。HTTP 错误在此分类成「确定性失败」或交外层重试。
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
DownloadCanceled: 取消事件置位
|
DownloadCanceled: 取消事件置位
|
||||||
DownloadException: 体积超限、CDN 4xx 等确定性失败
|
DownloadException: 体积超限、CDN 4xx 等确定性失败
|
||||||
_RetryFromScratch: 续传起点失效(残片已清,外层从头重下)
|
_RetryFromScratch: 续传起点失效(残片已清,外层从头重下)
|
||||||
"""
|
"""
|
||||||
headers = {}
|
expected_size = music_info.file_size or 0
|
||||||
mode = "wb"
|
self._safe_progress(progress_cb, resume_from, expected_size)
|
||||||
if resume_from > 0:
|
|
||||||
headers["Range"] = f"bytes={resume_from}-"
|
|
||||||
mode = "ab"
|
|
||||||
|
|
||||||
if progress_cb is not None:
|
|
||||||
try:
|
|
||||||
progress_cb(resume_from, music_info.file_size or 0)
|
|
||||||
except Exception:
|
|
||||||
pass # 进度回调绝不影响下载主流程
|
|
||||||
resp: requests.SyncResponse | None = None
|
resp: requests.SyncResponse | None = None
|
||||||
try:
|
try:
|
||||||
resp = get_http_session().get(
|
resp = get_http_session().get(
|
||||||
music_info.download_url,
|
music_info.download_url,
|
||||||
stream=True,
|
stream=True,
|
||||||
timeout=(10, 60),
|
timeout=(10, 60),
|
||||||
headers=headers,
|
headers=self._range_headers(resume_from),
|
||||||
)
|
)
|
||||||
# 服务器忽略 Range 返回 200 → 从头重写而非追加
|
mode = self._resolve_write_mode(resp, resume_from, part_path)
|
||||||
if resume_from > 0 and resp.status_code != 206:
|
|
||||||
mode = "wb"
|
|
||||||
# 206 必须从我们请求的起点开始;起点错位的分片拼起来是坏文件
|
|
||||||
if resume_from > 0 and resp.status_code == 206:
|
|
||||||
cr = resp.headers.get('Content-Range') or ''
|
|
||||||
m = re.match(r'bytes\s+(\d+)-', cr.strip())
|
|
||||||
if m and int(m.group(1)) != resume_from:
|
|
||||||
self._unlink_quiet(part_path)
|
|
||||||
raise _RetryFromScratch(
|
|
||||||
f"Content-Range 起点错位({m.group(1)} != {resume_from}),已从头重新下载")
|
|
||||||
# 206 却解析不出起点(头被中间层剥离)——盲追加可能拼坏文件,
|
|
||||||
# 与起点错位同口径处理:清残片从头下
|
|
||||||
if not m:
|
|
||||||
self._unlink_quiet(part_path)
|
|
||||||
raise _RetryFromScratch("206 响应缺少可用 Content-Range,已从头重新下载")
|
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
# 下载体积硬上限(配置 500MB),超限即弃,防单文件拖垮磁盘
|
self._stream_body(resp, part_path, mode, resume_from,
|
||||||
max_bytes = _MAX_DOWNLOAD_BYTES
|
expected_size, cancel_event, progress_cb)
|
||||||
written = resume_from if mode == 'ab' else 0
|
|
||||||
over_limit = False
|
|
||||||
with open(part_path, mode) as f:
|
|
||||||
for chunk in resp.iter_content(chunk_size=DL_CHUNK_SIZE):
|
|
||||||
if cancel_event is not None and cancel_event.is_set():
|
|
||||||
raise DownloadCanceled("下载已取消")
|
|
||||||
if chunk:
|
|
||||||
f.write(chunk)
|
|
||||||
written += len(chunk)
|
|
||||||
if written > max_bytes:
|
|
||||||
# 超限残片没有任何续传价值。
|
|
||||||
# Windows 无法删除打开中的文件——置标志 break,
|
|
||||||
# with 关闭后再清理(原实现在打开中 unlink 被静默吞掉)
|
|
||||||
over_limit = True
|
|
||||||
break
|
|
||||||
if progress_cb is not None:
|
|
||||||
try:
|
|
||||||
progress_cb(written, music_info.file_size or 0)
|
|
||||||
except Exception:
|
|
||||||
pass # 进度回调绝不影响下载主流程
|
|
||||||
if over_limit:
|
|
||||||
self._unlink_quiet(part_path) # 文件已关闭,清理可达
|
|
||||||
raise DownloadException(
|
|
||||||
f"文件超过大小上限 {max_bytes // (1024*1024)}MB,已中止")
|
|
||||||
except requests.HTTPError as e:
|
except requests.HTTPError as e:
|
||||||
# Range 起点无效(如 CDN 环节变化):清除旧 part,从头重新下载
|
status = self._http_status(e)
|
||||||
status = getattr(getattr(e, "response", None), "status_code", None)
|
|
||||||
if status == 416:
|
if status == 416:
|
||||||
|
# Range 起点无效(如 CDN 环节变化):清除旧 part,从头重新下载
|
||||||
self._unlink_quiet(part_path)
|
self._unlink_quiet(part_path)
|
||||||
raise _RetryFromScratch("Range 起点失效,已从头重新下载") from e
|
raise _RetryFromScratch("Range 起点失效,已从头重新下载") from e
|
||||||
if status is not None and 400 <= status < 500 and status != 408:
|
if self._is_deterministic_client_error(status):
|
||||||
# CDN 4xx(403 防盗链/429 风控等)是确定性
|
|
||||||
# 失败,对同一失效直链白重试只会加剧风控、拖慢失败返回
|
|
||||||
# (408 请求超时保留可重试)
|
|
||||||
self._unlink_quiet(part_path)
|
self._unlink_quiet(part_path)
|
||||||
raise DownloadException(
|
raise DownloadException(
|
||||||
f"下载失败(HTTP {status}),下载链接可能已失效") from e
|
f"下载失败(HTTP {status}),下载链接可能已失效") from e
|
||||||
@@ -657,6 +609,97 @@ class MusicDownloader:
|
|||||||
if resp is not None:
|
if resp is not None:
|
||||||
resp.close()
|
resp.close()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _range_headers(resume_from: int) -> dict[str, str]:
|
||||||
|
"""续传时请求「起点之后的全部字节」,否则不带头。"""
|
||||||
|
if resume_from > 0:
|
||||||
|
return {"Range": f"bytes={resume_from}-"}
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def _resolve_write_mode(self, resp, resume_from: int, part_path: Path) -> str:
|
||||||
|
"""落盘方式:续传且服务器确实给了 206 才追加,否则从头重写。
|
||||||
|
|
||||||
|
服务器忽略 Range 返回 200 时若继续追加,会把整段拼在残片后面;206 还须
|
||||||
|
校验分片起点(见 _verify_content_range)。
|
||||||
|
"""
|
||||||
|
if resume_from <= 0 or resp.status_code != 206:
|
||||||
|
return "wb"
|
||||||
|
self._verify_content_range(resp, resume_from, part_path)
|
||||||
|
return "ab"
|
||||||
|
|
||||||
|
def _verify_content_range(self, resp, resume_from: int, part_path: Path) -> None:
|
||||||
|
"""校验 206 的分片起点;无法确认起点就清残片从头下。
|
||||||
|
|
||||||
|
起点错位的分片拼起来是坏文件——代价远大于多下一次,故与「头被中间层
|
||||||
|
剥离、解析不出起点」同口径处理。
|
||||||
|
"""
|
||||||
|
content_range = (resp.headers.get("Content-Range") or "").strip()
|
||||||
|
m = re.match(r"bytes\s+(\d+)-", content_range)
|
||||||
|
if m is not None and int(m.group(1)) == resume_from:
|
||||||
|
return
|
||||||
|
self._unlink_quiet(part_path)
|
||||||
|
if m is None:
|
||||||
|
raise _RetryFromScratch("206 响应缺少可用 Content-Range,已从头重新下载")
|
||||||
|
raise _RetryFromScratch(
|
||||||
|
f"Content-Range 起点错位({m.group(1)} != {resume_from}),已从头重新下载")
|
||||||
|
|
||||||
|
def _stream_body(
|
||||||
|
self,
|
||||||
|
resp,
|
||||||
|
part_path: Path,
|
||||||
|
mode: str,
|
||||||
|
resume_from: int,
|
||||||
|
expected_size: int,
|
||||||
|
cancel_event: threading.Event | None,
|
||||||
|
progress_cb,
|
||||||
|
) -> None:
|
||||||
|
"""逐块写入 part_path:取消检查 + 体积硬上限(500MB)+ 进度上报。
|
||||||
|
|
||||||
|
超限不能就地删除——Windows 无法删除仍被打开的文件,先置标志退出 with
|
||||||
|
再清理(原实现在打开中 unlink 被静默吞掉,超限残片留在磁盘上)。
|
||||||
|
"""
|
||||||
|
max_bytes = _MAX_DOWNLOAD_BYTES
|
||||||
|
written = resume_from if mode == "ab" else 0
|
||||||
|
over_limit = False
|
||||||
|
with open(part_path, mode) as f:
|
||||||
|
for chunk in resp.iter_content(chunk_size=DL_CHUNK_SIZE):
|
||||||
|
if cancel_event is not None and cancel_event.is_set():
|
||||||
|
raise DownloadCanceled("下载已取消")
|
||||||
|
if not chunk:
|
||||||
|
continue
|
||||||
|
f.write(chunk)
|
||||||
|
written += len(chunk)
|
||||||
|
if written > max_bytes:
|
||||||
|
over_limit = True
|
||||||
|
break
|
||||||
|
self._safe_progress(progress_cb, written, expected_size)
|
||||||
|
if over_limit:
|
||||||
|
self._unlink_quiet(part_path) # 文件已关闭,清理可达
|
||||||
|
raise DownloadException(
|
||||||
|
f"文件超过大小上限 {max_bytes // (1024*1024)}MB,已中止")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _safe_progress(progress_cb, written: int, total: int) -> None:
|
||||||
|
"""上报一次进度;回调抛异常绝不影响下载主流程。"""
|
||||||
|
if progress_cb is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
progress_cb(written, total)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _http_status(exc: BaseException) -> int | None:
|
||||||
|
"""从 HTTPError 上取响应状态码(无响应对象时返回 None)。"""
|
||||||
|
return getattr(getattr(exc, "response", None), "status_code", None)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_deterministic_client_error(status: int | None) -> bool:
|
||||||
|
"""确定性 4xx(403 防盗链 / 429 风控等):对同一失效直链重试只会加剧风控、
|
||||||
|
拖慢失败返回。408 请求超时是唯一保留可重试的 4xx。
|
||||||
|
"""
|
||||||
|
return status is not None and 400 <= status < 500 and status != 408
|
||||||
|
|
||||||
def _fetch_via_aria2(
|
def _fetch_via_aria2(
|
||||||
self,
|
self,
|
||||||
backend: Aria2Backend,
|
backend: Aria2Backend,
|
||||||
|
|||||||
@@ -394,6 +394,26 @@ def test_408_is_retried_then_reported_as_failure(env):
|
|||||||
assert len(session.calls) == 2, '408 请求超时应保留可重试'
|
assert len(session.calls) == 2, '408 请求超时应保留可重试'
|
||||||
|
|
||||||
|
|
||||||
|
def test_5xx_is_retried_and_keeps_part(env):
|
||||||
|
"""5xx 是服务端瞬时故障(非确定性失败):应可重试,且保留 .part 供续传。
|
||||||
|
|
||||||
|
与 4xx 的分界在这里钉住:error 分类只看 400<=status<500,5xx 落回裸重抛,
|
||||||
|
由外层 RequestException 分支退避重试。此前这条路径无测试覆盖。
|
||||||
|
"""
|
||||||
|
dl, _state, install = env
|
||||||
|
info = _info(size=100)
|
||||||
|
fp, part = _paths(dl, info)
|
||||||
|
part.write_bytes(b'p' * 10)
|
||||||
|
session = install(lambda a, u, kw: _FakeResponse([], status=500))
|
||||||
|
|
||||||
|
r = dl.download_music_file(101, music_info=info, retries=1)
|
||||||
|
|
||||||
|
assert not r.success and '下载请求失败' in r.error_message
|
||||||
|
assert len(session.calls) == 2, '5xx 非确定性失败,应重试'
|
||||||
|
assert not fp.exists(), '未交付不得留下最终文件'
|
||||||
|
assert part.stat().st_size == 10, '.part 应保留供下轮续传'
|
||||||
|
|
||||||
|
|
||||||
def test_incomplete_size_keeps_part_and_reports(env):
|
def test_incomplete_size_keeps_part_and_reports(env):
|
||||||
dl, _state, install = env
|
dl, _state, install = env
|
||||||
info = _info(size=100)
|
info = _info(size=100)
|
||||||
|
|||||||
Reference in new issue
Block a user