This repository has been archived on 2026-10-02. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
NeteaseUrl/tests/test_downloader.py
T
snowy-yang 9910e4dbec refactor: _fetch_via_http 拆解(94 行 / 圈 21 → 46 行 / 圈 5)
为什么:_fetch_via_http 是全仓内置下载通道的落盘实现,也是本文件此前最大的
圈复杂度所在。94 行的单体里叠着三段互不相关的职责——Range 起点校验、流式写盘
与体积上限、HTTP 错误分类——还共享一个 try/except/finally。上一轮拆
download_music_file 时它是唯一没动的重函数。

改法:拆成 7 个具名成员,主函数只留四步编排(构造 Range 头 → 发请求 →
定追加/重写并校验分片起点 → 流式落盘)。
  _range_headers                  续传时请求「起点之后的全部字节」
  _resolve_write_mode             续传且服务器确实给 206 才追加,否则从头重写
  _verify_content_range           校验 206 分片起点;无法确认就清残片从头下
  _stream_body                    逐块写盘 + 取消检查 + 体积硬上限(500MB)
  _safe_progress                  上报进度,回调异常吞掉(原两处 try/except 收敛)
  _http_status                    从 HTTPError 取状态码
  _is_deterministic_client_error  4xx 判定(408 例外,唯一保留可重试的 4xx)

复杂度(同口径 AST 实测,HEAD 与工作区用同一脚本):
  _fetch_via_http     94 行 / 圈 21 / 嵌套 7  →  46 行 / 圈 5 / 嵌套 3
  文件内最大圈复杂度    21(当时就是它)        →  16(余下最大是 download_music_file)
  文件行数 710 → 753(+43,全是新签名与 docstring)· 函数数 19 → 26
  口径说明:上一轮口头报的「圈 35」用的是另一脚本(另计入布尔运算与比较),
  本轮 HEAD 同口径实测为 21。以本轮为准——HEAD 与重构后由同一脚本测出,可比。

等价性证据(两道,互相独立):
1. 反向注入验证:把 HEAD 版 downloader 注入成 music_downloader.downloader,用同一套
   tests/test_downloader.py 跑 → 30 例**全过**。证明这批测试忠实编码了旧行为,
   因而对新实现构成等价性验证(含本次新增的 5xx 用例,它在旧实现上同样通过)。
2. AST 字面量多重集比对 + 逐条位置定位:数值/布尔 26 种、字符串 80 种,
   差异全部归因,且**没有一条用户可见文案或阈值被改动**:
     206  2→1:旧的两段独立 if(`!= 206` 归零 mode、`== 206` 校验起点)合并进
               _resolve_write_mode 的一个判断
     0   26→24:两处 progress 的 `or 0` 收敛成 expected_size(-1);
               两处 206 判断各含一个 `> 0` 合并成一个 `<= 0`(-1)
     'wb' 2→1:初始赋值与条件改写两处合并成一个 return
   500MB 阈值、各条错误文案原样保留。

重写条件表达式必须补分支穷举(字面量审计不管这一层),已逐路核对:
  resume_from <= 0                     → 'wb'(旧:不进任何 if,保持初始 'wb')
  resume_from > 0 且 status != 206     → 'wb'(旧:第一个 if 命中)
  resume_from > 0 且 status == 206     → 校验起点 + 'ab'(旧:第二个 if 命中)
三路与旧实现一致。

刻意保留的语义(是决定,不是疏漏):
- 超限清理仍**延后到 with 退出后**——Windows 无法删除仍被打开的文件,原实现在
  打开中 unlink 被静默吞掉,残片会留在磁盘上。
- 进度回调异常一律吞掉,绝不影响下载主流程。
- 4xx 里 408 是唯一例外(保留可重试);416 与「206 头被剥离解析不出起点」同口径
  清残片从头下(盲追加拼出的是坏文件)。
- 非 416/4xx 的错误走**裸 raise**(不用 `raise e`),保持原 traceback 不被改动。

测试:新增 1 例 test_5xx_is_retried_and_keeps_part —— 5xx 是服务端瞬时故障而非
确定性失败,须可重试且保留 .part 供续传。这条路径此前无覆盖,且它正是
_is_deterministic_client_error 与 4xx 的分界所在。

顺带记录一个验证工具自身的坑(第一次跑反向注入时 30 例全红,看似「测试不忠实」,
实为注入没生效、测试静默走了真实网络,报错为 Cannot connect to host cdn.test):
只覆盖 sys.modules['music_downloader.downloader'] 不够——pytest 的 monkeypatch
解析目标字符串走 _pytest.monkeypatch.resolve(),是「逐级 getattr 父包」而非纯
sys.modules 查找,包的 downloader 属性仍指向真模块。必须同时 setattr 父包属性。
教训:等价性验证的**工具本身**也要能证伪,否则结论会 180 度反转。

验证:ruff 全过 · pyright 0/0 · pytest 275 passed(274→275)· 行尾 i/lf w/lf
与仓库一致 · 生产 data/ 未被触碰(全部用例走 tmp_path 沙箱)。
2026-09-27 08:05:44 +08:00

602 lines
19 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# -*- coding: utf-8 -*-
"""MusicDownloader.download_music_file 的行为契约测试(沙箱,不触碰生产 data/)。
为什么:download_music_file 是**唯一**的音频落盘路径——单曲 worker 与音乐批量
ZIP 都经它,约 205 行 / 圈复杂度 47,此前**零直接测试覆盖**(test_zip_workers 里
只有一个 stub)。它同时承担缓存校验、断点续传、内容嗅探、体积上限、取消语义、
重试退避与元数据写入,任何一条被拆坏都不会有测试报警。
断言只依赖「调用后世界里发生了什么」——返回值、磁盘上的文件、发往网络的请求——
不引用任何内部函数名,因此本文件可作为后续拆分的等价性验证器。
替身约定(三处,均为模块级符号,替换点在 downloader 模块命名空间):
- 网络:``get_http_session`` / ``get_backend``(后者返回 None = 走内置 HTTP 通道)
- 内容嗅探:``_sniff_audio_header``(默认放行,按需置 False)
- 元数据写入:``MusicDownloader._write_metadata``(不真跑 mutagen)
"""
import threading
from pathlib import Path
from typing import Any
import pytest
from core import sync_http
from music_downloader import DownloadCanceled, DownloadException, MusicInfo
from music_downloader.downloader import MusicDownloader
# ============ 替身 ============
class _FakeResponse:
"""core.sync_http.SyncResponse 的替身。
chunks 保留原始 iterable(**不要** list() 归一化):生成器的副作用
(如中途置取消位)若在构造时刻就求值,「中途取消」会被悄悄测成
「开始前取消」——两条不同的路径混成一条,而测试仍是绿的。
"""
def __init__(self, chunks, status=200, headers=None):
self._chunks = chunks
self.status_code = status
self.headers = dict(headers or {})
self.closed = False
def iter_content(self, chunk_size=0):
yield from self._chunks
def raise_for_status(self):
if self.status_code >= 400:
raise sync_http.HTTPError(f'HTTP {self.status_code}', self)
def close(self):
self.closed = True
class _FakeSession:
"""记录每次 GET(含 headers)并按序产出响应。"""
def __init__(self, responder):
self._responder = responder
self.calls: list[tuple[str, dict[str, Any]]] = []
self.closed = False
def get(self, url, **kw):
self.calls.append((url, kw))
return self._responder(len(self.calls), url, kw)
def close(self):
self.closed = True
def _body(size, status=200, content_range=None):
"""固定返回 size 字节的响应工厂(每次调用产出新响应对象)。"""
def _mk(_attempt, _url, _kw):
headers = {'Content-Range': content_range} if content_range else {}
return _FakeResponse([b'x' * size], status=status, headers=headers)
return _mk
def _info(mid=101, size=20, file_type='mp3'):
return MusicInfo(
id=mid, name='测试曲', artists='歌手', album='专辑', pic_url='',
duration=180, track_number=1, download_url='http://cdn.test/a.mp3',
file_type=file_type, file_size=size, quality='standard',
)
def _paths(dl, info):
"""按生产同一函数推算落盘路径与 .part 路径(纯函数,可直接复用)。"""
fp = dl._build_file_path(info)
return fp, fp.with_name(fp.name + '.part')
@pytest.fixture()
def env(tmp_path, monkeypatch):
"""返回 (downloader, state, install)。
- state['sniff']:内容嗅探结果,默认 True
- state['meta']:_write_metadata 的调用记录 [(文件路径, fields), ...]
- install(responder) -> _FakeSession:安装网络替身,返回该替身以便断言
"""
state: dict[str, Any] = {'sniff': True, 'meta': []}
box: dict[str, _FakeSession] = {}
def install(responder):
box['session'] = _FakeSession(responder)
return box['session']
def _record_meta(self, file_path, music_info, fields=None):
state['meta'].append((str(file_path), fields))
return True
monkeypatch.setattr('music_downloader.downloader.get_http_session',
lambda: box['session'])
monkeypatch.setattr('music_downloader.downloader.get_backend', lambda: None)
monkeypatch.setattr('music_downloader.downloader._sniff_audio_header',
lambda _p: state['sniff'])
monkeypatch.setattr(MusicDownloader, '_write_metadata', _record_meta)
install(lambda *a: pytest.fail('本用例不应发出网络请求'))
dl = MusicDownloader(download_dir=str(tmp_path))
return dl, state, install
# ============ 正常落盘 ============
def test_success_promotes_part_to_final(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.error_message == ''
assert Path(r.file_path) == fp
assert r.file_size == 20 and fp.read_bytes() == b'x' * 20
assert not part.exists(), '完成后 .part 应改名转正'
def test_empty_chunks_skipped(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
install(lambda a, u, kw: _FakeResponse([b'', b'x' * 20, b'']))
seen: list[int] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, _t: seen.append(w))
assert r.success and fp.stat().st_size == 20
assert seen == [0, 20], '空分块不写盘也不触发进度回调'
def test_progress_reports_resume_start_then_each_chunk(env):
dl, _state, install = env
info = _info(size=20)
install(lambda a, u, kw: _FakeResponse([b'a' * 8, b'b' * 12]))
seen: list[tuple[int, int]] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, t: seen.append((w, t)))
assert r.success
assert seen == [(0, 20), (8, 20), (20, 20)], '起始回调带续传起点,此后每块一次'
def test_progress_callback_exception_never_breaks_download(env):
dl, _state, install = env
info = _info(size=20)
install(_body(20))
def _boom(_w, _t):
raise RuntimeError('回调炸了')
r = dl.download_music_file(101, music_info=info, progress_cb=_boom)
assert r.success, '进度回调异常绝不影响下载主流程'
def test_music_info_fetched_when_not_provided(env, monkeypatch):
dl, _state, install = env
info = _info(size=20)
seen: list[tuple] = []
monkeypatch.setattr(dl, 'get_music_info',
lambda mid, q, c=None: (seen.append((mid, q)), info)[1])
install(_body(20))
r = dl.download_music_file(101, 'lossless')
assert r.success and seen == [(101, 'lossless')]
# ============ 缓存校验 ============
def test_cache_hit_skips_network(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
fp.write_bytes(b'c' * 20)
session = install(lambda *a: pytest.fail('缓存命中不应发网络请求'))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.file_size == 20
assert session.calls == []
def test_cache_size_mismatch_triggers_redownload(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
fp.write_bytes(b'c' * 15) # 与本次期望不符 = 缓存 miss
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'x' * 20
assert len(session.calls) == 1, '坏缓存应被删除并重下,而不是当结果交付'
def test_zero_byte_cache_rejected_when_expected_size_unknown(env):
dl, _state, install = env
info = _info(size=0)
fp, _ = _paths(dl, info)
fp.write_bytes(b'')
session = install(lambda a, u, kw: _FakeResponse([b'good']))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'good'
assert len(session.calls) == 1, '期望未知时至少拒绝 0 字节坏文件'
# ============ 断点续传 ============
def test_resume_sends_range_and_appends(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
session = install(lambda a, u, kw: _FakeResponse(
[b'q' * 10], status=206, headers={'Content-Range': 'bytes 10-19/20'}))
seen: list[tuple[int, int]] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, t: seen.append((w, t)))
assert r.success
assert session.calls[0][1]['headers'].get('Range') == 'bytes=10-'
assert fp.read_bytes() == b'p' * 10 + b'q' * 10, '续传应追加而非覆盖'
assert seen[0] == (10, 20), '起始进度应带续传起点'
def test_server_ignoring_range_rewrites_from_zero(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
install(lambda a, u, kw: _FakeResponse([b'r' * 20], status=200))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'r' * 20, '服务器返回 200 须从头重写'
def test_complete_part_promoted_without_network(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'z' * 20) # 已下完但未改名
session = install(lambda *a: pytest.fail('完整残片不应发网络请求'))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.file_size == 20
assert session.calls == []
assert fp.read_bytes() == b'z' * 20 and not part.exists()
def test_oversized_part_discarded_starts_from_zero(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'o' * 30) # 超尺寸残片没有续传价值
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success
assert 'Range' not in session.calls[0][1]['headers']
def test_zero_byte_part_discarded(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'')
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and 'Range' not in session.calls[0][1]['headers']
def test_content_range_offset_mismatch_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'q' * 5], status=206,
headers={'Content-Range': 'bytes 3-7/20'})
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers'], '清残片后应从头下'
assert fp.read_bytes() == b'f' * 20
def test_206_without_content_range_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'q' * 5], status=206) # 头被中间层剥离
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers']
def test_http_416_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([], status=416)
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers']
assert fp.read_bytes() == b'f' * 20
# ============ 失败与重试 ============
def test_cdn_4xx_is_deterministic_and_not_retried(env):
dl, _state, install = env
info = _info(size=20)
session = install(lambda a, u, kw: _FakeResponse([], status=403))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info, retries=2)
assert 'HTTP 403' in str(ei.value)
assert len(session.calls) == 1, '确定性失败对同一失效直链白重试毫无意义'
def test_408_is_retried_then_reported_as_failure(env):
dl, _state, install = env
info = _info(size=20)
session = install(lambda a, u, kw: _FakeResponse([], status=408))
r = dl.download_music_file(101, music_info=info, retries=1)
assert not r.success
assert '下载请求失败' in r.error_message
assert len(session.calls) == 2, '408 请求超时应保留可重试'
def test_5xx_is_retried_and_keeps_part(env):
"""5xx 是服务端瞬时故障(非确定性失败):应可重试,且保留 .part 供续传。
与 4xx 的分界在这里钉住:error 分类只看 400<=status<500,5xx 落回裸重抛,
由外层 RequestException 分支退避重试。此前这条路径无测试覆盖。
"""
dl, _state, install = env
info = _info(size=100)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
session = install(lambda a, u, kw: _FakeResponse([], status=500))
r = dl.download_music_file(101, music_info=info, retries=1)
assert not r.success and '下载请求失败' in r.error_message
assert len(session.calls) == 2, '5xx 非确定性失败,应重试'
assert not fp.exists(), '未交付不得留下最终文件'
assert part.stat().st_size == 10, '.part 应保留供下轮续传'
def test_incomplete_size_keeps_part_and_reports(env):
dl, _state, install = env
info = _info(size=100)
fp, part = _paths(dl, info)
install(_body(10))
r = dl.download_music_file(101, music_info=info, retries=0)
assert not r.success and '下载不完整' in r.error_message
assert part.exists() and part.stat().st_size == 10, '.part 应保留以供续传'
assert not fp.exists(), '不完整的下载不得转正交付'
def test_incomplete_then_resumes_on_next_attempt(env):
dl, _state, install = env
info = _info(size=100)
_fp, part = _paths(dl, info)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'a' * 10], status=200)
return _FakeResponse([b'b' * 5], status=206,
headers={'Content-Range': 'bytes 10-14/100'})
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert not r.success and '下载不完整' in r.error_message
assert session.calls[1][1]['headers'].get('Range') == 'bytes=10-', '下轮应续传'
assert part.stat().st_size == 15
def test_network_error_keeps_part_for_resume(env):
dl, _state, install = env
info = _info(size=100)
fp, part = _paths(dl, info)
def chunks():
yield b'a' * 10
raise sync_http.RequestException('连接中断')
install(lambda a, u, kw: _FakeResponse(chunks()))
r = dl.download_music_file(101, music_info=info, retries=0)
assert not r.success and '下载请求失败' in r.error_message
assert part.exists() and part.stat().st_size == 10, '网络错误应保留 .part'
assert not fp.exists()
# ============ 内容校验与体积上限 ============
def test_unknown_size_non_audio_content_rejected(env):
dl, state, install = env
info = _info(size=0)
fp, part = _paths(dl, info)
state['sniff'] = False
install(lambda a, u, kw: _FakeResponse([b'<html>blocked</html>']))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info)
assert '非音频数据' in str(ei.value)
assert not part.exists() and not fp.exists()
def test_unknown_size_audio_content_accepted(env):
dl, state, install = env
info = _info(size=0)
fp, _ = _paths(dl, info)
state['sniff'] = True
install(lambda a, u, kw: _FakeResponse([b'ID3audio']))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'ID3audio'
def test_size_limit_discards_part_and_fails(env, monkeypatch):
dl, _state, install = env
monkeypatch.setattr('music_downloader.downloader._MAX_DOWNLOAD_BYTES', 16)
info = _info(size=0)
fp, part = _paths(dl, info)
install(lambda a, u, kw: _FakeResponse([b'x' * 100]))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info)
assert '大小上限' in str(ei.value)
assert not part.exists() and not fp.exists(), '超限残片无续传价值,须清理'
# ============ 取消 ============
def test_cancel_before_transfer_clears_part_and_skips_network(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'p' * 5)
cancel = threading.Event()
cancel.set()
session = install(lambda *a: pytest.fail('已取消不应发网络请求'))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel)
assert session.calls == []
assert not part.exists(), '取消须清掉 .part'
def test_cancel_midway_clears_half_written_file(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def chunks():
yield b'a' * 5
cancel.set()
yield b'b' * 15
install(lambda a, u, kw: _FakeResponse(chunks()))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel)
assert not part.exists() and not fp.exists()
def test_network_error_while_canceled_reports_canceled(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def chunks():
yield b'a' * 5
cancel.set()
raise OSError('句柄异常')
install(lambda a, u, kw: _FakeResponse(chunks()))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel, retries=2)
assert not part.exists() and not fp.exists(), '已取消时残片没有续传价值'
def test_cancel_after_promote_keeps_file_but_skips_metadata(env):
dl, state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def cb(written, _total):
if written >= 20:
cancel.set()
install(_body(20))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel,
progress_cb=cb)
assert fp.exists() and not part.exists(), '已转正的文件不回滚'
assert state['meta'] == [], '取消后不应再写元数据'
# ============ 元数据 ============
def test_metadata_written_only_when_requested(env):
dl, state, install = env
install(_body(20))
dl.download_music_file(101, music_info=_info(mid=101), write_meta=False)
assert state['meta'] == [], 'write_meta=False 不应写元数据'
r = dl.download_music_file(102, music_info=_info(mid=102), write_meta=True,
meta_fields=['title'])
assert r.success
assert [f for _p, f in state['meta']] == [['title']], 'fields 应原样透传'