This repository has been archived on 2026-10-02. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
NeteaseUrl/tests/test_downloader.py
T
snowy-yang 8716617d5d refactor: download_music_file 拆解(205 行 / 圈 47 → 125 行 / 圈 20)
为什么:download_music_file 是**唯一**的音频落盘路径(单曲 worker、音乐批量 ZIP
都经它),205 行 / 圈复杂度 47 / 嵌套 7 的单体里叠着缓存校验、断点续传、内容嗅探、
体积上限、取消语义、重试退避、元数据限流七件事,却**零直接测试覆盖**
(test_zip_workers 里只有一个 stub)。任何一处被改坏都不会有测试报警。

改法:拆成 8 个具名方法,主函数只留「5 步管线 + 重试策略」:
  _cached_result          锁内复查缓存(命中即返回,坏缓存删除重下)
  _promote_complete_part  已下完未改名的 .part 直接转正,不发起网络请求
  _resume_offset          断点续传起点,并清理没有续传价值的残片
  _transfer               占全局名额 + 选通道(aria2c / 内置 aiohttp)
  _size_mismatch_error    完整性判定(已知期望精确相等 / 未知期望至少非空)
  _publish                嗅探 → 原子转正 → 写元数据
  _write_metadata_locked  标签信号量内的元数据写入(原三处重复的 with 收敛)
  _backoff_wait           退避秒数(原五处重复的 0.5*(attempt+1) 收敛)

复杂度(同口径 AST 实测):
  download_music_file  205 行 / 圈 47 / 嵌套 7  →  125 行 / 圈 20 / 嵌套 4
                       (125 行里 26 行是 docstring,正文 87 行)
  文件内嵌套 >=5 层     2 → 1(余下的是本次未动的 _fetch_via_http)
  文件内最大圈复杂度     47 → 35(同样落在未动的 _fetch_via_http 上)
  文件行数             614 → 710(+96,全是新签名与 docstring)

等价性证据(两道,互相独立):
1. 29 例行为契约测试**在未改动的原始实现上一次性全绿**(先写测试、后动代码;改完
   再跑仍全绿)——这批测试因此忠实编码了旧行为,构成等价性验证,而不是「照着新
   实现写的新行为测试」。断言只依赖返回值 / 磁盘上的文件 / 发往网络的请求,不引用
   任何内部函数名,故可跨后续重构复用。
2. AST 字面量多重集比对 + **逐条位置定位**:数值/布尔 27 种、字符串 80 种。差异
   全部归因,且**没有一条用户可见文案或阈值被改动**——
     0   +2:_resume_offset 的三个 return 0 取代原 resume_from = 0
     0.0 +1、0.5 −3、1 −3:四处重复的 0.5*(attempt+1) 收敛进 _backoff_wait
     None +16:新增方法的 6 处类型标注 + 4 处 return None + 2 处 is not None
     ''  +1:_size_mismatch_error 的内部哨兵(非用户可见)
   续传起点逻辑另做逐例核对:promote 命中 / existing<=0 / 超尺寸 / 部分 四路等价。

刻意保留的语义(是决定,不是疏漏):
- 退避睡眠仍在文件锁外执行——同名并发任务不陪睡。
- 缓存命中时若 write_meta 仍按本次请求重跑一次标签写入(幂等);write_meta=False
  则不写。
- 已转正(os.replace 之后)才发现取消时**不回滚文件**,只跳过元数据——原行为如此,
  已用 test_cancel_after_promote_keeps_file_but_skips_metadata 固化。
- 未知期望尺寸(file_size=0)走「非空即通过」+ 内容嗅探兜底;已知期望尺寸要求
  **精确相等**(放宽到容差会交付截断的坏文件还照扣次数)。
- 缓存有效性用 st_size != file_size(而非 >=):跨账号音质降级 / 上游变更都会让
  同 ID 同音质的实际文件不同,命中即返回会把旧文件当本次结果交付。

测试:新增 tests/test_downloader.py(29 例)——正常落盘与 .part 转正、空分块跳过、
进度回调(起点 / 每块 / 异常吞掉)、缓存命中不发请求、坏缓存重下、0 字节缓存拒绝、
续传发 Range 并追加、服务器忽略 Range 时重写、完整残片免网络转正、超尺寸残片丢弃、
206 起点错位 / 206 缺 Content-Range / 416 三路从头重下、CDN 4xx 不重试、408 可重试、
不完整保留 .part、下轮续传、未知尺寸非音频拒绝、体积上限、五种取消路径、元数据开关。

验证:ruff 全过 · pyright 0/0 · pytest 274 passed(245→274)· 生产 data/ 未被触碰
(全部用例走 tmp_path 沙箱)。
2026-09-27 07:56:01 +08:00

582 lines
19 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# -*- coding: utf-8 -*-
"""MusicDownloader.download_music_file 的行为契约测试(沙箱,不触碰生产 data/)。
为什么:download_music_file 是**唯一**的音频落盘路径——单曲 worker 与音乐批量
ZIP 都经它,约 205 行 / 圈复杂度 47,此前**零直接测试覆盖**(test_zip_workers 里
只有一个 stub)。它同时承担缓存校验、断点续传、内容嗅探、体积上限、取消语义、
重试退避与元数据写入,任何一条被拆坏都不会有测试报警。
断言只依赖「调用后世界里发生了什么」——返回值、磁盘上的文件、发往网络的请求——
不引用任何内部函数名,因此本文件可作为后续拆分的等价性验证器。
替身约定(三处,均为模块级符号,替换点在 downloader 模块命名空间):
- 网络:``get_http_session`` / ``get_backend``(后者返回 None = 走内置 HTTP 通道)
- 内容嗅探:``_sniff_audio_header``(默认放行,按需置 False)
- 元数据写入:``MusicDownloader._write_metadata``(不真跑 mutagen)
"""
import threading
from pathlib import Path
from typing import Any
import pytest
from core import sync_http
from music_downloader import DownloadCanceled, DownloadException, MusicInfo
from music_downloader.downloader import MusicDownloader
# ============ 替身 ============
class _FakeResponse:
"""core.sync_http.SyncResponse 的替身。
chunks 保留原始 iterable(**不要** list() 归一化):生成器的副作用
(如中途置取消位)若在构造时刻就求值,「中途取消」会被悄悄测成
「开始前取消」——两条不同的路径混成一条,而测试仍是绿的。
"""
def __init__(self, chunks, status=200, headers=None):
self._chunks = chunks
self.status_code = status
self.headers = dict(headers or {})
self.closed = False
def iter_content(self, chunk_size=0):
yield from self._chunks
def raise_for_status(self):
if self.status_code >= 400:
raise sync_http.HTTPError(f'HTTP {self.status_code}', self)
def close(self):
self.closed = True
class _FakeSession:
"""记录每次 GET(含 headers)并按序产出响应。"""
def __init__(self, responder):
self._responder = responder
self.calls: list[tuple[str, dict[str, Any]]] = []
self.closed = False
def get(self, url, **kw):
self.calls.append((url, kw))
return self._responder(len(self.calls), url, kw)
def close(self):
self.closed = True
def _body(size, status=200, content_range=None):
"""固定返回 size 字节的响应工厂(每次调用产出新响应对象)。"""
def _mk(_attempt, _url, _kw):
headers = {'Content-Range': content_range} if content_range else {}
return _FakeResponse([b'x' * size], status=status, headers=headers)
return _mk
def _info(mid=101, size=20, file_type='mp3'):
return MusicInfo(
id=mid, name='测试曲', artists='歌手', album='专辑', pic_url='',
duration=180, track_number=1, download_url='http://cdn.test/a.mp3',
file_type=file_type, file_size=size, quality='standard',
)
def _paths(dl, info):
"""按生产同一函数推算落盘路径与 .part 路径(纯函数,可直接复用)。"""
fp = dl._build_file_path(info)
return fp, fp.with_name(fp.name + '.part')
@pytest.fixture()
def env(tmp_path, monkeypatch):
"""返回 (downloader, state, install)。
- state['sniff']:内容嗅探结果,默认 True
- state['meta']:_write_metadata 的调用记录 [(文件路径, fields), ...]
- install(responder) -> _FakeSession:安装网络替身,返回该替身以便断言
"""
state: dict[str, Any] = {'sniff': True, 'meta': []}
box: dict[str, _FakeSession] = {}
def install(responder):
box['session'] = _FakeSession(responder)
return box['session']
def _record_meta(self, file_path, music_info, fields=None):
state['meta'].append((str(file_path), fields))
return True
monkeypatch.setattr('music_downloader.downloader.get_http_session',
lambda: box['session'])
monkeypatch.setattr('music_downloader.downloader.get_backend', lambda: None)
monkeypatch.setattr('music_downloader.downloader._sniff_audio_header',
lambda _p: state['sniff'])
monkeypatch.setattr(MusicDownloader, '_write_metadata', _record_meta)
install(lambda *a: pytest.fail('本用例不应发出网络请求'))
dl = MusicDownloader(download_dir=str(tmp_path))
return dl, state, install
# ============ 正常落盘 ============
def test_success_promotes_part_to_final(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.error_message == ''
assert Path(r.file_path) == fp
assert r.file_size == 20 and fp.read_bytes() == b'x' * 20
assert not part.exists(), '完成后 .part 应改名转正'
def test_empty_chunks_skipped(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
install(lambda a, u, kw: _FakeResponse([b'', b'x' * 20, b'']))
seen: list[int] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, _t: seen.append(w))
assert r.success and fp.stat().st_size == 20
assert seen == [0, 20], '空分块不写盘也不触发进度回调'
def test_progress_reports_resume_start_then_each_chunk(env):
dl, _state, install = env
info = _info(size=20)
install(lambda a, u, kw: _FakeResponse([b'a' * 8, b'b' * 12]))
seen: list[tuple[int, int]] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, t: seen.append((w, t)))
assert r.success
assert seen == [(0, 20), (8, 20), (20, 20)], '起始回调带续传起点,此后每块一次'
def test_progress_callback_exception_never_breaks_download(env):
dl, _state, install = env
info = _info(size=20)
install(_body(20))
def _boom(_w, _t):
raise RuntimeError('回调炸了')
r = dl.download_music_file(101, music_info=info, progress_cb=_boom)
assert r.success, '进度回调异常绝不影响下载主流程'
def test_music_info_fetched_when_not_provided(env, monkeypatch):
dl, _state, install = env
info = _info(size=20)
seen: list[tuple] = []
monkeypatch.setattr(dl, 'get_music_info',
lambda mid, q, c=None: (seen.append((mid, q)), info)[1])
install(_body(20))
r = dl.download_music_file(101, 'lossless')
assert r.success and seen == [(101, 'lossless')]
# ============ 缓存校验 ============
def test_cache_hit_skips_network(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
fp.write_bytes(b'c' * 20)
session = install(lambda *a: pytest.fail('缓存命中不应发网络请求'))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.file_size == 20
assert session.calls == []
def test_cache_size_mismatch_triggers_redownload(env):
dl, _state, install = env
info = _info(size=20)
fp, _ = _paths(dl, info)
fp.write_bytes(b'c' * 15) # 与本次期望不符 = 缓存 miss
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'x' * 20
assert len(session.calls) == 1, '坏缓存应被删除并重下,而不是当结果交付'
def test_zero_byte_cache_rejected_when_expected_size_unknown(env):
dl, _state, install = env
info = _info(size=0)
fp, _ = _paths(dl, info)
fp.write_bytes(b'')
session = install(lambda a, u, kw: _FakeResponse([b'good']))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'good'
assert len(session.calls) == 1, '期望未知时至少拒绝 0 字节坏文件'
# ============ 断点续传 ============
def test_resume_sends_range_and_appends(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
session = install(lambda a, u, kw: _FakeResponse(
[b'q' * 10], status=206, headers={'Content-Range': 'bytes 10-19/20'}))
seen: list[tuple[int, int]] = []
r = dl.download_music_file(101, music_info=info,
progress_cb=lambda w, t: seen.append((w, t)))
assert r.success
assert session.calls[0][1]['headers'].get('Range') == 'bytes=10-'
assert fp.read_bytes() == b'p' * 10 + b'q' * 10, '续传应追加而非覆盖'
assert seen[0] == (10, 20), '起始进度应带续传起点'
def test_server_ignoring_range_rewrites_from_zero(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
install(lambda a, u, kw: _FakeResponse([b'r' * 20], status=200))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'r' * 20, '服务器返回 200 须从头重写'
def test_complete_part_promoted_without_network(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'z' * 20) # 已下完但未改名
session = install(lambda *a: pytest.fail('完整残片不应发网络请求'))
r = dl.download_music_file(101, music_info=info)
assert r.success and r.file_size == 20
assert session.calls == []
assert fp.read_bytes() == b'z' * 20 and not part.exists()
def test_oversized_part_discarded_starts_from_zero(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'o' * 30) # 超尺寸残片没有续传价值
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success
assert 'Range' not in session.calls[0][1]['headers']
def test_zero_byte_part_discarded(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'')
session = install(_body(20))
r = dl.download_music_file(101, music_info=info)
assert r.success and 'Range' not in session.calls[0][1]['headers']
def test_content_range_offset_mismatch_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'q' * 5], status=206,
headers={'Content-Range': 'bytes 3-7/20'})
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers'], '清残片后应从头下'
assert fp.read_bytes() == b'f' * 20
def test_206_without_content_range_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'q' * 5], status=206) # 头被中间层剥离
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers']
def test_http_416_restarts_from_scratch(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
part.write_bytes(b'p' * 10)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([], status=416)
return _FakeResponse([b'f' * 20], status=200)
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert r.success and len(session.calls) == 2
assert 'Range' not in session.calls[1][1]['headers']
assert fp.read_bytes() == b'f' * 20
# ============ 失败与重试 ============
def test_cdn_4xx_is_deterministic_and_not_retried(env):
dl, _state, install = env
info = _info(size=20)
session = install(lambda a, u, kw: _FakeResponse([], status=403))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info, retries=2)
assert 'HTTP 403' in str(ei.value)
assert len(session.calls) == 1, '确定性失败对同一失效直链白重试毫无意义'
def test_408_is_retried_then_reported_as_failure(env):
dl, _state, install = env
info = _info(size=20)
session = install(lambda a, u, kw: _FakeResponse([], status=408))
r = dl.download_music_file(101, music_info=info, retries=1)
assert not r.success
assert '下载请求失败' in r.error_message
assert len(session.calls) == 2, '408 请求超时应保留可重试'
def test_incomplete_size_keeps_part_and_reports(env):
dl, _state, install = env
info = _info(size=100)
fp, part = _paths(dl, info)
install(_body(10))
r = dl.download_music_file(101, music_info=info, retries=0)
assert not r.success and '下载不完整' in r.error_message
assert part.exists() and part.stat().st_size == 10, '.part 应保留以供续传'
assert not fp.exists(), '不完整的下载不得转正交付'
def test_incomplete_then_resumes_on_next_attempt(env):
dl, _state, install = env
info = _info(size=100)
_fp, part = _paths(dl, info)
def responder(n, _url, _kw):
if n == 1:
return _FakeResponse([b'a' * 10], status=200)
return _FakeResponse([b'b' * 5], status=206,
headers={'Content-Range': 'bytes 10-14/100'})
session = install(responder)
r = dl.download_music_file(101, music_info=info, retries=1)
assert not r.success and '下载不完整' in r.error_message
assert session.calls[1][1]['headers'].get('Range') == 'bytes=10-', '下轮应续传'
assert part.stat().st_size == 15
def test_network_error_keeps_part_for_resume(env):
dl, _state, install = env
info = _info(size=100)
fp, part = _paths(dl, info)
def chunks():
yield b'a' * 10
raise sync_http.RequestException('连接中断')
install(lambda a, u, kw: _FakeResponse(chunks()))
r = dl.download_music_file(101, music_info=info, retries=0)
assert not r.success and '下载请求失败' in r.error_message
assert part.exists() and part.stat().st_size == 10, '网络错误应保留 .part'
assert not fp.exists()
# ============ 内容校验与体积上限 ============
def test_unknown_size_non_audio_content_rejected(env):
dl, state, install = env
info = _info(size=0)
fp, part = _paths(dl, info)
state['sniff'] = False
install(lambda a, u, kw: _FakeResponse([b'<html>blocked</html>']))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info)
assert '非音频数据' in str(ei.value)
assert not part.exists() and not fp.exists()
def test_unknown_size_audio_content_accepted(env):
dl, state, install = env
info = _info(size=0)
fp, _ = _paths(dl, info)
state['sniff'] = True
install(lambda a, u, kw: _FakeResponse([b'ID3audio']))
r = dl.download_music_file(101, music_info=info)
assert r.success and fp.read_bytes() == b'ID3audio'
def test_size_limit_discards_part_and_fails(env, monkeypatch):
dl, _state, install = env
monkeypatch.setattr('music_downloader.downloader._MAX_DOWNLOAD_BYTES', 16)
info = _info(size=0)
fp, part = _paths(dl, info)
install(lambda a, u, kw: _FakeResponse([b'x' * 100]))
with pytest.raises(DownloadException) as ei:
dl.download_music_file(101, music_info=info)
assert '大小上限' in str(ei.value)
assert not part.exists() and not fp.exists(), '超限残片无续传价值,须清理'
# ============ 取消 ============
def test_cancel_before_transfer_clears_part_and_skips_network(env):
dl, _state, install = env
info = _info(size=20)
_fp, part = _paths(dl, info)
part.write_bytes(b'p' * 5)
cancel = threading.Event()
cancel.set()
session = install(lambda *a: pytest.fail('已取消不应发网络请求'))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel)
assert session.calls == []
assert not part.exists(), '取消须清掉 .part'
def test_cancel_midway_clears_half_written_file(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def chunks():
yield b'a' * 5
cancel.set()
yield b'b' * 15
install(lambda a, u, kw: _FakeResponse(chunks()))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel)
assert not part.exists() and not fp.exists()
def test_network_error_while_canceled_reports_canceled(env):
dl, _state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def chunks():
yield b'a' * 5
cancel.set()
raise OSError('句柄异常')
install(lambda a, u, kw: _FakeResponse(chunks()))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel, retries=2)
assert not part.exists() and not fp.exists(), '已取消时残片没有续传价值'
def test_cancel_after_promote_keeps_file_but_skips_metadata(env):
dl, state, install = env
info = _info(size=20)
fp, part = _paths(dl, info)
cancel = threading.Event()
def cb(written, _total):
if written >= 20:
cancel.set()
install(_body(20))
with pytest.raises(DownloadCanceled):
dl.download_music_file(101, music_info=info, cancel_event=cancel,
progress_cb=cb)
assert fp.exists() and not part.exists(), '已转正的文件不回滚'
assert state['meta'] == [], '取消后不应再写元数据'
# ============ 元数据 ============
def test_metadata_written_only_when_requested(env):
dl, state, install = env
install(_body(20))
dl.download_music_file(101, music_info=_info(mid=101), write_meta=False)
assert state['meta'] == [], 'write_meta=False 不应写元数据'
r = dl.download_music_file(102, music_info=_info(mid=102), write_meta=True,
meta_fields=['title'])
assert r.success
assert [f for _p, f in state['meta']] == [['title']], 'fields 应原样透传'