Archived
refactor: 两个 ZIP 打包 worker 合并为一个参数化引擎(604 行重复骨架 → 一份)
为什么:_batch_zip_worker(327 行 / 圈 78 / 嵌套 9)与 _podcast_zip_worker
(277 行 / 圈 84 / 嵌套 9)是两份平行实现——任务状态机、断线暂停、全局批量
名额、「逐条失败不计整批」、取消与异常的四路终态收敛、账本结算全部各写一遍,
连注释都逐字重复。上一轮修的 zip_valid 未绑定缺陷正是同型出现在这两处,
重复代码在持续制造同类 bug。
改法:新增 services/_zipcore.py 只保留共用骨架,差异收敛到 ZipJob 钩子:
prepare / download_one / write_entry / zip_filename 必实现,prefetch /
fail_label / done_msg / log_label / all_failed_msg / partial_msg / supplement /
on_delivered / cleanup 按需覆写。batch.py 与 podcast.py 只剩薄包装 + flavor 类。
线程模型(改动中最容易踩坏的一条,已写进模块 docstring):
ZipFile 不是线程安全的,且包内名去重是「读改写 used_arcnames」。因此把下载与
落包拆成两个钩子——只有 download_one 进线程池,zf.write 与去重一律在调用线程
串行。旧的音乐批量之所以安全,只是因为它的 zf.write 恰好写在主线程的
as_completed 循环里;照搬成「整个下载单元进池」会引入静默的 ZIP 损坏。
复杂度(同口径 AST 实测):
两个 worker 合计 604 行 / 圈 162 / 嵌套 9,9
→ run_zip_job 167 行 / 圈 37 / 嵌套 4
+ _fill_zip 64 行 / 圈 13 / 嵌套 5
全仓最大圈复杂度 84 → 37;>80 行函数 3 → 2(余下两个之一是本次未改动的
_podcast_download_worker)。两文件改为三文件后总行数 930 → 1071,多出的
主要是引擎 docstring——注释比代码贵,但它把「为什么这么拆」留给了下一个人。
逐条对齐的分叉(合并必须做的决定,此处全部列明):
1. 账本终态:播客全失败走自己的 except DownloadException → 落 'settled',音乐
同语义事件落 'error'。统一为 'error'。全仓仅读取 'open'(对账/清理/计数),
其余取值只作排障留痕 → 这是本次**唯一可观察的语义变更**,由新增用例固化。
2. 通用异常路径的「部分交付」:音乐侧会发布已打包的 ZIP 并挂 result,播客侧
直接丢弃(却已按 delivered 结算 = 用户扣了费拿不到货)。统一采用音乐侧口径。
该路径实际不可达(ZIP 写完后到发布前只有取消检查会抛错,已被取消分支接管)。
3. 全失败判定:音乐用 `ok == 0 and failed`、播客用早抛 DownloadException。
统一为显式 `ok == 0`——好处是不再发布空 ZIP、不再把空批次标 done。
4. 早抛业务失败(无可下载条目 / 补扣失败):播客走 DownloadException 分支,
音乐无此分支(落通用异常,用户看到的是「打包失败,请稍后重试」)。统一保留
该分支,音乐侧同类失败从此也给出具体原因。
5. 错误终态写法:播客在手写任务字典(重复一遍「已终态不覆盖」,且漏掉 /F-E 的
「worker 线程已死则按账本补结算」兜底),统一走 _mark_task_terminal。
6. 暂停检查点:统一采用音乐侧更严的一处——拿到 BATCH_ZIP_SEM 后再查一次
pause/cancel。播客此前会多下完一集才停。
7. total 字段:音乐由提交侧设置、播客由 worker 设置,统一为 worker 设
len(items)(对音乐是同一个值)。
8. 日志文案统一(tag / 名词提成类属性):'批量打包异常' 与 '批量ZIP任务异常'
两种前缀收敛为一种;完成日志统一带上 task_id 与退回次数。
删除的不可达分支(可证明):
两个 worker 在 zip_valid = True 之后各留了一段 `if not zip_valid:` 失败收敛
(音乐 16 行 / 播客 18 行,含「ZIP 打包失败(可能磁盘空间不足)…」文案)。
赋值紧跟其后、中间无任何语句可改写 zip_valid,故恒不可达——是原注释所称
「复审 修正」留下的护栏。zip_valid 在异常路径仍有意义(决定 delivered 是否为 0)。
等价性证据(三道,互相独立):
1. AST 字面量多重集比对 + 逐条位置定位:数值 16 种一个不少。少掉的 8 处 `0`
全部来自上面删的不可达分支与重复字面量合并(`delivered = … else 0` 5 处→3 处)。
2. 反向注入验证:把 git HEAD 的旧 worker 注入成 services.batch / services.podcast,
用同一套 tests/test_zip_workers.py 去跑 → 第 1 批 9 例行为契约**全过**,证明
这套测试忠实编码了旧行为、因而对新实现构成等价性验证;新增 4 例中只有账本
终态那 1 例失败('settled' == 'error'),恰好把上面第 1 条对齐实证出来。该用例
里「全部单集下载失败: …(已退回 2 次下载次数)」的文案断言在旧实现上也通过
→ 用户可见文案逐字保留。
3. 新增 4 例固化测试:同名歌声的音频/歌词/封面三条目共享序号后缀(F-C)、
两边的全失败文案与账本终态、单集失败后任务目录不留中间文件与 .part 残片。
顺带修掉一个自己引入的缺陷:引擎化后播客侧的单条失败文案本会退化成整个条目
字典,字面量审计里 '(' ')' 各少 1 处把它逼了出来 → 补 ZipJob.fail_label 覆盖。
验证:ruff 全过 · pyright 0/0 · pytest 213 passed(209→213)· 生产 data/ 未被
触碰(全部用例走 tmp_path 沙箱)。
This commit is contained in:
1 parent
65a81af94a
commit
5c8f941d19
4 files changed
+848
-579
No files matched your search
@@ -0,0 +1,439 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""批量 ZIP 打包引擎:两个打包 worker 的共用骨架。
|
||||||
|
|
||||||
|
音乐歌单(``services.batch``)与播客整台(``services.podcast``)的打包 worker
|
||||||
|
此前是两份约 300 行的平行实现——任务状态机、断线暂停、全局批量名额、
|
||||||
|
「逐条失败不计整批」、取消与异常的四路终态收敛、账本结算全部各写一遍,
|
||||||
|
连注释都逐字重复(上一轮修掉的 ``zip_valid`` 未绑定缺陷正是同型出现在这两处)。
|
||||||
|
真正不同的只有五件事:清单来源、单条怎么下、包内条目怎么拼、并发模型、
|
||||||
|
交付包名与文案。
|
||||||
|
|
||||||
|
因此本模块保留共用骨架,把上述差异收敛到 :class:`ZipJob`;引擎内任何一处
|
||||||
|
修订(账本口径、取消检查点、结算幂等)从此只需改一次。
|
||||||
|
|
||||||
|
术语:下文「条目」指一个下载单元——音乐歌单里是一首歌,播客整台里是一集。
|
||||||
|
|
||||||
|
线程模型(本轮最容易踩坏的地方,改动前请先读):
|
||||||
|
- 下载 I/O 在 ``ThreadPoolExecutor`` 里跑,窗口大小 = ``job.concurrency``;
|
||||||
|
- **ZIP 写入与包内名去重只发生在调用线程**(``as_completed`` 消费结果时),
|
||||||
|
:class:`zipfile.ZipFile` 不是线程安全的,且去重读改写 ``used_arcnames``
|
||||||
|
需要串行——故下载与落包被拆成 ``download_one`` / ``write_entry`` 两个钩子,
|
||||||
|
只有前者进线程池。
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import traceback
|
||||||
|
import zipfile
|
||||||
|
from collections.abc import Callable
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from core.state import (
|
||||||
|
BATCH_DURATIONS,
|
||||||
|
BATCH_DURATIONS_LOCK,
|
||||||
|
DOWNLOAD_TASKS,
|
||||||
|
DOWNLOAD_TASKS_LOCK,
|
||||||
|
)
|
||||||
|
from music_downloader import (
|
||||||
|
DownloadCanceled,
|
||||||
|
DownloadException,
|
||||||
|
)
|
||||||
|
|
||||||
|
from ._shared import (
|
||||||
|
BATCH_ZIP_SEM,
|
||||||
|
)
|
||||||
|
from ._taskcore import (
|
||||||
|
_mark_task_terminal,
|
||||||
|
_settle_batch_billing,
|
||||||
|
_task_work_dir,
|
||||||
|
_wait_if_paused,
|
||||||
|
)
|
||||||
|
from .api_service import (
|
||||||
|
api_service,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ItemFailed(Exception):
|
||||||
|
"""单个条目失败(不影响整批)。
|
||||||
|
|
||||||
|
``label`` 已是最终展示文案(引擎不再拼后缀)。与
|
||||||
|
:class:`~music_downloader.DownloadCanceled` 必须区分——取消要中止整批,
|
||||||
|
本异常只记一次失败。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, label: str) -> None:
|
||||||
|
super().__init__(label)
|
||||||
|
self.label = label
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ZipContext:
|
||||||
|
"""单次打包运行中,引擎交给 ZipJob 的上下文(引擎持有,任务侧只读)。"""
|
||||||
|
|
||||||
|
task_id: str
|
||||||
|
params: dict[str, Any]
|
||||||
|
cancel_ev: threading.Event
|
||||||
|
work_dir: Path
|
||||||
|
zf: zipfile.ZipFile
|
||||||
|
progress: Callable[..., None]
|
||||||
|
total: int
|
||||||
|
used_arcnames: set[str] = field(default_factory=set)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _Tally:
|
||||||
|
"""成功/失败计数。
|
||||||
|
|
||||||
|
两个 worker 此前把 ``ok_count`` 当外层局部变量,异常处理器读到的是「异常
|
||||||
|
发生那一刻」的值(``zip_valid`` 为假时该值不影响退款,但日志里的「成功N」
|
||||||
|
会失真)。收成一个对象后,异常路径读到的恒是真实值。
|
||||||
|
"""
|
||||||
|
|
||||||
|
ok: int = 0
|
||||||
|
failed: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class ZipJob:
|
||||||
|
"""一次批量打包任务的「因任务而异」部分。
|
||||||
|
|
||||||
|
子类至少实现 :meth:`prepare` / :meth:`download_one` / :meth:`write_entry` /
|
||||||
|
:meth:`zip_filename`,其余钩子按需覆写。**实例每次运行新建**——下载器信息
|
||||||
|
预取池等运行期状态就挂在实例上。
|
||||||
|
"""
|
||||||
|
|
||||||
|
tag = '批量打包' # 日志前缀
|
||||||
|
noun = '项' # 计数单位(文案里的「N 首/N 集」)
|
||||||
|
item_noun = '条目' # 全称(文案里的「部分xx已打包完成」)
|
||||||
|
tmp_prefix = 'batch_zip_' # 临时 ZIP 文件名前缀
|
||||||
|
start_msg = '打包中...' # 进入 running 时的进度文案
|
||||||
|
concurrency = 1 # 每窗口条目数(1 = 顺序)
|
||||||
|
record_duration = False # 是否把耗时记入 BATCH_DURATIONS(供前端估算等待)
|
||||||
|
|
||||||
|
# ---- 必须实现 ----
|
||||||
|
|
||||||
|
def prepare(self, params: dict[str, Any], task_id: str, failed: list[str]) -> list[Any]:
|
||||||
|
"""组装条目清单。可抛 ``DownloadException`` 让任务按业务失败收敛。
|
||||||
|
|
||||||
|
``failed`` 可被直接追加——清单阶段就有条目已知失败(如按 ID 列表时个别
|
||||||
|
详情拉取失败),这些失败要与下载阶段的一起计入。
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def download_one(self, item: Any, idx: int, ctx: ZipContext) -> Any:
|
||||||
|
"""下载单个条目(**在下载线程池内执行**)。
|
||||||
|
|
||||||
|
返回的 ``payload`` 会原样交给 :meth:`write_entry`(放临时文件名、
|
||||||
|
已取到的元信息等)。失败请抛异常:``ItemFailed`` 用其自带文案,
|
||||||
|
其他异常由引擎按 ``fail_label(item)(异常)`` 记录。
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def write_entry(self, item: Any, idx: int, payload: Any, ctx: ZipContext) -> None:
|
||||||
|
"""把下载结果写入 ZIP(**在引擎调用线程内执行,天然串行**)。
|
||||||
|
|
||||||
|
包内名去重在这里做——需要读写 ``ctx.used_arcnames``,必须串行。
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def zip_filename(self, ok_count: int) -> str:
|
||||||
|
"""交付 ZIP 的文件名。"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
# ---- 可选钩子 ----
|
||||||
|
|
||||||
|
def prefetch(self, window: list[Any], ctx: ZipContext) -> None:
|
||||||
|
"""窗口级预热(每窗口一次、提交下载前调用)。默认无操作。"""
|
||||||
|
|
||||||
|
def fail_label(self, item: Any) -> str:
|
||||||
|
"""单条失败文案前缀(引擎拼成 ``f'{label}({异常})'``)。"""
|
||||||
|
return str(item)
|
||||||
|
|
||||||
|
def done_msg(self, item: Any, idx: int, ctx: ZipContext) -> str:
|
||||||
|
"""条目成功后的进度文案(空串 = 沿用当前文案)。默认不追加文案。"""
|
||||||
|
return ''
|
||||||
|
|
||||||
|
def log_label(self) -> str:
|
||||||
|
"""日志里区隔本次任务的名字(如电台名),默认空串。"""
|
||||||
|
return ''
|
||||||
|
|
||||||
|
def all_failed_msg(self, failed: list[str], refunded: int) -> str:
|
||||||
|
"""全部条目失败时的用户可见文案(含退费说明)。"""
|
||||||
|
return f"全部失败({len(failed)} {self.noun})" + (
|
||||||
|
f",已退回 {refunded} 次" if refunded else '')
|
||||||
|
|
||||||
|
def partial_msg(self, refunded: int) -> str:
|
||||||
|
"""已打包部分交付时的用户可见文案(仅异常路径会用到)。"""
|
||||||
|
return f"部分{self.item_noun}已打包完成,可领取" + (
|
||||||
|
f",退回 {refunded} 次" if refunded else '')
|
||||||
|
|
||||||
|
def supplement(self, task_id: str, params: dict[str, Any], count: int) -> None:
|
||||||
|
"""清单确定后的计费补扣(默认无操作)。"""
|
||||||
|
|
||||||
|
def on_delivered(self, items: list[Any], task_id: str, params: dict[str, Any]) -> None:
|
||||||
|
"""交付落定后的操作记录(默认无操作)。"""
|
||||||
|
|
||||||
|
def cleanup(self) -> None:
|
||||||
|
"""释放运行期资源(如信息预取池)。默认无操作。"""
|
||||||
|
|
||||||
|
|
||||||
|
def _discard(path: str | None) -> None:
|
||||||
|
"""删除未发布的临时 ZIP(坏包/空包/取消包)——失败静默(清理非关键路径)。"""
|
||||||
|
if not path:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
os.remove(path)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def run_zip_job(job: ZipJob, task_id: str, *,
|
||||||
|
settle: Callable[[str, int, str], int] = _settle_batch_billing) -> None:
|
||||||
|
"""执行一次批量打包任务(在 worker 线程内调用,自身不向外抛异常)。
|
||||||
|
|
||||||
|
``settle`` 由调用方注入,默认走 :func:`services._taskcore._settle_batch_billing`。
|
||||||
|
做成参数是为了让测试能在**模块级**替换结算函数(包装层在调用时读取模块全局,
|
||||||
|
替换即生效)。
|
||||||
|
"""
|
||||||
|
zip_tmp_path: str | None = None
|
||||||
|
# zip_valid 必须在函数头初始化:ZIP 构建前的业务错误(参数缺失/无条目/
|
||||||
|
# 补扣失败)会 early raise,异常处理器若引用未绑定名会 NameError 二次崩线程
|
||||||
|
zip_valid = False
|
||||||
|
tally = _Tally()
|
||||||
|
items: list[Any] = []
|
||||||
|
try:
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
task = DOWNLOAD_TASKS[task_id]
|
||||||
|
task['status'] = 'running'
|
||||||
|
task['msg'] = job.start_msg
|
||||||
|
params = task['params']
|
||||||
|
cancel_ev = task['cancel']
|
||||||
|
pause_ev = task['pause']
|
||||||
|
# 计耗时只算「下载+打包」这一段(不含排队),故不用 task['start']
|
||||||
|
t0 = time.time()
|
||||||
|
|
||||||
|
items = job.prepare(params, task_id, tally.failed)
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
DOWNLOAD_TASKS[task_id]['total'] = len(items)
|
||||||
|
job.supplement(task_id, params, len(items))
|
||||||
|
|
||||||
|
def _progress(msg: str = '') -> None:
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS[task_id]
|
||||||
|
t['done'] = tally.ok + len(tally.failed)
|
||||||
|
t['failed'] = len(tally.failed)
|
||||||
|
if msg:
|
||||||
|
t['msg'] = msg
|
||||||
|
|
||||||
|
# ZIP 构建放 downloads 卷(原实现落系统 %TEMP%——500 首无损可达 GB 级,
|
||||||
|
# 小系统盘一次大单即塞爆),且建在任务专属目录(见 _task_work_dir):
|
||||||
|
# downloads 根目录会被缓存清扫按 mtime 回收
|
||||||
|
work_dir = _task_work_dir(task_id)
|
||||||
|
with tempfile.NamedTemporaryFile(prefix=job.tmp_prefix, suffix='.zip',
|
||||||
|
dir=str(work_dir), delete=False) as zip_tmp:
|
||||||
|
zip_tmp_path = zip_tmp.name
|
||||||
|
# 音频本身已压缩,条目按 ZIP_STORED(免压缩)大幅提速;文本条目仍压缩
|
||||||
|
with zipfile.ZipFile(zip_tmp, 'w', zipfile.ZIP_DEFLATED) as zf:
|
||||||
|
ctx = ZipContext(task_id=task_id, params=params, cancel_ev=cancel_ev,
|
||||||
|
work_dir=work_dir, zf=zf, progress=_progress,
|
||||||
|
total=len(items))
|
||||||
|
_fill_zip(job, items, ctx, tally, task, pause_ev)
|
||||||
|
# zip_valid 只在 ZIP with 块正常退出(中央目录完整写出)后才置 True。
|
||||||
|
# 此前两个 worker 在这之后还各留了一段 `if not zip_valid:` 失败处理——
|
||||||
|
# 紧跟赋值、中间无任何可改zip_valid的语句,是**不可达**的复审残留,已删。
|
||||||
|
zip_valid = True
|
||||||
|
|
||||||
|
if job.record_duration and tally.ok > 0:
|
||||||
|
with BATCH_DURATIONS_LOCK:
|
||||||
|
BATCH_DURATIONS.append(time.time() - t0)
|
||||||
|
|
||||||
|
# 全部失败(含清单非空但一条都没成):不发布空 ZIP、不按交付结算,
|
||||||
|
# 转 error 并全额退费。原音乐批量用 `ok==0 and failed`、播客用早抛
|
||||||
|
# DownloadException——这里统一为显式收敛,两边的账本终态一致
|
||||||
|
if tally.ok == 0:
|
||||||
|
_discard(zip_tmp_path)
|
||||||
|
zip_tmp_path = None
|
||||||
|
refunded = settle(task_id, 0, 'error')
|
||||||
|
_mark_task_terminal(task_id, 'error', job.all_failed_msg(tally.failed, refunded))
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
if t is not None:
|
||||||
|
t['failed'] = len(tally.failed)
|
||||||
|
t['refunded'] = refunded
|
||||||
|
api_service.logger.info(
|
||||||
|
f"{job.tag}全部失败: {task_id} 失败{len(tally.failed)} 退回{refunded}次")
|
||||||
|
return
|
||||||
|
|
||||||
|
# 正常交付:按入包成功数结算('settled' 是 _settle_batch_billing 的默认值,
|
||||||
|
# 此处显式写出,让「成功/取消/失败」三种账本终态在本文件里一目了然)
|
||||||
|
refunded = settle(task_id, tally.ok, 'settled')
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
# 发布前锁内复查取消/失效状态——取消请求落在「检查后、拿锁前」的窗口
|
||||||
|
# 也不得覆盖终态
|
||||||
|
if t is None or t.get('status') in ('canceled', 'error'):
|
||||||
|
raise DownloadCanceled("任务已取消")
|
||||||
|
if cancel_ev.is_set():
|
||||||
|
raise DownloadCanceled("任务已取消")
|
||||||
|
t['status'] = 'done'
|
||||||
|
t['completed_at'] = time.time()
|
||||||
|
t['done'] = tally.ok + len(tally.failed)
|
||||||
|
t['failed'] = len(tally.failed)
|
||||||
|
t['msg'] = f"成功{tally.ok} 失败{len(tally.failed)}" + (
|
||||||
|
f" 退回{refunded}次" if refunded else '')
|
||||||
|
t['refunded'] = refunded
|
||||||
|
t['result'] = {
|
||||||
|
'zip_path': zip_tmp_path,
|
||||||
|
'filename': job.zip_filename(tally.ok),
|
||||||
|
'audio_files': [], # 缓存文件不归任务所有,不进入清理清单
|
||||||
|
}
|
||||||
|
job.on_delivered(items, task_id, params)
|
||||||
|
api_service.logger.info(
|
||||||
|
f"{job.tag}完成: {job.log_label()}成功{tally.ok} 失败{len(tally.failed)} "
|
||||||
|
f"退回{refunded}次 ({task_id})")
|
||||||
|
except DownloadCanceled:
|
||||||
|
# 部分成功后取消——已入包部分照常交付(成果守恒),未完成部分按账本退费。
|
||||||
|
# ZIP 未正常完成(写坏)时不按 ok_count 交付——按全额退费收敛
|
||||||
|
delivered = tally.ok if zip_valid else 0
|
||||||
|
refunded = settle(task_id, delivered, 'canceled')
|
||||||
|
if delivered > 0 and zip_tmp_path:
|
||||||
|
_mark_task_terminal(task_id, 'canceled',
|
||||||
|
f"已取消(已打包 {tally.ok} {job.noun}可领取)" +
|
||||||
|
(f",退回 {refunded} 次" if refunded else ''))
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
if t is not None:
|
||||||
|
t['refunded'] = refunded
|
||||||
|
t['result'] = {
|
||||||
|
'zip_path': zip_tmp_path,
|
||||||
|
'filename': job.zip_filename(tally.ok),
|
||||||
|
'audio_files': [],
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
_discard(zip_tmp_path)
|
||||||
|
_mark_task_terminal(task_id, 'canceled',
|
||||||
|
'任务已取消' + (
|
||||||
|
f"(已退回 {refunded} 次下载次数)" if refunded else ''))
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
if t is not None:
|
||||||
|
t['refunded'] = refunded
|
||||||
|
api_service.logger.info(
|
||||||
|
f"{job.tag}已取消: {task_id}(成功{tally.ok} 退回{refunded}次)")
|
||||||
|
except DownloadException as e:
|
||||||
|
# 业务失败(清单阶段:无可下载条目 / 补扣下载次数失败 / 上游不可用)。
|
||||||
|
# 文案是我们自己拼的用户可读文本,保留原文(同播客单集下载)
|
||||||
|
delivered = tally.ok if zip_valid else 0
|
||||||
|
refunded = settle(task_id, delivered, 'error')
|
||||||
|
_apply_error(task_id, str(e), refunded)
|
||||||
|
_discard(zip_tmp_path)
|
||||||
|
api_service.logger.error(
|
||||||
|
f"{job.tag}失败: {task_id} {e}(成功{tally.ok} 退回{refunded}次)")
|
||||||
|
except Exception as e:
|
||||||
|
# 非预期异常原文 logger 先行(结算本身可能再抛,先落下 traceback),
|
||||||
|
# 用户可见文案泛化;带上退款结果再补一条,避免排障时只看到「异常」
|
||||||
|
api_service.logger.error(f"{job.tag}异常: {task_id} {e}\n{traceback.format_exc()}")
|
||||||
|
delivered = tally.ok if zip_valid else 0
|
||||||
|
refunded = settle(task_id, delivered, 'error')
|
||||||
|
if delivered > 0 and zip_tmp_path:
|
||||||
|
_apply_error(task_id, job.partial_msg(refunded), refunded)
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
if t is not None:
|
||||||
|
t['result'] = {
|
||||||
|
'zip_path': zip_tmp_path,
|
||||||
|
'filename': job.zip_filename(tally.ok),
|
||||||
|
'audio_files': [],
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
_apply_error(task_id, '打包失败,请稍后重试' + (
|
||||||
|
f"(已退回 {refunded} 次下载次数)" if refunded else ''), refunded)
|
||||||
|
_discard(zip_tmp_path)
|
||||||
|
api_service.logger.error(
|
||||||
|
f"{job.tag}异常: {task_id}(成功{tally.ok} 退回{refunded}次)")
|
||||||
|
finally:
|
||||||
|
# 回收信息预取线程池,避免线程泄漏(未启动的预取任务一并取消)
|
||||||
|
job.cleanup()
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_error(task_id: str, msg: str, refunded: int) -> None:
|
||||||
|
"""异常/业务失败的终态收敛:置 error(终态先到先得)、记账退款额。
|
||||||
|
|
||||||
|
走 :func:`_mark_task_terminal` 而非就地写任务字典——后者是播客侧的旧写法,
|
||||||
|
除了重复一遍「已终态不覆盖」判断外,还漏掉了 worker 线程已死时按账本补结算
|
||||||
|
的兜底(``/F-E``)。
|
||||||
|
"""
|
||||||
|
_mark_task_terminal(task_id, 'error', msg)
|
||||||
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
|
t = DOWNLOAD_TASKS.get(task_id)
|
||||||
|
if t is not None:
|
||||||
|
t['refunded'] = refunded
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_zip(job: ZipJob, items: list[Any], ctx: ZipContext, tally: _Tally,
|
||||||
|
task: dict[str, Any], pause_ev: threading.Event) -> None:
|
||||||
|
"""逐窗口下载并写入 ZIP(返回时中央目录尚未写出,由调用方的 with 退出时落盘)。
|
||||||
|
|
||||||
|
窗口 = ``job.concurrency`` 个条目:下载并行、落包串行。窗口大小取 1 时
|
||||||
|
退化为顺序下载(播客整台即如此)。
|
||||||
|
"""
|
||||||
|
concurrency = max(1, int(job.concurrency))
|
||||||
|
pool: ThreadPoolExecutor | None = None
|
||||||
|
try:
|
||||||
|
# 不用 with 管理下载池——取消时 with 退出会 wait=True 等在途 future
|
||||||
|
# (其内部阻塞在预取 HTTP 上最长 30s),改为不等待 + 取消排队任务
|
||||||
|
pool = ThreadPoolExecutor(max_workers=concurrency)
|
||||||
|
offset = 0
|
||||||
|
while offset < len(items):
|
||||||
|
if ctx.cancel_ev.is_set():
|
||||||
|
raise DownloadCanceled("任务已取消")
|
||||||
|
# 断线暂停:释放名额等待恢复/取消,恢复后重新进入排队
|
||||||
|
_wait_if_paused(task, pause_ev, ctx.cancel_ev, tally.ok + len(tally.failed))
|
||||||
|
# 每个窗口持有 BATCH_ZIP_SEM(同一时刻全局最多 N 个批量任务在下载)
|
||||||
|
# 等待名额期间可被取消,不再无限阻塞线程与退款
|
||||||
|
if not BATCH_ZIP_SEM.acquire_cancellable(ctx.cancel_ev):
|
||||||
|
raise DownloadCanceled("任务已取消")
|
||||||
|
try:
|
||||||
|
# 暂停/取消落在「等待恢复返回」与「拿到名额」之间:立刻放名额回到
|
||||||
|
# 循环顶部重新处理暂停(原播客侧缺这一步,会多下完这一集才停)
|
||||||
|
if (ctx.cancel_ev.is_set() or pause_ev.is_set()
|
||||||
|
or task['status'] == 'paused'):
|
||||||
|
continue
|
||||||
|
window = items[offset:offset + concurrency]
|
||||||
|
start_idx = offset + 1 # 条目序号从 1 起(进度文案用)
|
||||||
|
offset += len(window)
|
||||||
|
job.prefetch(window, ctx) # 只预取本窗口
|
||||||
|
futures = {
|
||||||
|
pool.submit(job.download_one, item, start_idx + i, ctx): (item, start_idx + i)
|
||||||
|
for i, item in enumerate(window)
|
||||||
|
}
|
||||||
|
for future in as_completed(futures):
|
||||||
|
if ctx.cancel_ev.is_set():
|
||||||
|
raise DownloadCanceled("任务已取消")
|
||||||
|
item, idx = futures[future]
|
||||||
|
try:
|
||||||
|
payload = future.result()
|
||||||
|
# 落包在主线程:ZipFile 非线程安全 + 包内名去重要串行
|
||||||
|
job.write_entry(item, idx, payload, ctx)
|
||||||
|
except DownloadCanceled:
|
||||||
|
raise
|
||||||
|
except ItemFailed as f:
|
||||||
|
tally.failed.append(f.label)
|
||||||
|
except Exception as e:
|
||||||
|
tally.failed.append(f"{job.fail_label(item)}({e})")
|
||||||
|
else:
|
||||||
|
tally.ok += 1
|
||||||
|
ctx.progress(job.done_msg(item, idx, ctx))
|
||||||
|
finally:
|
||||||
|
BATCH_ZIP_SEM.release()
|
||||||
|
finally:
|
||||||
|
if pool is not None:
|
||||||
|
try:
|
||||||
|
# 取消/异常时不等待在途下载线程(其可能阻塞在预取 HTTP 上),
|
||||||
|
# 排队任务一并取消;在途线程自行收尾(.part 残片由续传/清扫兜底)
|
||||||
|
pool.shutdown(wait=False, cancel_futures=True)
|
||||||
|
except TypeError:
|
||||||
|
pool.shutdown(wait=False)
|
||||||
+151
-336
@@ -1,40 +1,174 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""歌单/专辑批量打包 worker。
|
"""歌单/专辑批量打包 worker。
|
||||||
|
|
||||||
|
打包骨架(状态机 / 暂停 / 名额 / 终态收敛 / 结算)在 :mod:`services._zipcore`,
|
||||||
|
本模块只保留「音乐歌单」这一侧的策略:窗口化预取与并行下载、每首三条目
|
||||||
|
(音频 + 歌词 + 封面)入包、同名去重、包名与文案。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
import traceback
|
|
||||||
import zipfile
|
import zipfile
|
||||||
from pathlib import Path
|
from concurrent.futures import (
|
||||||
|
Future,
|
||||||
from core.state import (
|
ThreadPoolExecutor,
|
||||||
BATCH_DURATIONS,
|
|
||||||
BATCH_DURATIONS_LOCK,
|
|
||||||
DOWNLOAD_TASKS,
|
|
||||||
DOWNLOAD_TASKS_LOCK,
|
|
||||||
)
|
)
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
from core.utils import format_lyric as _format_lyric
|
from core.utils import format_lyric as _format_lyric
|
||||||
from music_downloader import (
|
from music_downloader import (
|
||||||
DownloadCanceled,
|
|
||||||
fetch_cover_bytes,
|
fetch_cover_bytes,
|
||||||
)
|
)
|
||||||
|
|
||||||
from ._shared import (
|
from ._shared import (
|
||||||
BATCH_ZIP_SEM,
|
|
||||||
_build_download_base,
|
_build_download_base,
|
||||||
)
|
)
|
||||||
from ._taskcore import (
|
from ._taskcore import (
|
||||||
_mark_task_terminal,
|
|
||||||
_settle_batch_billing,
|
_settle_batch_billing,
|
||||||
_task_work_dir,
|
)
|
||||||
_wait_if_paused,
|
from ._zipcore import (
|
||||||
|
ItemFailed,
|
||||||
|
ZipContext,
|
||||||
|
ZipJob,
|
||||||
|
run_zip_job,
|
||||||
)
|
)
|
||||||
from .api_service import (
|
from .api_service import (
|
||||||
api_service,
|
api_service,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _SongPayload:
|
||||||
|
"""一首歌下载完成后的入包素材(下载线程产出,落包线程消费)。"""
|
||||||
|
|
||||||
|
path: str
|
||||||
|
name: str
|
||||||
|
artists: str
|
||||||
|
file_type: str
|
||||||
|
lyric: str
|
||||||
|
cover: bytes
|
||||||
|
|
||||||
|
|
||||||
|
class _SonglistZipJob(ZipJob):
|
||||||
|
"""音乐歌单打包:窗口化并行下载,每首写 音频 + 歌词 + 封面 三条目。"""
|
||||||
|
|
||||||
|
tag = '批量ZIP'
|
||||||
|
noun = '首'
|
||||||
|
item_noun = '歌曲'
|
||||||
|
tmp_prefix = 'batch_zip_'
|
||||||
|
start_msg = '下载打包中...'
|
||||||
|
record_duration = True
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._info_pool: ThreadPoolExecutor | None = None
|
||||||
|
self._info_futures: dict[Any, Future[Any]] = {}
|
||||||
|
|
||||||
|
def prepare(self, params: dict[str, Any], task_id: str, failed: list[str]) -> list[Any]:
|
||||||
|
self.concurrency = int(params['concurrency'])
|
||||||
|
return list(params['ids'])
|
||||||
|
|
||||||
|
def prefetch(self, window: list[Any], ctx: ZipContext) -> None:
|
||||||
|
"""按窗口预取歌曲信息。
|
||||||
|
|
||||||
|
原实现一次性提交全部(最多 500 首 ×3 接口)预取任务,大歌单瞬间以 10 并发
|
||||||
|
连发上游约 1500 次请求打风控;改为每个下载窗口只预取本窗口的歌(与下载节奏
|
||||||
|
对齐),``download_one`` 语义不变(等预取结果再下载)。
|
||||||
|
"""
|
||||||
|
if self._info_pool is None:
|
||||||
|
self._info_pool = ThreadPoolExecutor(max_workers=min(4, max(1, self.concurrency)))
|
||||||
|
quality = ctx.params['quality']
|
||||||
|
cookies = ctx.params['cookies']
|
||||||
|
for mid in window:
|
||||||
|
if mid not in self._info_futures:
|
||||||
|
self._info_futures[mid] = self._info_pool.submit(
|
||||||
|
api_service.downloader.get_music_info, mid, quality, cookies
|
||||||
|
)
|
||||||
|
|
||||||
|
def download_one(self, item: Any, idx: int, ctx: ZipContext) -> _SongPayload:
|
||||||
|
"""并行下载单元:等待预取信息 → 下载 → 复用预取歌词与缓存封面(零重复请求)。"""
|
||||||
|
mid = item
|
||||||
|
info = self._info_futures[mid].result()
|
||||||
|
params = ctx.params
|
||||||
|
result = api_service.downloader.download_music_file(
|
||||||
|
mid, params['quality'],
|
||||||
|
write_meta=params['write_meta'],
|
||||||
|
meta_fields=params['meta_fields'],
|
||||||
|
cookies=params['cookies'],
|
||||||
|
cancel_event=ctx.cancel_ev,
|
||||||
|
music_info=info,
|
||||||
|
)
|
||||||
|
lyric = ''
|
||||||
|
cover = b''
|
||||||
|
if result.success:
|
||||||
|
# 歌词已随 get_music_info 预取返回,无需再调一次歌词接口
|
||||||
|
lyric = getattr(info, 'lyric', '') or ''
|
||||||
|
# 封面走共享缓存(同一专辑只拉一次网络);取消后不再发起新请求
|
||||||
|
if not ctx.cancel_ev.is_set() and info.pic_url:
|
||||||
|
cover = fetch_cover_bytes(info.pic_url) or b''
|
||||||
|
if not result.success:
|
||||||
|
raise ItemFailed(str(mid))
|
||||||
|
assert result.file_path is not None
|
||||||
|
if not Path(result.file_path).exists():
|
||||||
|
# 文件可能已被同名并发任务入包删除,按失败处理而非炸掉整个任务
|
||||||
|
raise ItemFailed(str(mid))
|
||||||
|
mi = result.music_info
|
||||||
|
if mi is None:
|
||||||
|
# 与上面 exists() 同口径:成功但缺元信息(上游详情缺失)按单曲失败计,
|
||||||
|
# 不让 mi.name 抛 AttributeError 把整批转 error
|
||||||
|
raise ItemFailed(str(mid))
|
||||||
|
return _SongPayload(path=result.file_path, name=mi.name, artists=mi.artists,
|
||||||
|
file_type=mi.file_type, lyric=lyric, cover=cover)
|
||||||
|
|
||||||
|
def write_entry(self, item: Any, idx: int, payload: _SongPayload, ctx: ZipContext) -> None:
|
||||||
|
mid = item
|
||||||
|
base = _build_download_base(ctx.params['filename_format'],
|
||||||
|
{'name': payload.name, 'artist_string': payload.artists})
|
||||||
|
lyric_format = ctx.params['lyric_format']
|
||||||
|
# 同名去重:歌单里同一首歌出现两次/同名不同版本时,追加序号避免 ZIP 内重名
|
||||||
|
# 互相覆盖(用户被扣 2 次只拿到 1 首)。F-C:音频/歌词/封面三个 arcname 都要
|
||||||
|
# 查重——同名同歌手但音源扩展不同(flac vs mp3)时歌词/封面也会互相覆盖
|
||||||
|
final_name = f"{base}.{payload.file_type}"
|
||||||
|
lrc_name = f"{base}.{lyric_format}"
|
||||||
|
jpg_name = f"{base}.jpg"
|
||||||
|
k = 2
|
||||||
|
while (final_name in ctx.used_arcnames
|
||||||
|
or lrc_name in ctx.used_arcnames
|
||||||
|
or jpg_name in ctx.used_arcnames):
|
||||||
|
final_name = f"{base} ({k}).{payload.file_type}"
|
||||||
|
lrc_name = f"{base} ({k}).{lyric_format}"
|
||||||
|
jpg_name = f"{base} ({k}).jpg"
|
||||||
|
k += 1
|
||||||
|
ctx.used_arcnames.update((final_name, lrc_name, jpg_name))
|
||||||
|
# F-D:exists 与 zf.write 之间的缓存竞态(另一任务缓存 miss 删文件)
|
||||||
|
# 按单曲失败计,不再让整批转 error
|
||||||
|
try:
|
||||||
|
ctx.zf.write(payload.path, arcname=final_name, compress_type=zipfile.ZIP_STORED)
|
||||||
|
except OSError as e:
|
||||||
|
raise ItemFailed(f"{mid}(缓存文件读取失败: {e})") from e
|
||||||
|
# 缓存文件是跨任务共享资源,任务不得删除;磁盘回收交给监控线程的下载缓存
|
||||||
|
# 清扫(按文件年龄)。歌词与封面已在并行下载线程中拉取,这里直接写包(无网络等待)
|
||||||
|
if payload.lyric:
|
||||||
|
try:
|
||||||
|
ctx.zf.writestr(lrc_name, _format_lyric(payload.lyric, lyric_format),
|
||||||
|
compress_type=zipfile.ZIP_DEFLATED)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if payload.cover:
|
||||||
|
try:
|
||||||
|
ctx.zf.writestr(jpg_name, payload.cover, compress_type=zipfile.ZIP_STORED)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def zip_filename(self, ok_count: int) -> str:
|
||||||
|
return f"批量下载_{ok_count}首.zip"
|
||||||
|
|
||||||
|
def cleanup(self) -> None:
|
||||||
|
if self._info_pool is not None:
|
||||||
|
try:
|
||||||
|
self._info_pool.shutdown(wait=False, cancel_futures=True)
|
||||||
|
except TypeError:
|
||||||
|
self._info_pool.shutdown(wait=False)
|
||||||
|
|
||||||
|
|
||||||
def _batch_zip_worker(task_id: str) -> None:
|
def _batch_zip_worker(task_id: str) -> None:
|
||||||
"""后台批量打包:窗口化下载 → 写 ZIP(音频免压缩),支持取消与断线暂停
|
"""后台批量打包:窗口化下载 → 写 ZIP(音频免压缩),支持取消与断线暂停
|
||||||
|
|
||||||
@@ -42,323 +176,4 @@ def _batch_zip_worker(task_id: str) -> None:
|
|||||||
保存进度进入暂停等待;重新打开网站(心跳恢复)后自动回到队列继续下载
|
保存进度进入暂停等待;重新打开网站(心跳恢复)后自动回到队列继续下载
|
||||||
- 音频入包即删,缓解磁盘占用
|
- 音频入包即删,缓解磁盘占用
|
||||||
"""
|
"""
|
||||||
import tempfile as _tempfile
|
run_zip_job(_SonglistZipJob(), task_id, settle=_settle_batch_billing)
|
||||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
|
||||||
|
|
||||||
zip_tmp_path: str | None = None
|
|
||||||
# zip_valid 必须在函数头初始化:ZIP 构建前的业务错误(参数缺失/无歌曲等)
|
|
||||||
# 会 early raise,异常处理器若引用未绑定名会 NameError 二次崩溃 worker 线程
|
|
||||||
zip_valid = False
|
|
||||||
ok_count = 0
|
|
||||||
failed: list[str] = []
|
|
||||||
used_arcnames = set() # ZIP 内已占用文件名(同名歌曲去重,防止包内互相覆盖)
|
|
||||||
info_pool = None
|
|
||||||
info_futures = {}
|
|
||||||
try:
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
task = DOWNLOAD_TASKS[task_id]
|
|
||||||
task['status'] = 'running'
|
|
||||||
task['msg'] = '下载打包中...'
|
|
||||||
params = task['params']
|
|
||||||
cancel_ev = task['cancel']
|
|
||||||
pause_ev = task['pause']
|
|
||||||
ids = list(params['ids'])
|
|
||||||
quality = params['quality']
|
|
||||||
filename_format = params['filename_format']
|
|
||||||
write_meta = params['write_meta']
|
|
||||||
meta_fields = params['meta_fields']
|
|
||||||
lyric_format = params['lyric_format']
|
|
||||||
concurrency = params['concurrency']
|
|
||||||
cookies = params['cookies']
|
|
||||||
batch_start = time.time()
|
|
||||||
|
|
||||||
# 预取按窗口进行——原实现一次性提交全部(最多 500 首 ×3 接口)预取任务,
|
|
||||||
# 大歌单瞬间以 10 并发连发上游约 1500 次请求打风控;改为每个下载窗口只预取本窗口
|
|
||||||
# 的歌(与下载节奏对齐),_download_one 语义不变(等预取结果再下载)
|
|
||||||
from concurrent.futures import ThreadPoolExecutor as _InfoPool
|
|
||||||
info_pool = _InfoPool(max_workers=min(4, max(1, concurrency)))
|
|
||||||
|
|
||||||
def _prefetch(window_ids):
|
|
||||||
for mid in window_ids:
|
|
||||||
if mid not in info_futures:
|
|
||||||
info_futures[mid] = info_pool.submit(
|
|
||||||
api_service.downloader.get_music_info, mid, quality, cookies
|
|
||||||
)
|
|
||||||
|
|
||||||
def _download_one(mid):
|
|
||||||
"""并行下载单元:等待预取信息 → 下载 → 复用预取歌词与缓存封面(零重复请求)"""
|
|
||||||
info = info_futures[mid].result()
|
|
||||||
result = api_service.downloader.download_music_file(
|
|
||||||
mid, quality,
|
|
||||||
write_meta=write_meta,
|
|
||||||
meta_fields=meta_fields,
|
|
||||||
cookies=cookies,
|
|
||||||
cancel_event=cancel_ev,
|
|
||||||
music_info=info,
|
|
||||||
)
|
|
||||||
extra = {'lyric': '', 'cover': b''}
|
|
||||||
if result.success:
|
|
||||||
# 歌词已随 get_music_info 预取返回,无需再调一次歌词接口
|
|
||||||
extra['lyric'] = getattr(info, 'lyric', '') or ''
|
|
||||||
# 封面走共享缓存(同一专辑只拉一次网络);取消后不再发起新请求
|
|
||||||
if not cancel_ev.is_set() and info.pic_url:
|
|
||||||
cover = fetch_cover_bytes(info.pic_url)
|
|
||||||
if cover:
|
|
||||||
extra['cover'] = cover
|
|
||||||
return result, extra
|
|
||||||
|
|
||||||
def _update_progress() -> None:
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
DOWNLOAD_TASKS[task_id]['done'] = ok_count + len(failed)
|
|
||||||
DOWNLOAD_TASKS[task_id]['failed'] = len(failed)
|
|
||||||
|
|
||||||
# ZIP 构建放 downloads 卷(原实现落系统 %TEMP%——500 首无损可达 GB 级,
|
|
||||||
# 小系统盘一次大单即塞爆;单曲/播客早已指定 dir,唯批量漏了)
|
|
||||||
# 且建在任务专属目录(见 _task_work_dir):根目录会被缓存清扫按 mtime 回收
|
|
||||||
_task_dir_for_zip = _task_work_dir(task_id)
|
|
||||||
# 中央目录完整写出后(下方)才置 True,否则一律按失败全额退费
|
|
||||||
with _tempfile.NamedTemporaryFile(prefix='batch_zip_', suffix='.zip',
|
|
||||||
dir=str(_task_dir_for_zip),
|
|
||||||
delete=False) as zip_tmp:
|
|
||||||
zip_tmp_path = zip_tmp.name
|
|
||||||
# 音频本身已压缩,用 ZIP_STORED(免压缩)大幅提速;歌词等文本仍压缩
|
|
||||||
with zipfile.ZipFile(zip_tmp, 'w', zipfile.ZIP_DEFLATED) as zf:
|
|
||||||
# 不用 with 管理下载池——取消时 with 退出会 wait=True 等在途
|
|
||||||
# future(其内部阻塞在预取 HTTP 上最长 30s),改为不等待 + 取消排队任务
|
|
||||||
pool = ThreadPoolExecutor(max_workers=concurrency)
|
|
||||||
try:
|
|
||||||
while ids:
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
# 断线暂停:释放名额等待恢复/取消,恢复后重新进入排队
|
|
||||||
_wait_if_paused(task, pause_ev, cancel_ev, ok_count + len(failed))
|
|
||||||
# 每个窗口持有 BATCH_ZIP_SEM(同一时刻全局最多 3 个批量任务在下载)
|
|
||||||
# 等待名额期间可被取消,不再无限阻塞线程与退款
|
|
||||||
if not BATCH_ZIP_SEM.acquire_cancellable(cancel_ev):
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
try:
|
|
||||||
if cancel_ev.is_set() or pause_ev.is_set() or task['status'] == 'paused':
|
|
||||||
continue # 立即释放名额,回到循环顶部处理暂停
|
|
||||||
window = ids[:concurrency]
|
|
||||||
ids = ids[concurrency:]
|
|
||||||
_prefetch(window) # 只预取本窗口
|
|
||||||
future_map = {
|
|
||||||
pool.submit(_download_one, mid): mid
|
|
||||||
for mid in window
|
|
||||||
}
|
|
||||||
for future in as_completed(future_map):
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
mid = future_map[future]
|
|
||||||
try:
|
|
||||||
result, extra = future.result()
|
|
||||||
except DownloadCanceled:
|
|
||||||
raise
|
|
||||||
except Exception as e:
|
|
||||||
failed.append(f"{mid}({e})")
|
|
||||||
_update_progress()
|
|
||||||
continue
|
|
||||||
if not result.success:
|
|
||||||
failed.append(str(mid))
|
|
||||||
_update_progress()
|
|
||||||
continue
|
|
||||||
assert result.file_path is not None
|
|
||||||
fp = Path(result.file_path)
|
|
||||||
if not fp.exists():
|
|
||||||
# 文件可能已被同名并发任务入包删除,按失败处理而非炸掉整个任务
|
|
||||||
failed.append(str(mid))
|
|
||||||
_update_progress()
|
|
||||||
continue
|
|
||||||
mi = result.music_info
|
|
||||||
if mi is None:
|
|
||||||
# 与上面 fp.exists() 同口径:成功但缺元信息(上游详情缺失)
|
|
||||||
# 按单曲失败计,不让 mi.name 抛 AttributeError 把整批转 error
|
|
||||||
failed.append(str(mid))
|
|
||||||
_update_progress()
|
|
||||||
continue
|
|
||||||
base = _build_download_base(filename_format, {'name': mi.name, 'artist_string': mi.artists})
|
|
||||||
# 同名去重:歌单里同一首歌出现两次/同名不同版本时,
|
|
||||||
# 追加序号避免 ZIP 内重名互相覆盖(用户被扣 2 次只拿到 1 首)
|
|
||||||
# F-C:音频/歌词/封面三个 arcname 都要查重——同名同歌手但
|
|
||||||
# 音源扩展不同(flac vs mp3)时歌词/封面条目也会互相覆盖
|
|
||||||
final_name = f"{base}.{mi.file_type}"
|
|
||||||
lrc_name = f"{base}.{lyric_format}"
|
|
||||||
jpg_name = f"{base}.jpg"
|
|
||||||
k = 2
|
|
||||||
while (final_name in used_arcnames
|
|
||||||
or lrc_name in used_arcnames
|
|
||||||
or jpg_name in used_arcnames):
|
|
||||||
final_name = f"{base} ({k}).{mi.file_type}"
|
|
||||||
lrc_name = f"{base} ({k}).{lyric_format}"
|
|
||||||
jpg_name = f"{base} ({k}).jpg"
|
|
||||||
k += 1
|
|
||||||
used_arcnames.update((final_name, lrc_name, jpg_name))
|
|
||||||
# F-D:exists 与 zf.write 之间的缓存竞态(另一任务缓存 miss
|
|
||||||
# 删文件)按单曲失败计,不再让整批转 error
|
|
||||||
try:
|
|
||||||
zf.write(str(fp), arcname=final_name, compress_type=zipfile.ZIP_STORED)
|
|
||||||
except OSError as e:
|
|
||||||
failed.append(f"{mid}(缓存文件读取失败: {e})")
|
|
||||||
_update_progress()
|
|
||||||
continue
|
|
||||||
# 缓存文件是跨任务共享资源,任务不得删除;
|
|
||||||
# 磁盘回收交给监控线程的下载缓存清扫(按文件年龄)
|
|
||||||
# 歌词与封面已在并行下载线程中拉取,直接写入 ZIP(无网络等待)
|
|
||||||
lrc = extra.get('lyric', '')
|
|
||||||
if lrc:
|
|
||||||
try:
|
|
||||||
zf.writestr(lrc_name, _format_lyric(lrc, lyric_format), compress_type=zipfile.ZIP_DEFLATED)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
if extra.get('cover'):
|
|
||||||
try:
|
|
||||||
zf.writestr(jpg_name, extra['cover'], compress_type=zipfile.ZIP_STORED)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
ok_count += 1
|
|
||||||
_update_progress()
|
|
||||||
finally:
|
|
||||||
BATCH_ZIP_SEM.release()
|
|
||||||
finally:
|
|
||||||
# 取消/异常时不等待在途下载线程(其可能阻塞在预取 HTTP 上),
|
|
||||||
# 排队任务一并取消;在途线程自行收尾(.part 残片由续传/清扫兜底)
|
|
||||||
try:
|
|
||||||
pool.shutdown(wait=False, cancel_futures=True)
|
|
||||||
except TypeError:
|
|
||||||
pool.shutdown(wait=False)
|
|
||||||
# 复审 修正:zip_valid = True 必须在内层 with zipfile 退出之后
|
|
||||||
# (中央目录在 close 时才写出)——原缩进 16 在 with 内部,守卫不可达
|
|
||||||
zip_valid = True
|
|
||||||
|
|
||||||
if ok_count > 0:
|
|
||||||
with BATCH_DURATIONS_LOCK:
|
|
||||||
BATCH_DURATIONS.append(time.time() - batch_start)
|
|
||||||
|
|
||||||
# F-B:ZIP 未正常完成(磁盘满写坏中央目录等)——不发布坏包,按全额退费收敛
|
|
||||||
if not zip_valid:
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
zip_tmp_path = None
|
|
||||||
refunded = _settle_batch_billing(task_id, 0, 'error')
|
|
||||||
_mark_task_terminal(task_id, 'error',
|
|
||||||
f"ZIP 打包失败(可能磁盘空间不足),已退回 {refunded} 次" if refunded
|
|
||||||
else "ZIP 打包失败(可能磁盘空间不足),费用未扣或已退回")
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['failed'] = t.get('failed', 0) + ok_count
|
|
||||||
t['refunded'] = refunded
|
|
||||||
api_service.logger.error(f"批量ZIP构建失败未发布: {task_id} ok_count={ok_count} 退回{refunded}次")
|
|
||||||
return
|
|
||||||
|
|
||||||
# 全部失败不发布空 ZIP、不标 done —— 按失败收敛并全额退费
|
|
||||||
if ok_count == 0 and failed:
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
zip_tmp_path = None
|
|
||||||
refunded = _settle_batch_billing(task_id, 0, 'error')
|
|
||||||
_mark_task_terminal(task_id, 'error',
|
|
||||||
f"全部失败({len(failed)} 首)" + (f",已退回 {refunded} 次" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
DOWNLOAD_TASKS[task_id]['failed'] = len(failed)
|
|
||||||
DOWNLOAD_TASKS[task_id]['refunded'] = refunded
|
|
||||||
api_service.logger.info(f"批量ZIP全部失败: {task_id} 失败{len(failed)} 退回{refunded}次")
|
|
||||||
return
|
|
||||||
|
|
||||||
refunded = _settle_batch_billing(task_id, ok_count)
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
# 发布前锁内复查取消/失效状态——取消请求落在"检查后、拿锁前"的窗口也不得覆盖终态
|
|
||||||
if t is None or t.get('status') in ('canceled', 'error'):
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
t['status'] = 'done'
|
|
||||||
t['completed_at'] = time.time()
|
|
||||||
t['done'] = ok_count + len(failed)
|
|
||||||
t['failed'] = len(failed)
|
|
||||||
t['msg'] = f"成功{ok_count} 失败{len(failed)}" + (f" 退回{refunded}次" if refunded else '')
|
|
||||||
t['refunded'] = refunded
|
|
||||||
t['result'] = {
|
|
||||||
'zip_path': zip_tmp_path,
|
|
||||||
'filename': f"批量下载_{ok_count}首.zip",
|
|
||||||
'audio_files': [], # 缓存文件不归任务所有,不进入清理清单
|
|
||||||
}
|
|
||||||
api_service.logger.info(f"批量ZIP完成: 成功{ok_count} 失败{len(failed)} 退回{refunded}次")
|
|
||||||
except DownloadCanceled:
|
|
||||||
# 部分成功后取消——已入包部分照常交付(成果守恒),未完成部分按账本退费
|
|
||||||
# F-B:ZIP 未正常完成(写坏)时不按 ok_count 交付——按全额退费收敛
|
|
||||||
delivered = ok_count if zip_valid else 0
|
|
||||||
refunded = _settle_batch_billing(task_id, delivered, 'canceled')
|
|
||||||
if delivered > 0 and zip_tmp_path:
|
|
||||||
_mark_task_terminal(task_id, 'canceled',
|
|
||||||
f"已取消(已打包 {ok_count} 首可领取)" +
|
|
||||||
(f",退回 {refunded} 次" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
t['result'] = {
|
|
||||||
'zip_path': zip_tmp_path,
|
|
||||||
'filename': f"批量下载_{ok_count}首.zip",
|
|
||||||
'audio_files': [],
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
_mark_task_terminal(task_id, 'canceled',
|
|
||||||
'任务已取消' + (f"(已退回 {refunded} 次下载次数)" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
api_service.logger.info(f"批量ZIP已取消: {task_id}(成功{ok_count} 退回{refunded}次)")
|
|
||||||
except Exception as e:
|
|
||||||
# 通用异常与取消同一套交付/结算规则:有成果则交付,按账本退未交付部分
|
|
||||||
# F-B:ZIP 未正常完成(写坏)时按全额退费,不发布坏包
|
|
||||||
# 非预期异常原文 logger 先行,用户可见文案泛化
|
|
||||||
api_service.logger.error(f"批量打包异常: {task_id} {e}\n{traceback.format_exc()}")
|
|
||||||
delivered = ok_count if zip_valid else 0
|
|
||||||
refunded = _settle_batch_billing(task_id, delivered, 'error')
|
|
||||||
if delivered > 0 and zip_tmp_path:
|
|
||||||
_mark_task_terminal(task_id, 'error',
|
|
||||||
"部分歌曲已打包完成,可领取" +
|
|
||||||
(f",退回 {refunded} 次" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
t['result'] = {
|
|
||||||
'zip_path': zip_tmp_path,
|
|
||||||
'filename': f"批量下载_{ok_count}首.zip",
|
|
||||||
'audio_files': [],
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
_mark_task_terminal(task_id, 'error',
|
|
||||||
'打包失败,请稍后重试' + (f"(已退回 {refunded} 次下载次数)" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
api_service.logger.error(f"批量ZIP任务异常: {task_id} {e}(成功{ok_count} 退回{refunded}次)\n{traceback.format_exc()}")
|
|
||||||
finally:
|
|
||||||
# 回收信息预取线程池,避免线程泄漏(未启动的预取任务一并取消)
|
|
||||||
if info_pool is not None:
|
|
||||||
try:
|
|
||||||
info_pool.shutdown(wait=False, cancel_futures=True)
|
|
||||||
except TypeError:
|
|
||||||
info_pool.shutdown(wait=False)
|
|
||||||
+123
-236
@@ -1,13 +1,17 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""播客单集与整台打包 worker。
|
"""播客单集与整台打包 worker。
|
||||||
"""
|
|
||||||
|
|
||||||
|
整台打包的骨架(状态机 / 暂停 / 名额 / 终态收敛 / 结算)在
|
||||||
|
:mod:`services._zipcore`,本模块只保留播客侧策略:清单组装(整台或指定单集)、
|
||||||
|
逐集顺序下载、按实际集数补扣、单条目入包(首集附封面)。
|
||||||
|
"""
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
import traceback
|
import traceback
|
||||||
import zipfile
|
import zipfile
|
||||||
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -27,14 +31,16 @@ from music_downloader import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
from ._shared import (
|
from ._shared import (
|
||||||
BATCH_ZIP_SEM,
|
|
||||||
_record_log,
|
_record_log,
|
||||||
)
|
)
|
||||||
from ._taskcore import (
|
from ._taskcore import (
|
||||||
_mark_task_terminal,
|
_mark_task_terminal,
|
||||||
_settle_batch_billing,
|
_settle_batch_billing,
|
||||||
_task_work_dir,
|
)
|
||||||
_wait_if_paused,
|
from ._zipcore import (
|
||||||
|
ZipContext,
|
||||||
|
ZipJob,
|
||||||
|
run_zip_job,
|
||||||
)
|
)
|
||||||
from .api_service import (
|
from .api_service import (
|
||||||
api_service,
|
api_service,
|
||||||
@@ -287,32 +293,37 @@ def _podcast_cleanup_files(paths: list[str | None]) -> None:
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
def _podcast_zip_worker(task_id: str) -> None:
|
@dataclass(frozen=True)
|
||||||
"""整台播客打包:逐集 下载→入包即删,支持取消与断线暂停(复用批量任务机制)"""
|
class _EpisodePayload:
|
||||||
import tempfile as _tempfile
|
"""一集下载完成后的入包素材(下载线程产出,落包线程消费)。"""
|
||||||
|
|
||||||
zip_tmp_path: str | None = None
|
path: str
|
||||||
ok_count = 0
|
base: str
|
||||||
failed: list[str] = []
|
ext: str
|
||||||
# zip_valid / radio_name 必须在函数头初始化——空电台/无单集/补扣失败等
|
|
||||||
# 正常业务错误在 try 内早期 raise,异常处理器引用未绑定名会二次崩溃 worker 线程
|
|
||||||
zip_valid = False
|
class _PodcastZipJob(ZipJob):
|
||||||
radio_name = ''
|
"""整台播客打包:逐集顺序下载(窗口=1)→ 单条目入包,首集附封面。
|
||||||
try:
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
并发刻意等于音乐批量:同样占用全局批量名额(BATCH_ZIP_SEM,全局 3),
|
||||||
task = DOWNLOAD_TASKS[task_id]
|
逐集 acquire/release——暂停等待/取消时不长期占名额;音频本身仍受全局
|
||||||
task['status'] = 'running'
|
下载信号量约束。原实现不受批量并发控制,多用户同时打包会无限并发。
|
||||||
task['msg'] = '解析单集列表...'
|
"""
|
||||||
params = task['params']
|
|
||||||
cancel_ev = task['cancel']
|
tag = '播客打包'
|
||||||
pause_ev = task['pause']
|
noun = '集'
|
||||||
radio_id = params['radio_id']
|
item_noun = '单集'
|
||||||
preset_ids = params.get('ids')
|
tmp_prefix = 'podcast_batch_'
|
||||||
limit = params['limit']
|
start_msg = '解析单集列表...'
|
||||||
quality = params['quality']
|
concurrency = 1
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._radio_name = ''
|
||||||
|
|
||||||
|
def prepare(self, params: dict[str, Any], task_id: str, failed: list[str]) -> list[Any]:
|
||||||
cookies = params['cookies']
|
cookies = params['cookies']
|
||||||
remote_ip = params.get('remote_ip') or '-'
|
radio_id = params['radio_id']
|
||||||
|
limit = params['limit']
|
||||||
# 组装单集清单:整台电台(正序全量)或指定单集ID列表
|
# 组装单集清单:整台电台(正序全量)或指定单集ID列表
|
||||||
if radio_id:
|
if radio_id:
|
||||||
episodes = podcast.list_radio_all_program_ids(radio_id, cookies, max_count=limit)
|
episodes = podcast.list_radio_all_program_ids(radio_id, cookies, max_count=limit)
|
||||||
@@ -320,7 +331,7 @@ def _podcast_zip_worker(task_id: str) -> None:
|
|||||||
raise DownloadException("该电台暂无可下载的单集")
|
raise DownloadException("该电台暂无可下载的单集")
|
||||||
else:
|
else:
|
||||||
episodes = []
|
episodes = []
|
||||||
for pid in (preset_ids or []):
|
for pid in (params.get('ids') or []):
|
||||||
try:
|
try:
|
||||||
prog = podcast.get_program_detail(pid, cookies)
|
prog = podcast.get_program_detail(pid, cookies)
|
||||||
episodes.append({
|
episodes.append({
|
||||||
@@ -337,15 +348,17 @@ def _podcast_zip_worker(task_id: str) -> None:
|
|||||||
if not episodes:
|
if not episodes:
|
||||||
raise DownloadException("没有可下载的单集")
|
raise DownloadException("没有可下载的单集")
|
||||||
|
|
||||||
radio_name = _podcast_safe_name(params.get('radio_name') or '', '播客')
|
self._radio_name = _podcast_safe_name(params.get('radio_name') or '', '播客')
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
return episodes
|
||||||
DOWNLOAD_TASKS[task_id]['total'] = len(episodes)
|
|
||||||
|
|
||||||
# 按次计费补扣:整台电台提交时只预扣 1 次,此处按实际单集数补足差额。
|
def supplement(self, task_id: str, params: dict[str, Any], count: int) -> None:
|
||||||
# 补扣失败(余额不足/数据库异常)任务必须失败并按账本退费——不允许无账本继续交付
|
"""按次计费补扣:整台电台提交时只预扣 1 次,此处按实际单集数补足差额。
|
||||||
|
|
||||||
|
补扣失败(余额不足/数据库异常)任务必须失败并按账本退费——不允许无账本继续交付。
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
_buid = params.get('billing_uid')
|
_buid = params.get('billing_uid')
|
||||||
_extra = len(episodes) - int(params.get('billed') or 1)
|
_extra = count - int(params.get('billed') or 1)
|
||||||
if _buid and _extra > 0:
|
if _buid and _extra > 0:
|
||||||
if not auth_system.ledger_supplement(task_id, _extra):
|
if not auth_system.ledger_supplement(task_id, _extra):
|
||||||
raise DownloadException(
|
raise DownloadException(
|
||||||
@@ -353,7 +366,7 @@ def _podcast_zip_worker(task_id: str) -> None:
|
|||||||
if _buid:
|
if _buid:
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
with DOWNLOAD_TASKS_LOCK:
|
||||||
DOWNLOAD_TASKS[task_id]['billing_user_id'] = int(_buid)
|
DOWNLOAD_TASKS[task_id]['billing_user_id'] = int(_buid)
|
||||||
DOWNLOAD_TASKS[task_id]['billing_cost'] = len(episodes)
|
DOWNLOAD_TASKS[task_id]['billing_cost'] = count
|
||||||
except DownloadCanceled:
|
except DownloadCanceled:
|
||||||
raise
|
raise
|
||||||
except DownloadException:
|
except DownloadException:
|
||||||
@@ -361,206 +374,80 @@ def _podcast_zip_worker(task_id: str) -> None:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise DownloadException(f"补扣下载次数异常: {e}") from e
|
raise DownloadException(f"补扣下载次数异常: {e}") from e
|
||||||
|
|
||||||
def _update_progress(msg: str = '') -> None:
|
def download_one(self, item: Any, idx: int, ctx: ZipContext) -> _EpisodePayload:
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
ep = item
|
||||||
t = DOWNLOAD_TASKS[task_id]
|
label = f"[{idx}/{ctx.total}] {ep['name'][:24]}"
|
||||||
t['done'] = ok_count + len(failed)
|
ctx.progress(f"{label} 解析中...")
|
||||||
t['failed'] = len(failed)
|
cookies = ctx.params['cookies']
|
||||||
if msg:
|
prog = ep.get('detail') or podcast.get_program_detail(ep['id'], cookies)
|
||||||
t['msg'] = msg
|
url_info = _podcast_resolve_audio(prog, ctx.params['quality'], cookies)
|
||||||
|
ctx.progress(f"{label} 下载中...")
|
||||||
# 对齐音乐批量 F-B 口径——zip_valid 仅在 ZIP with 块正常
|
ext = (url_info.get('type') or 'mp3').lower().lstrip('.')
|
||||||
# 退出(中央目录完整写出)后才置 True(初始 False 已上移函数头);
|
serial = int(ep.get('serial_num') or 0)
|
||||||
# 未 valid 一律不发布、不按 ok_count 结算
|
prefix = f"{serial:03d} - " if serial else ''
|
||||||
# 同音乐批量——ZIP 构建放 downloads 卷,不落系统 %TEMP%
|
base = _podcast_safe_name(f"{prefix}{ep['name']}")
|
||||||
# 与音乐批量打包同口径:建在任务专属目录 tasks/<task_id>/,根目录缓存清扫不再触及交付物
|
# 中间文件移入任务目录——落 downloads 根目录会被缓存清扫按 mtime 打断慢速
|
||||||
_podcast_task_dir = _task_work_dir(task_id)
|
# 单集(Linux 上 unlink 打开中的文件)
|
||||||
with _tempfile.NamedTemporaryFile(prefix='podcast_batch_', suffix='.zip',
|
tmp_audio = ctx.work_dir / f"podcast_zip_tmp_{ctx.task_id}_{idx}.{ext}"
|
||||||
dir=str(_podcast_task_dir),
|
|
||||||
delete=False) as zip_tmp:
|
|
||||||
zip_tmp_path = zip_tmp.name
|
|
||||||
used_arcnames = set() # 包内条目名去重(同名单集/同号不同集不得互相覆盖)
|
|
||||||
with zipfile.ZipFile(zip_tmp, 'w', zipfile.ZIP_DEFLATED) as zf:
|
|
||||||
for idx, ep in enumerate(episodes, 1):
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
_wait_if_paused(task, pause_ev, cancel_ev, ok_count + len(failed))
|
|
||||||
# 并发对齐音乐批量:播客打包同样占用全局批量名额(BATCH_ZIP_SEM,全局 3),
|
|
||||||
# 逐集 acquire/release——暂停等待/取消时不长期占名额;音频本身仍受
|
|
||||||
# 全局下载信号量约束。原实现不受批量并发控制,多用户同时打包会无限并发
|
|
||||||
if not BATCH_ZIP_SEM.acquire_cancellable(cancel_ev):
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
try:
|
|
||||||
label = f"[{idx}/{len(episodes)}] {ep['name'][:24]}"
|
|
||||||
_update_progress(f"{label} 解析中...")
|
|
||||||
try:
|
|
||||||
prog = ep.get('detail') or podcast.get_program_detail(ep['id'], cookies)
|
|
||||||
url_info = _podcast_resolve_audio(prog, quality, cookies)
|
|
||||||
_update_progress(f"{label} 下载中...")
|
|
||||||
ext = (url_info.get('type') or 'mp3').lower().lstrip('.')
|
|
||||||
serial = int(ep.get('serial_num') or 0)
|
|
||||||
prefix = f"{serial:03d} - " if serial else ''
|
|
||||||
base = _podcast_safe_name(f"{prefix}{ep['name']}")
|
|
||||||
# 中间文件移入任务目录——落 downloads 根目录会被
|
|
||||||
# 缓存清扫按 mtime 打断慢速单集(Linux 上 unlink 打开中的文件)
|
|
||||||
tmp_audio = _podcast_task_dir / f"podcast_zip_tmp_{task_id}_{idx}.{ext}"
|
|
||||||
try:
|
|
||||||
_podcast_stream_download(url_info['url'], tmp_audio,
|
|
||||||
url_info.get('size', 0), cancel_ev)
|
|
||||||
# 重名条目自动追加序号
|
|
||||||
arcname = f"{base}.{ext}"
|
|
||||||
k = 2
|
|
||||||
while arcname in used_arcnames:
|
|
||||||
arcname = f"{base} ({k}).{ext}"
|
|
||||||
k += 1
|
|
||||||
used_arcnames.add(arcname)
|
|
||||||
zf.write(str(tmp_audio), arcname=arcname, compress_type=zipfile.ZIP_STORED)
|
|
||||||
finally:
|
|
||||||
# finally 连带清理 .part 残片(原实现只删正文,
|
|
||||||
# 取消/失败时残片要等 1 小时缓存清扫兜底)
|
|
||||||
_podcast_cleanup_files([str(tmp_audio)])
|
|
||||||
# 首集附带封面,便于识别
|
|
||||||
if idx == 1 and ep.get('cover_url'):
|
|
||||||
try:
|
|
||||||
cover = fetch_cover_bytes(ep['cover_url'])
|
|
||||||
if cover:
|
|
||||||
zf.writestr("cover.jpg", cover, compress_type=zipfile.ZIP_STORED)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
ok_count += 1
|
|
||||||
_update_progress(f"{label} 完成")
|
|
||||||
except DownloadCanceled:
|
|
||||||
raise
|
|
||||||
except Exception as e:
|
|
||||||
failed.append(f"{ep['name']}({e})")
|
|
||||||
_update_progress()
|
|
||||||
finally:
|
|
||||||
BATCH_ZIP_SEM.release()
|
|
||||||
|
|
||||||
# 走到这里说明 ZIP with 块正常退出、中央目录完整,才允许发布
|
|
||||||
zip_valid = True
|
|
||||||
|
|
||||||
if ok_count == 0:
|
|
||||||
raise DownloadException(f"全部单集下载失败: {'; '.join(failed[:3])}")
|
|
||||||
|
|
||||||
# ZIP 未正常完成(磁盘满写坏中央目录等)——不发布坏包,按全额退费收敛
|
|
||||||
if not zip_valid:
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
zip_tmp_path = None
|
|
||||||
refunded = _settle_batch_billing(task_id, 0, 'error')
|
|
||||||
_mark_task_terminal(task_id, 'error',
|
|
||||||
f"ZIP 打包失败(可能磁盘空间不足),已退回 {refunded} 次" if refunded
|
|
||||||
else "ZIP 打包失败(可能磁盘空间不足),费用未扣或已退回")
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['failed'] = t.get('failed', 0) + ok_count
|
|
||||||
t['refunded'] = refunded
|
|
||||||
api_service.logger.error(f"播客打包构建失败未发布: {task_id} ok_count={ok_count} 退回{refunded}次")
|
|
||||||
return
|
|
||||||
|
|
||||||
# 最终发布前的取消检查(末段取消不得覆盖为 done)
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
refunded = _settle_batch_billing(task_id, ok_count)
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
# 发布前锁内复查取消/失效状态(同音乐批量)
|
|
||||||
if t is None or t.get('status') in ('canceled', 'error'):
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
if cancel_ev.is_set():
|
|
||||||
raise DownloadCanceled("任务已取消")
|
|
||||||
t['status'] = 'done'
|
|
||||||
t['completed_at'] = time.time()
|
|
||||||
t['done'] = ok_count + len(failed)
|
|
||||||
t['failed'] = len(failed)
|
|
||||||
t['msg'] = f"成功{ok_count} 失败{len(failed)}" + (f" 退回{refunded}次" if refunded else '')
|
|
||||||
t['refunded'] = refunded
|
|
||||||
t['result'] = {
|
|
||||||
'zip_path': zip_tmp_path,
|
|
||||||
'filename': f"播客_{radio_name}_{ok_count}集.zip",
|
|
||||||
'audio_files': [],
|
|
||||||
}
|
|
||||||
try:
|
try:
|
||||||
first_name = episodes[0].get('name') if episodes else ''
|
_podcast_stream_download(url_info['url'], tmp_audio,
|
||||||
_record_log('下载', remote_ip, f"[播客打包] {radio_name} · {first_name}")
|
url_info.get('size', 0), ctx.cancel_ev)
|
||||||
|
except Exception:
|
||||||
|
# 下载失败即清(.part 残片同清);成功时由 write_entry 的 finally 清
|
||||||
|
_podcast_cleanup_files([str(tmp_audio)])
|
||||||
|
raise
|
||||||
|
return _EpisodePayload(path=str(tmp_audio), base=base, ext=ext)
|
||||||
|
|
||||||
|
def write_entry(self, item: Any, idx: int, payload: _EpisodePayload, ctx: ZipContext) -> None:
|
||||||
|
ep = item
|
||||||
|
try:
|
||||||
|
# 重名条目自动追加序号(同名单集/同号不同集不得互相覆盖)
|
||||||
|
arcname = f"{payload.base}.{payload.ext}"
|
||||||
|
k = 2
|
||||||
|
while arcname in ctx.used_arcnames:
|
||||||
|
arcname = f"{payload.base} ({k}).{payload.ext}"
|
||||||
|
k += 1
|
||||||
|
ctx.used_arcnames.add(arcname)
|
||||||
|
ctx.zf.write(payload.path, arcname=arcname, compress_type=zipfile.ZIP_STORED)
|
||||||
|
# 首集附带封面,便于识别
|
||||||
|
if idx == 1 and ep.get('cover_url'):
|
||||||
|
try:
|
||||||
|
cover = fetch_cover_bytes(ep['cover_url'])
|
||||||
|
if cover:
|
||||||
|
ctx.zf.writestr("cover.jpg", cover, compress_type=zipfile.ZIP_STORED)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
# finally 连带清理 .part 残片(原实现只删正文,取消/失败时残片要等
|
||||||
|
# 1 小时缓存清扫兜底)
|
||||||
|
_podcast_cleanup_files([payload.path])
|
||||||
|
|
||||||
|
def fail_label(self, item: Any) -> str:
|
||||||
|
"""失败文案用单集名而非整个条目字典(引擎拼成「单集名(异常)」)。"""
|
||||||
|
return str(item.get('name') or '')
|
||||||
|
|
||||||
|
def done_msg(self, item: Any, idx: int, ctx: ZipContext) -> str:
|
||||||
|
return f"[{idx}/{ctx.total}] {item['name'][:24]} 完成"
|
||||||
|
|
||||||
|
def zip_filename(self, ok_count: int) -> str:
|
||||||
|
return f"播客_{self._radio_name or '打包'}_{ok_count}集.zip"
|
||||||
|
|
||||||
|
def log_label(self) -> str:
|
||||||
|
return f"{self._radio_name} "
|
||||||
|
|
||||||
|
def all_failed_msg(self, failed: list[str], refunded: int) -> str:
|
||||||
|
return f"全部单集下载失败: {'; '.join(failed[:3])}" + (
|
||||||
|
f"(已退回 {refunded} 次下载次数)" if refunded else '')
|
||||||
|
|
||||||
|
def on_delivered(self, items: list[Any], task_id: str, params: dict[str, Any]) -> None:
|
||||||
|
try:
|
||||||
|
first_name = items[0].get('name') if items else ''
|
||||||
|
_record_log('下载', params.get('remote_ip') or '-',
|
||||||
|
f"[播客打包] {self._radio_name} · {first_name}")
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
api_service.logger.info(f"播客打包完成: {radio_name} 成功{ok_count} 失败{len(failed)} ({task_id})")
|
|
||||||
except DownloadCanceled:
|
|
||||||
# 部分成功后取消——已入包部分照常交付,未完成部分按账本退费
|
def _podcast_zip_worker(task_id: str) -> None:
|
||||||
# ZIP 未正常完成(写坏)时不按 ok_count 交付——按全额退费收敛
|
"""整台播客打包:逐集 下载→入包即删,支持取消与断线暂停(复用批量任务机制)"""
|
||||||
delivered = ok_count if zip_valid else 0
|
run_zip_job(_PodcastZipJob(), task_id, settle=_settle_batch_billing)
|
||||||
refunded = _settle_batch_billing(task_id, delivered, 'canceled')
|
|
||||||
if delivered > 0 and zip_tmp_path:
|
|
||||||
_mark_task_terminal(task_id, 'canceled',
|
|
||||||
f"已取消(已打包 {ok_count} 集可领取)" +
|
|
||||||
(f",退回 {refunded} 次" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
t['result'] = {
|
|
||||||
'zip_path': zip_tmp_path,
|
|
||||||
'filename': f"播客_{radio_name or '打包'}_{ok_count}集.zip",
|
|
||||||
'audio_files': [],
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
_mark_task_terminal(task_id, 'canceled',
|
|
||||||
'任务已取消' + (f"(已退回 {refunded} 次下载次数)" if refunded else ''))
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
if t is not None:
|
|
||||||
t['refunded'] = refunded
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
api_service.logger.info(f"播客打包已取消: {task_id}(成功{ok_count} 退回{refunded}次)")
|
|
||||||
except DownloadException as e:
|
|
||||||
delivered = ok_count if zip_valid else 0
|
|
||||||
refunded = _settle_batch_billing(task_id, delivered)
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
_t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
# 终态先到先得——已终态(如用户取消/超时兜底)
|
|
||||||
# 不再被覆盖为 error(堵 canceled→error 非法迁移边)
|
|
||||||
if _t and _t.get('status') not in ('done', 'error', 'canceled'):
|
|
||||||
_t['status'] = 'error'
|
|
||||||
_t['completed_at'] = time.time() # 保留期从完成时刻起算
|
|
||||||
_t['msg'] = str(e) + (f"(已退回 {refunded} 次下载次数)" if refunded else '')
|
|
||||||
_t['refunded'] = refunded
|
|
||||||
elif _t:
|
|
||||||
_t['refunded'] = refunded
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
api_service.logger.error(f"播客打包失败: {task_id} {e}(退回{refunded}次)")
|
|
||||||
except Exception as e:
|
|
||||||
# 非预期异常原文 logger 先行,用户可见文案泛化
|
|
||||||
# (DownloadException 的文案是我们自己拼的用户可读文本,保留 str(e))
|
|
||||||
api_service.logger.error(f"播客打包异常: {task_id} {e}\n{traceback.format_exc()}")
|
|
||||||
delivered = ok_count if zip_valid else 0
|
|
||||||
refunded = _settle_batch_billing(task_id, delivered)
|
|
||||||
with DOWNLOAD_TASKS_LOCK:
|
|
||||||
_t = DOWNLOAD_TASKS.get(task_id)
|
|
||||||
# 同上——已终态不再覆盖为 error
|
|
||||||
if _t and _t.get('status') not in ('done', 'error', 'canceled'):
|
|
||||||
_t['status'] = 'error'
|
|
||||||
_t['completed_at'] = time.time() # 保留期从完成时刻起算
|
|
||||||
_t['msg'] = '打包失败,请稍后重试' + (f"(已退回 {refunded} 次下载次数)" if refunded else '')
|
|
||||||
_t['refunded'] = refunded
|
|
||||||
elif _t:
|
|
||||||
_t['refunded'] = refunded
|
|
||||||
if zip_tmp_path:
|
|
||||||
try:
|
|
||||||
os.remove(zip_tmp_path)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
api_service.logger.error(f"播客打包异常: {task_id} {e}(退回{refunded}次)\n{traceback.format_exc()}")
|
|
||||||
+135
-7
@@ -1,13 +1,17 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""两个 ZIP 打包 worker 的行为契约测试(沙箱,不触碰生产 data/)。
|
"""两个 ZIP 打包 worker 的行为契约测试(沙箱,不触碰生产 data/)。
|
||||||
|
|
||||||
为什么:``_batch_zip_worker`` / ``_podcast_zip_worker`` 合计约 642 行此前**零测试覆盖**,
|
为什么:``_batch_zip_worker`` / ``_podcast_zip_worker`` 合并前合计约 642 行、
|
||||||
而两者高度重复——连 ``_wait_if_paused`` 那 16 行都逐字相同(上一次修的
|
**零测试覆盖**,且两者高度重复——连 ``_wait_if_paused`` 那 16 行都逐字相同
|
||||||
``zip_valid`` 可能未绑定缺陷就是同型出现在这两处,重复代码在持续制造同类 bug)。
|
(``zip_valid`` 未绑定那类缺陷就是同型出现在这两处)。抽取公共实现
|
||||||
|
(``services/_zipcore.py``)之前,先用**行为级**测试钉住可观察契约:任务终态、
|
||||||
|
ZIP 内容、账本守恒、暂停/恢复/取消语义。断言只依赖「调用 worker 后世界里发生了
|
||||||
|
什么」,不引用任何内部函数名——所以合并实现时本文件一行未改,它同时就是等价性
|
||||||
|
验证器。
|
||||||
|
|
||||||
抽公共实现之前,先用**行为级**测试钉住可观察契约:任务终态、ZIP 内容、
|
末尾「合并后的契约固化」一节是**新增**的钉子:合并时逐条对齐了几处分叉
|
||||||
账本守恒、暂停/恢复/取消语义。断言只依赖「调用 worker 后世界里发生了什么」,
|
(详见各用例 docstring),这些用例在合并前必然失败——留着它们,那次对齐才不会
|
||||||
不引用任何内部函数名——这样重构时测试不必跟着改,才具备等价性验证能力。
|
被后来者当成「本来就如此」而回退。
|
||||||
|
|
||||||
账本口径:一律用 ``reserve_and_open`` 真预扣(``ledger_open`` 只记账不扣款,
|
账本口径:一律用 ``reserve_and_open`` 真预扣(``ledger_open`` 只记账不扣款,
|
||||||
会让「退款」凭空给余额加次数,守恒断言就失去意义)。
|
会让「退款」凭空给余额加次数,守恒断言就失去意义)。
|
||||||
@@ -15,6 +19,7 @@
|
|||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
import zipfile
|
import zipfile
|
||||||
|
from dataclasses import replace
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -23,7 +28,7 @@ import pytest
|
|||||||
import auth_system as auth
|
import auth_system as auth
|
||||||
import core.db as core_db
|
import core.db as core_db
|
||||||
from core.state import DOWNLOAD_TASKS, DOWNLOAD_TASKS_LOCK
|
from core.state import DOWNLOAD_TASKS, DOWNLOAD_TASKS_LOCK
|
||||||
from music_downloader import DownloadCanceled, DownloadResult, MusicInfo
|
from music_downloader import DownloadCanceled, DownloadException, DownloadResult, MusicInfo
|
||||||
from services import batch as batch_svc
|
from services import batch as batch_svc
|
||||||
from services import podcast as podcast_svc
|
from services import podcast as podcast_svc
|
||||||
from services.api_service import api_service
|
from services.api_service import api_service
|
||||||
@@ -276,6 +281,7 @@ def test_batch_zip_pause_then_cancel(sandbox, monkeypatch):
|
|||||||
|
|
||||||
assert _status(tid) == 'canceled', f"终态={_status(tid)} msg={task.get('msg')}"
|
assert _status(tid) == 'canceled', f"终态={_status(tid)} msg={task.get('msg')}"
|
||||||
assert task['result'] is None, '未交付不应发布 ZIP'
|
assert task['result'] is None, '未交付不应发布 ZIP'
|
||||||
|
assert task['msg'] == '任务已取消(已退回 3 次下载次数)', task['msg']
|
||||||
assert _balance() == before, '取消后应全额退回预扣'
|
assert _balance() == before, '取消后应全额退回预扣'
|
||||||
row = auth.ledger_get(tid)
|
row = auth.ledger_get(tid)
|
||||||
assert row is not None and row['delivered'] == 0 and row['status'] != 'open'
|
assert row is not None and row['delivered'] == 0 and row['status'] != 'open'
|
||||||
@@ -454,3 +460,125 @@ def test_podcast_zip_all_fail_refunds(podcast_stub, monkeypatch):
|
|||||||
assert _balance() == before, '全失败应全额退回'
|
assert _balance() == before, '全失败应全额退回'
|
||||||
row = auth.ledger_get(tid)
|
row = auth.ledger_get(tid)
|
||||||
assert row is not None and row['delivered'] == 0
|
assert row is not None and row['delivered'] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============ 合并后的契约固化(分叉点对齐 / 拆分风险点) ============
|
||||||
|
#
|
||||||
|
# 以下用例在「两个 worker 各写一份」的旧实现上**必然失败**——它们钉的是合并时
|
||||||
|
# 逐条对齐的结果,以及「下载」与「落包」拆成两个钩子后新增的边界。
|
||||||
|
|
||||||
|
class _SameNameDownloader(_FakeDownloader):
|
||||||
|
"""所有歌曲同名同歌手、且都带封面——用于验证包内三条目共享序号后缀。"""
|
||||||
|
|
||||||
|
def get_music_info(self, mid: Any, quality: str, cookies: Any) -> MusicInfo:
|
||||||
|
return replace(super().get_music_info(mid, quality, cookies),
|
||||||
|
name='同名歌', artists='同歌手',
|
||||||
|
pic_url='http://example.invalid/cover.jpg')
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_zip_dedups_three_arcnames_with_shared_suffix(sandbox, monkeypatch):
|
||||||
|
"""同名同歌手两首:音频/歌词/封面三条目**共享同一序号后缀**。
|
||||||
|
|
||||||
|
F-C 去重口径:同名同歌手但音源扩展不同(flac vs mp3)时,歌词/封面条目也会
|
||||||
|
互相覆盖。合并把「落包」拆成独立钩子后,去重读取的 ``used_arcnames`` 仍是
|
||||||
|
串行读改写,且三条目必须同步跳号(只给音频加序号就等于没修 F-C)。
|
||||||
|
"""
|
||||||
|
uid = _uid()
|
||||||
|
monkeypatch.setattr(api_service, 'downloader', _SameNameDownloader(sandbox['cache']))
|
||||||
|
monkeypatch.setattr(batch_svc, 'fetch_cover_bytes', lambda _u: b'\xff\xd8\xff\xe0')
|
||||||
|
tid, task = _register_batch(sandbox, [701, 702], reserved=2, user_id=uid)
|
||||||
|
|
||||||
|
th = _run(batch_svc._batch_zip_worker, tid)
|
||||||
|
th.join(timeout=30)
|
||||||
|
|
||||||
|
assert _status(tid) == 'done', f"终态={_status(tid)} msg={task.get('msg')}"
|
||||||
|
base = '同歌手 - 同名歌'
|
||||||
|
assert sorted(_zip_names(task['result']['zip_path'])) == sorted([
|
||||||
|
f'{base}.mp3', f'{base}.lrc', f'{base}.jpg',
|
||||||
|
f'{base} (2).mp3', f'{base} (2).lrc', f'{base} (2).jpg',
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_all_fail_message_and_ledger_status(sandbox, monkeypatch):
|
||||||
|
"""全失败:用户可见文案含条目数与退费说明;账本终态 error。
|
||||||
|
|
||||||
|
纯固化钉——合并前后的文案与账本终态在此路径上本就一致(对齐时以音乐侧为准)。
|
||||||
|
"""
|
||||||
|
uid = _uid()
|
||||||
|
monkeypatch.setattr(api_service, 'downloader',
|
||||||
|
_FakeDownloader(sandbox['cache'], fail_ids=(801, 802)))
|
||||||
|
tid, task = _register_batch(sandbox, [801, 802], reserved=2, user_id=uid)
|
||||||
|
|
||||||
|
th = _run(batch_svc._batch_zip_worker, tid)
|
||||||
|
th.join(timeout=30)
|
||||||
|
|
||||||
|
assert task['msg'] == '全部失败(2 首),已退回 2 次', task['msg']
|
||||||
|
row = auth.ledger_get(tid)
|
||||||
|
assert row is not None and row['status'] == 'error'
|
||||||
|
|
||||||
|
|
||||||
|
def test_podcast_all_fail_message_and_ledger_status(podcast_stub, monkeypatch):
|
||||||
|
"""全失败:文案含前 3 条失败明细与退费说明;**账本终态统一为 error**。
|
||||||
|
|
||||||
|
合并前这条路径走播客自己的 ``except DownloadException`` 处理器,账本落
|
||||||
|
``'settled'``,与音乐侧同一语义事件落的 ``'error'`` 不一致。整个代码库里
|
||||||
|
只有 ``'open'`` 被读取(对账、清理、计数),其余取值仅作排障留痕,故合并时
|
||||||
|
统一为语义更准确的 ``'error'``——本用例就是那次对齐的印记。
|
||||||
|
"""
|
||||||
|
uid = _uid()
|
||||||
|
before = _balance()
|
||||||
|
|
||||||
|
def _boom(prog, quality, cookies):
|
||||||
|
raise RuntimeError('上游不可用')
|
||||||
|
|
||||||
|
monkeypatch.setattr(podcast_svc, '_podcast_resolve_audio', _boom)
|
||||||
|
tid, task = _register_podcast(podcast_stub, [31, 32], reserved=2, user_id=uid)
|
||||||
|
|
||||||
|
th = _run(podcast_svc._podcast_zip_worker, tid)
|
||||||
|
th.join(timeout=30)
|
||||||
|
|
||||||
|
# 失败文案一律「单集名(异常)」——引擎的 fail_label 钩子,漏覆盖会退化成整个字典
|
||||||
|
assert task['msg'] == (
|
||||||
|
'全部单集下载失败: 单集31(上游不可用); 单集32(上游不可用)(已退回 2 次下载次数)'
|
||||||
|
), task['msg']
|
||||||
|
assert task['result'] is None, '不发布空包'
|
||||||
|
assert _balance() == before, '全失败应全额退回'
|
||||||
|
row = auth.ledger_get(tid)
|
||||||
|
assert row is not None and row['status'] == 'error' and row['delivered'] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_podcast_cleans_temp_files_on_episode_failure(podcast_stub, monkeypatch):
|
||||||
|
"""单集下载中途失败:任务目录不留中间音频与 .part 残片。
|
||||||
|
|
||||||
|
合并把「下载」与「落包」拆成两个钩子后,临时文件生命周期跨越两者:失败路径
|
||||||
|
由 download_one 自己清,成功路径由 write_entry 的 finally 清(原实现在同一段
|
||||||
|
try/finally 里,拆开后若只保留一处就会漏清)。
|
||||||
|
"""
|
||||||
|
uid = _uid()
|
||||||
|
before = _balance()
|
||||||
|
|
||||||
|
def _flaky(url: str, target: Path, total_size: int, cancel_ev: Any,
|
||||||
|
task_id: str = '', progress_label: str = '') -> int:
|
||||||
|
if '/102.' in url:
|
||||||
|
target.write_bytes(b'\x00' * 64) # 正文残留
|
||||||
|
target.with_name(target.name + '.part').write_bytes(b'\x00' * 32) # 残片
|
||||||
|
raise DownloadException('上游断流')
|
||||||
|
target.write_bytes(b'\xff\xfb\x90\x00' + b'\x00' * 128)
|
||||||
|
return target.stat().st_size
|
||||||
|
|
||||||
|
monkeypatch.setattr(podcast_svc, '_podcast_stream_download', _flaky)
|
||||||
|
tid, task = _register_podcast(podcast_stub, [101, 102], reserved=2, user_id=uid)
|
||||||
|
|
||||||
|
th = _run(podcast_svc._podcast_zip_worker, tid)
|
||||||
|
th.join(timeout=30)
|
||||||
|
|
||||||
|
assert _status(tid) == 'done', f"终态={_status(tid)} msg={task.get('msg')}"
|
||||||
|
# done 是「已完成条目数」(含失败),不是成功数——与旧实现同口径
|
||||||
|
assert (task['done'], task['failed']) == (2, 1)
|
||||||
|
assert task['msg'] == '成功1 失败1 退回1次', task['msg']
|
||||||
|
assert sum(1 for n in _zip_names(task['result']['zip_path']) if n.endswith('.mp3')) == 1
|
||||||
|
assert _balance() == before - 1, '只对已入包的 1 集扣费'
|
||||||
|
|
||||||
|
work_dir = podcast_stub['downloads'] / 'tasks' / tid
|
||||||
|
leftover = sorted(p.name for p in work_dir.glob('podcast_zip_tmp_*'))
|
||||||
|
assert leftover == [], f'中间文件未清理: {leftover}'
|
||||||
Reference in new issue
Block a user