hhhge/NeteaseUrl
Archived
1
0 Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

refactor: auth_system 全面异步化（aiosqlite），拆掉 web/services 层的 to_thread 桥接

Browse Source 
拆掉两类垫片的最后一处：auth_system 原为同步 sqlite3 栈，web 层与 services 的
worker 只能靠 asyncio.to_thread 逐个桥接才不阻塞事件循环——每次 DB 访问都白付一次
线程池调度，且「同步调异步」的边界散落在 100+ 个调用点。B1 已把 worker 变成协程，
本批把访问层本身也迁到原生异步，两条线在此合拢。

改动（按子系统）：
- auth_system：db.py 照 core/db.py 现有范式重建为原生异步访问层（aiosqlite + WAL）
  ——读路径 _q 每次独立连接、无锁、可并发；写路径 _x / write_transaction 用
  asyncio.Lock 串行化，写者不阻塞读者。users / cards / credits / ledger / logs /
  settings / mail / geo / backup 九模块随之全部 async/await；security.py 是纯计算
  （pbkdf2/normalize）不动。__init__ 去掉 _DB_LOCK 与 get_conn 导出。
- web：app / deps / billing / tasks_registry 与 routers/{admin_site,auth,music,
  register,user} 直接 await，拆掉包裹的 to_thread。deps.secret_key 改 async，
  连带 _parse_cookie 也 async（会话解析要读密钥）。
- services：_taskcore / tasks / downloads / podcast 的账本调用与查询直接 await。
  4 处 start_task 调用点（download.py）确认已 await。

设计取舍（为什么这么做，而不是直接包一层）：
- 锁按**事件循环**隔离（_LOOP_LOCKS 以 id(loop) 为键）：asyncio.Lock 绑定创建它的
  loop，应用主 loop 与测试各自的 loop 并存，跨 loop 复用会抛 "bound to a different
  event loop"。
- WAL 切换用**独立锁** _wal_lock 而非 _write_lock：execute() 已持写锁，再在锁内
  connect→_configure 会自锁死（asyncio.Lock 非重入）。切换 + 标记在专用锁内串行并
  二次复查，否则并发首连会同时执行 PRAGMA journal_mode=WAL 争排他锁报
  database is locked（core/db.py 同款教训，实测复现）。
- force_wal 保留：journal_mode 持久化在库文件头，库文件被删重建时新文件不继承，
  必须重设。exists() 判定须在 aiosqlite.connect **之前**（connect 会创建空文件）。
- connect() 用 DB_PATH.parent 兜底建目录，优于 DATA_DIR：生产两者相同，但测试
  monkeypatch 只换 DB_PATH 时兜底仍能命中正确目录。

留在 to_thread 上的仅剩**正当的阻塞卸载**，不是同步门面：PBKDF2 的 10 万次迭代
（登录链路每次都要跑）、smtplib 发信、备份文件的 copy2 / os.replace、secret.key
的文件 IO。原有 aria2 / downloader / zip 侧的 to_thread 未动。

不变量逐条保持（迁移只改执行模型，不改语义）：
- ledger_settle 幂等：账本非 open 直接 return 0，重启后重放不重复退款；
- 终态先到先得：_mark_task_terminal 对已 done/error/canceled 的任务直接 return，
  堵住 canceled→error 的非法迁移边；
- settle 必须在任务锁**外**调用（ledger_settle 要抢 SQLite 写锁，busy 最长 15s，
  持任务锁会让全站进度/提交/取消随库停摆）；
- reserve_and_open 的预扣与开账同事务，消除原两步之间的崩溃丢账窗口。

路径真源红线：DATA_DIR / DB_PATH / BACKUP_DIR / SECRET_KEY_PATH 仍是 auth_system.db
的**模块全局**（用与 core.db 相同的表达式，但**不是**它的别名）。测试只 patch
auth_system.db.*，若改成 core.db 的别名会静默把测试写到生产库。已核 data/site.db
读写前后 sha256 完全一致。
顺带清掉 get_conn：迁移后它只剩「转调 connect()」一层空壳、全仓零消费者（测试
直接开裸连接、备份用 db.connect()），保留等于留一条「拿裸连接自己写」的口子。

测试：19 个文件同步迁移——10 个文件的 web.deps.secret_key 替身改 async；
test_db_conn.py 的多线程首连用例按异步语义拆成 2 条（单 loop gather(16) 覆盖批量
并发首连、WAL 就绪后 8 线程各自 loop 建连覆盖跨 loop）；test_billing_scope.py 去掉
多余的 @pytest.mark.anyio；被 await 且**正常返回**的 monkeypatch 替身一律改 async
（仅 raise 的同步替身仍可用）。

门禁：ruff 全过 / pyright 0-0-0（standard 模式含 tests 与顶层模块）/ pytest
384 项全绿（0 失败 0 error 0 skipped；383→384 因 test_db_conn 拆用例）。
data/site.db 读写前后 sha256 一致（a19f5db…6487），无 -wal/-shm 边车残留。

下一步 B3：删 core/sync_http.py、core/async_bridge.py、core/http_errors.py 及
 tests/test_async_bridge.py——三者在生产侧已零消费者（http_errors 仅 sync_http
 自用，requests 已非声明依赖）。
This commit is contained in:
snowy-yang committed 2026-09-28 20:13:19 +08:00
1 parent 69a50c2c31
commit 96af56cc27
43 files changed
+1171 -1009
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

web/routers/admin_site.py
+47 -48
View File
@@ -3,8 +3,9 @@
鉴权：与 auth_system.admin_required 同口径（403 + error_code='forbidden'）。
说明：这些端点多为对 auth_system 同步函数（SQLite / SMTP / 文件）的薄封装，
统一经 asyncio.to_thread 执行以不阻塞事件循环；操作者名取当前管理员用户名。
说明：这些端点多为对 auth_system 的薄封装。auth_system 已迁移为**原生异步**
（aiosqlite / aiosmtplib 语义下的 to_thread 卸载），故直接 await；仅 PBKDF2 校验
（CPU 密集）与本地文件 stat 仍经 asyncio.to_thread 卸载。操作者名取当前管理员用户名。
"""
import asyncio
import math
@@ -74,14 +75,14 @@ def _int_or(value: Any, default: int = 0) -> int:
async def _log(admin: SiteUser, action: str, detail: str = '') -> None:
await asyncio.to_thread(auth.add_log, 'success', admin.username, '-', action, detail)
await auth.add_log('success', admin.username, '-', action, detail)
# ============ 用户管理 ============
@router.post('/admin/site/users')
async def users(_admin: AdminUser):
rows = await asyncio.to_thread(auth.list_users)
rows = await auth.list_users()
return success({'users': rows})
@@ -90,8 +91,8 @@ async def user_delete(admin: AdminUser, request: Request):
data = await _request_data(request)
uid = _int_or(data.get('id'))
try:
target = await asyncio.to_thread(auth.get_user_by_id, uid)
ok = await asyncio.to_thread(auth.delete_user, uid)
target = await auth.get_user_by_id(uid)
ok = await auth.delete_user(uid)
if not ok:
return error('用户不存在或为管理员')
await _log(admin, '删除用户', target['username'] if target else str(uid))
@@ -114,12 +115,12 @@ async def user_duration(admin: AdminUser, request: Request):
if count < -1:
return error('次数不能小于 -1（-1 表示不限次数）')
try:
target = await asyncio.to_thread(auth.get_user_by_id, uid)
target = await auth.get_user_by_id(uid)
if not target:
return error('用户不存在')
if target['role'] == 'admin':
return error('管理员不限次数')
await asyncio.to_thread(auth.set_user_credits, uid, count)
await auth.set_user_credits(uid, count)
await _log(admin, '修改用户次数', f'{target["username"]} → {count} 次')
return success(message=f'已修改为 {count} 次')
except Exception as e:
@@ -147,12 +148,11 @@ async def user_create(admin: AdminUser, request: Request):
if len(password) < 8 or len(password) > 128:
return error('密码长度需在 8-128 位之间')
try:
if await asyncio.to_thread(auth.get_user_by_name, username):
if await auth.get_user_by_name(username):
return error('用户名已存在')
if email and await asyncio.to_thread(auth.get_user_by_email, email):
if email and await auth.get_user_by_email(email):
return error('该邮箱已被注册，请更换邮箱')
await asyncio.to_thread(
auth.create_user, username, password, email=email, credits=credits,
await auth.create_user(username, password, email=email, credits=credits,
guard=auth.RegistrationGuard(ip='-', ua='admin创建'))
await _log(admin, '添加用户', f'{username} · {credits} 次')
return success(message=f'用户 {username} 创建成功（{credits} 次）')
@@ -168,7 +168,7 @@ async def user_reset_password(admin: AdminUser, request: Request):
if len(newpwd) < 8 or len(newpwd) > 128:
return error('新密码长度需在 8-128 位之间')
try:
target = await asyncio.to_thread(auth.get_user_by_id, uid)
target = await auth.get_user_by_id(uid)
if not target:
return error('用户不存在')
if target['role'] == 'admin':
@@ -179,10 +179,10 @@ async def user_reset_password(admin: AdminUser, request: Request):
target['password_hash'], oldpwd, target['salt'])
if not ok:
return error('原密码错误')
await asyncio.to_thread(auth.reset_password, uid, newpwd)
await auth.reset_password(uid, newpwd)
await _log(admin, '修改自己的密码', '管理员')
return success(message='密码已修改，下次登录请使用新密码（其他设备已全部下线）')
await asyncio.to_thread(auth.reset_password, uid, newpwd)
await auth.reset_password(uid, newpwd)
await _log(admin, '重置密码', target['username'])
return success(message=f'已重置 {target["username"]} 的密码')
except Exception as e:
@@ -194,14 +194,14 @@ async def user_toggle(admin: AdminUser, request: Request):
data = await _request_data(request)
uid = _int_or(data.get('id'))
try:
target = await asyncio.to_thread(auth.get_user_by_id, uid)
target = await auth.get_user_by_id(uid)
if not target:
return error('用户不存在')
if target['role'] == 'admin':
return error('不能对管理员账号执行该操作')
cur = target['status'] if 'status' in target.keys() else 1
new_status = 0 if cur else 1
await asyncio.to_thread(auth.set_user_status, uid, new_status)
await auth.set_user_status(uid, new_status)
await _log(admin, '禁用/启用',
f'{target["username"]} → {"启用" if new_status else "禁用"}')
return success({'status': new_status},
@@ -217,17 +217,17 @@ async def user_ban(admin: AdminUser, request: Request):
action = str(data.get('action') or 'ban')
reason = str(data.get('reason') or '')
try:
row = await asyncio.to_thread(auth.get_user_by_id, uid)
row = await auth.get_user_by_id(uid)
if not row:
return error('用户不存在')
if row['role'] == 'admin':
return error('不能对管理员执行该操作')
if action == 'ban':
if not await asyncio.to_thread(auth.ban_user, uid, reason or '管理员手动封禁'):
if not await auth.ban_user(uid, reason or '管理员手动封禁'):
return error('封禁失败')
await _log(admin, '封禁用户', f'{row["username"]}: {reason or "手动封禁"}')
return success(message=f'已封禁用户「{row["username"]}」，该账号已强制下线')
if not await asyncio.to_thread(auth.unban_user, uid):
if not await auth.unban_user(uid):
return error('解封失败')
await _log(admin, '解封用户', row['username'])
return success(message=f'已解封用户「{row["username"]}」')
@@ -239,7 +239,7 @@ async def user_ban(admin: AdminUser, request: Request):
async def user_cookie(_admin: AdminUser, request: Request):
data = await _request_data(request)
username = str(data.get('username') or '')
user = await asyncio.to_thread(auth.get_user_by_name, username)
user = await auth.get_user_by_name(username)
if not user:
return error('用户不存在', 404)
return success({'username': username, 'cookie': user['netease_cookie'] or ''})
@@ -249,7 +249,7 @@ async def user_cookie(_admin: AdminUser, request: Request):
@router.post('/admin/site/smtp/get')
async def smtp_get(_admin: AdminUser):
cfg = await asyncio.to_thread(auth.get_smtp_config)
cfg = await auth.get_smtp_config()
configured = bool(cfg.get('host') and cfg.get('sender') and cfg.get('password'))
return success({'host': cfg.get('host', ''), 'port': cfg.get('port', ''),
'sender': cfg.get('sender', ''), 'configured': configured})
@@ -269,9 +269,9 @@ async def smtp_save(admin: AdminUser, request: Request):
return error('SMTP 服务器与发件邮箱不能为空')
try:
if not password:
cfg = await asyncio.to_thread(auth.get_smtp_config)
cfg = await auth.get_smtp_config()
password = cfg.get('password', '')
await asyncio.to_thread(auth.save_smtp_config, host, port, sender, password)
await auth.save_smtp_config(host, port, sender, password)
await _log(admin, '保存SMTP配置', f'{host}:{port}')
return success(message='SMTP 配置已保存')
except Exception as e:
@@ -287,12 +287,12 @@ async def smtp_test(admin: AdminUser, request: Request):
if not _EMAIL_RE.match(to):
return error('收件邮箱格式不正确')
try:
await asyncio.to_thread(auth.send_email, to, 'SMTP 测试邮件',
await auth.send_email(to, 'SMTP 测试邮件',
'这是一封来自云村小助手的测试邮件，收到即说明 SMTP 配置成功。')
await _log(admin, 'SMTP测试', to)
return success(message='测试邮件已发送，请查收')
except Exception as e:
await asyncio.to_thread(auth.add_log, 'fail', admin.username, '-', 'SMTP测试', str(e))
await auth.add_log('fail', admin.username, '-', 'SMTP测试', str(e))
return error(f'发送失败: {e!s}', 500)
@@ -303,7 +303,7 @@ async def logs(_admin: AdminUser, request: Request):
data = await _request_data(request)
log_type = 'fail' if str(data.get('type')) == 'fail' else 'success'
page = max(1, _int_or(data.get('page'), 1))
rows, total = await asyncio.to_thread(auth.query_logs, log_type, 10, (page - 1) * 10)
rows, total = await auth.query_logs(log_type, 10, (page - 1) * 10)
return success({'logs': rows, 'type': log_type, 'page': page,
'total': total, 'pages': max(1, (total + 9) // 10)})
@@ -312,7 +312,7 @@ async def logs(_admin: AdminUser, request: Request):
async def logs_clear(admin: AdminUser, request: Request):
data = await _request_data(request)
log_type = 'fail' if str(data.get('type')) == 'fail' else 'success'
deleted = await asyncio.to_thread(auth.clear_logs, log_type)
deleted = await auth.clear_logs(log_type)
await _log(admin, '清除日志', f'{log_type} · {deleted} 条')
return success(message=f'已清除 {"失败" if log_type == "fail" else "成功"}日志 {deleted} 条')
@@ -335,8 +335,7 @@ async def cards_generate(admin: AdminUser, request: Request):
if credits <= 0:
return error('卡密次数必须大于 0')
try:
codes = await asyncio.to_thread(
auth.generate_cards, count, credits, created_by=admin.username,
codes = await auth.generate_cards(count, credits, created_by=admin.username,
prefix=str(data.get('prefix') or 'VIP'), note=str(data.get('note') or ''),
valid_days=valid_days)
await _log(admin, '生成卡密', f'{count} 张 · {credits} 次/张 · 有效{valid_days or "永久"}天')
@@ -351,7 +350,7 @@ async def cards_list(_admin: AdminUser, request: Request):
data = await _request_data(request)
limit = _int_or(data.get('limit'), 100) or 100
offset = _int_or(data.get('offset'), 0)
cards, total = await asyncio.to_thread(auth.list_cards, limit=limit, offset=offset)
cards, total = await auth.list_cards(limit=limit, offset=offset)
return success({'cards': cards, 'total': total})
@@ -363,7 +362,7 @@ async def cards_disable(admin: AdminUser, request: Request):
return error('缺少卡密')
disable = str(data.get('disabled') or '').strip().lower() in ('1', 'true')
try:
await asyncio.to_thread(auth.set_card_disabled, code, disable)
await auth.set_card_disabled(code, disable)
await _log(admin, '禁用卡密' if disable else '恢复卡密', code[:4] + '****')
return success(message=('已禁用 ' if disable else '已恢复 ') + code[:4] + '****')
except ValueError as e:
@@ -379,7 +378,7 @@ async def cards_delete(admin: AdminUser, request: Request):
if not code:
return error('缺少卡密')
try:
await asyncio.to_thread(auth.delete_card, code)
await auth.delete_card(code)
await _log(admin, '删除卡密', code[:4] + '****')
return success(message='已删除 ' + code[:4] + '****')
except ValueError as e:
@@ -392,14 +391,14 @@ async def cards_delete(admin: AdminUser, request: Request):
@router.post('/admin/site/ann/get')
async def ann_get(_admin: AdminUser):
return success(await asyncio.to_thread(auth.get_announcement))
return success(await auth.get_announcement())
@router.post('/admin/site/ann/save')
async def ann_save(admin: AdminUser, request: Request):
data = await _request_data(request)
try:
await asyncio.to_thread(auth.save_announcement, str(data.get('title') or ''),
await auth.save_announcement(str(data.get('title') or ''),
str(data.get('content') or ''), str(data.get('popup_mode') or 'off'))
await _log(admin, '保存公告', str(data.get('title') or ''))
return success(message='公告已保存')
@@ -411,17 +410,17 @@ async def ann_save(admin: AdminUser, request: Request):
@router.post('/admin/site/play_key/get')
async def play_key_get(_admin: AdminUser):
key = await asyncio.to_thread(auth.get_setting, 'playlist_play_key', '')
key = await auth.get_setting('playlist_play_key', '')
if not key:
key = secrets.token_urlsafe(16)
await asyncio.to_thread(auth.set_setting, 'playlist_play_key', key)
await auth.set_setting('playlist_play_key', key)
return success({'key': key})
@router.post('/admin/site/play_key/rotate')
async def play_key_rotate(admin: AdminUser):
key = secrets.token_urlsafe(16)
await asyncio.to_thread(auth.set_setting, 'playlist_play_key', key)
await auth.set_setting('playlist_play_key', key)
await _log(admin, '刷新播放API密钥', '旧密钥已失效')
return success({'key': key}, '密钥已刷新，旧密钥已全部失效')
@@ -431,8 +430,8 @@ async def play_key_rotate(admin: AdminUser):
@router.post('/admin/site/backup/create')
async def backup_create(admin: AdminUser):
try:
name = await asyncio.to_thread(auth.create_backup)
await asyncio.to_thread(auth.set_setting, 'last_backup_ts', str(time.time()))
name = await auth.create_backup()
await auth.set_setting('last_backup_ts', str(time.time()))
await _log(admin, '数据库备份', name)
return success({'name': name}, f'备份完成：{name}')
except Exception as e:
@@ -441,8 +440,8 @@ async def backup_create(admin: AdminUser):
@router.post('/admin/site/backup/list')
async def backup_list(_admin: AdminUser):
backups = await asyncio.to_thread(auth.list_backups)
interval = await asyncio.to_thread(auth.get_setting, 'backup_interval_hours', '0')
backups = await auth.list_backups()
interval = await auth.get_setting('backup_interval_hours', '0')
def _size() -> int:
return auth.db.DB_PATH.stat().st_size if auth.db.DB_PATH.exists() else 0
@@ -454,7 +453,7 @@ async def backup_list(_admin: AdminUser):
@router.get('/admin/site/backup/download')
async def backup_download(_admin: AdminUser, request: Request):
name = request.query_params.get('name') or ''
p = await asyncio.to_thread(auth.backup_path, name)
p = await auth.backup_path(name)
if not p:
return error('备份文件不存在', 404)
resp = FileResponse(str(p), filename=name)
@@ -492,7 +491,7 @@ async def backup_upload(admin: AdminUser, request: Request):
return error('不是有效的 SQLite 数据库文件')
await _log(admin, '上传数据库', dest.name)
try:
await asyncio.to_thread(auth.prune_backups, 20, 5)
await auth.prune_backups(20, 5)
except Exception:
pass
return success({'name': dest.name}, '上传成功，可在下方选择该备份进行恢复')
@@ -510,13 +509,13 @@ async def backup_restore(admin: AdminUser, request: Request):
data = await _request_data(request)
name = str(data.get('name') or '')
try:
await asyncio.to_thread(auth.restore_backup, name)
await auth.restore_backup(name)
await _log(admin, '恢复数据库', name)
return success(message='数据库已恢复，请刷新页面')
except ValueError as e:
return error(f'恢复失败: {e!s}', 400)
except Exception as e:
await asyncio.to_thread(auth.add_log, 'fail', admin.username, '-', '恢复数据库', str(e))
await auth.add_log('fail', admin.username, '-', '恢复数据库', str(e))
return error(f'恢复失败: {e!s}', 500)
@@ -525,7 +524,7 @@ async def backup_delete(_admin: AdminUser, request: Request):
data = await _request_data(request)
name = str(data.get('name') or '')
try:
if await asyncio.to_thread(auth.delete_backup, name):
if await auth.delete_backup(name):
return success(message='备份已删除')
return error('备份文件不存在', 404)
except Exception as e:
@@ -544,7 +543,7 @@ async def backup_auto(_admin: AdminUser, request: Request):
if hours < 0:
return error('间隔不能为负')
try:
await asyncio.to_thread(auth.set_setting, 'backup_interval_hours', str(hours))
await auth.set_setting('backup_interval_hours', str(hours))
return success(message=f'定时备份已开启，每 {hours} 小时自动备份' if hours > 0
else '定时备份已关闭')
except Exception as e: