hhhge/NeteaseUrl
Archived
1
0 Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

refactor: auth_system 全面异步化（aiosqlite），拆掉 web/services 层的 to_thread 桥接

Browse Source 
拆掉两类垫片的最后一处：auth_system 原为同步 sqlite3 栈，web 层与 services 的
worker 只能靠 asyncio.to_thread 逐个桥接才不阻塞事件循环——每次 DB 访问都白付一次
线程池调度，且「同步调异步」的边界散落在 100+ 个调用点。B1 已把 worker 变成协程，
本批把访问层本身也迁到原生异步，两条线在此合拢。

改动（按子系统）：
- auth_system：db.py 照 core/db.py 现有范式重建为原生异步访问层（aiosqlite + WAL）
  ——读路径 _q 每次独立连接、无锁、可并发；写路径 _x / write_transaction 用
  asyncio.Lock 串行化，写者不阻塞读者。users / cards / credits / ledger / logs /
  settings / mail / geo / backup 九模块随之全部 async/await；security.py 是纯计算
  （pbkdf2/normalize）不动。__init__ 去掉 _DB_LOCK 与 get_conn 导出。
- web：app / deps / billing / tasks_registry 与 routers/{admin_site,auth,music,
  register,user} 直接 await，拆掉包裹的 to_thread。deps.secret_key 改 async，
  连带 _parse_cookie 也 async（会话解析要读密钥）。
- services：_taskcore / tasks / downloads / podcast 的账本调用与查询直接 await。
  4 处 start_task 调用点（download.py）确认已 await。

设计取舍（为什么这么做，而不是直接包一层）：
- 锁按**事件循环**隔离（_LOOP_LOCKS 以 id(loop) 为键）：asyncio.Lock 绑定创建它的
  loop，应用主 loop 与测试各自的 loop 并存，跨 loop 复用会抛 "bound to a different
  event loop"。
- WAL 切换用**独立锁** _wal_lock 而非 _write_lock：execute() 已持写锁，再在锁内
  connect→_configure 会自锁死（asyncio.Lock 非重入）。切换 + 标记在专用锁内串行并
  二次复查，否则并发首连会同时执行 PRAGMA journal_mode=WAL 争排他锁报
  database is locked（core/db.py 同款教训，实测复现）。
- force_wal 保留：journal_mode 持久化在库文件头，库文件被删重建时新文件不继承，
  必须重设。exists() 判定须在 aiosqlite.connect **之前**（connect 会创建空文件）。
- connect() 用 DB_PATH.parent 兜底建目录，优于 DATA_DIR：生产两者相同，但测试
  monkeypatch 只换 DB_PATH 时兜底仍能命中正确目录。

留在 to_thread 上的仅剩**正当的阻塞卸载**，不是同步门面：PBKDF2 的 10 万次迭代
（登录链路每次都要跑）、smtplib 发信、备份文件的 copy2 / os.replace、secret.key
的文件 IO。原有 aria2 / downloader / zip 侧的 to_thread 未动。

不变量逐条保持（迁移只改执行模型，不改语义）：
- ledger_settle 幂等：账本非 open 直接 return 0，重启后重放不重复退款；
- 终态先到先得：_mark_task_terminal 对已 done/error/canceled 的任务直接 return，
  堵住 canceled→error 的非法迁移边；
- settle 必须在任务锁**外**调用（ledger_settle 要抢 SQLite 写锁，busy 最长 15s，
  持任务锁会让全站进度/提交/取消随库停摆）；
- reserve_and_open 的预扣与开账同事务，消除原两步之间的崩溃丢账窗口。

路径真源红线：DATA_DIR / DB_PATH / BACKUP_DIR / SECRET_KEY_PATH 仍是 auth_system.db
的**模块全局**（用与 core.db 相同的表达式，但**不是**它的别名）。测试只 patch
auth_system.db.*，若改成 core.db 的别名会静默把测试写到生产库。已核 data/site.db
读写前后 sha256 完全一致。
顺带清掉 get_conn：迁移后它只剩「转调 connect()」一层空壳、全仓零消费者（测试
直接开裸连接、备份用 db.connect()），保留等于留一条「拿裸连接自己写」的口子。

测试：19 个文件同步迁移——10 个文件的 web.deps.secret_key 替身改 async；
test_db_conn.py 的多线程首连用例按异步语义拆成 2 条（单 loop gather(16) 覆盖批量
并发首连、WAL 就绪后 8 线程各自 loop 建连覆盖跨 loop）；test_billing_scope.py 去掉
多余的 @pytest.mark.anyio；被 await 且**正常返回**的 monkeypatch 替身一律改 async
（仅 raise 的同步替身仍可用）。

门禁：ruff 全过 / pyright 0-0-0（standard 模式含 tests 与顶层模块）/ pytest
384 项全绿（0 失败 0 error 0 skipped；383→384 因 test_db_conn 拆用例）。
data/site.db 读写前后 sha256 一致（a19f5db…6487），无 -wal/-shm 边车残留。

下一步 B3：删 core/sync_http.py、core/async_bridge.py、core/http_errors.py 及
 tests/test_async_bridge.py——三者在生产侧已零消费者（http_errors 仅 sync_http
 自用，requests 已非声明依赖）。
This commit is contained in:
snowy-yang committed 2026-09-28 20:13:19 +08:00
1 parent 69a50c2c31
commit 96af56cc27
43 files changed
+1171 -1009
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

auth_system/users.py
+89 -97
View File
@@ -1,9 +1,10 @@
# -*- coding: utf-8 -*-
"""用户增删改查、登录校验与状态。
（自 auth_system.py 拆出，行为与原实现一致。）
（自 auth_system.py 拆出，行为与原实现一致；DB 访问已迁移为原生异步。）
"""
import asyncio
import json
import sqlite3
import time
@@ -11,10 +12,9 @@ from dataclasses import dataclass
from typing import Any
from .db import (
_DB_LOCK,
_q,
_x,
get_conn,
write_transaction,
)
from .security import (
_hash_new,
@@ -27,7 +27,7 @@ from .security import (
class RegistrationGuard:
"""注册风控上下文：同 IP 频率闸 + 设备赠送闸。
这些检查必须与建号 INSERT 在**同一把 _DB_LOCK 事务内**完成——锁外预检存在
这些检查必须与建号 INSERT 在**同一写事务内**完成——锁外预检存在
TOCTOU（并发同设备双注册各得一次赠送、并发同 IP 挤过限额），所以它们不是
「可选的附加参数」，而是事务内必须可见的输入。管理建号无风控需求，不传即可。
"""
@@ -40,9 +40,9 @@ class RegistrationGuard:
device_once: bool = False
def create_user(username: str, password: str, email: str = '', credits: int = 0,
role: str = 'user', *,
guard: RegistrationGuard | None = None) -> sqlite3.Row | None:
async def create_user(username: str, password: str, email: str = '', credits: int = 0,
role: str = 'user', *,
guard: RegistrationGuard | None = None) -> sqlite3.Row | None:
# 余额只接受 -1（不限）或 >= 0；非法负值一律按 0 处理，禁止出现 -2 这类"假不限"
try:
credits = int(credits)
@@ -51,165 +51,157 @@ def create_user(username: str, password: str, email: str = '', credits: int = 0,
if credits < -1:
credits = 0
# 注册入口传 guard=RegistrationGuard(ip_limit=3, device_once=True)——同 IP 24h
# 注册数检查在 _DB_LOCK 事务内与 INSERT 原子完成，堵住「锁外检查 + 并发插入」
# 注册数检查在写事务内与 INSERT 原子完成，堵住「锁外检查 + 并发插入」
# 的 TOCTOU 刷号；管理建号不传 guard（即不限制）。超限抛 ValueError 由调用方转 400/429。
guard = guard or RegistrationGuard()
now = int(time.time())
with _DB_LOCK:
conn = get_conn()
try:
if guard.ip_limit > 0 and guard.ip:
n = conn.execute('SELECT COUNT(*) AS n FROM users WHERE register_ip = ? AND created_at > ?',
(guard.ip, now - guard.ip_window)).fetchone()[0]
if n >= guard.ip_limit:
raise ValueError('当前网络注册过于频繁，请 24 小时后再试或联系管理员')
if guard.device_once and guard.device_id:
# 设备赠送闸并入同一事务——锁外 device_used 预检有 TOCTOU，
# 并发同设备双注册可各得一次赠送；锁内复查命中则本次建号不赠送（账号照常创建）
n = conn.execute("SELECT COUNT(*) AS n FROM users WHERE trial_device = ? AND role != 'admin'",
(guard.device_id,)).fetchone()[0]
if n > 0:
credits = 0
conn.execute(
'INSERT INTO users (username, password_hash, salt, email, role, duration_seconds, '
'trial_device, register_ip, register_ua, last_active, created_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
(username, _hash_new(password), '', email, role,
-1 if role == 'admin' else int(credits),
guard.device_id, guard.ip, guard.ua, now, now))
conn.commit()
finally:
conn.close()
return get_user_by_name(username)
# PBKDF2 10 万次迭代是 CPU 密集（~50ms）：在事务外先算好，避免持写锁时
# 阻塞事件循环（正当的 CPU 卸载，非同步门面）。
phash = await asyncio.to_thread(_hash_new, password)
async with write_transaction() as conn:
if guard.ip_limit > 0 and guard.ip:
cur = await conn.execute(
'SELECT COUNT(*) AS n FROM users WHERE register_ip = ? AND created_at > ?',
(guard.ip, now - guard.ip_window))
row = await cur.fetchone()
n = int(row['n']) if row else 0
if n >= guard.ip_limit:
raise ValueError('当前网络注册过于频繁，请 24 小时后再试或联系管理员')
if guard.device_once and guard.device_id:
# 设备赠送闸并入同一事务——锁外 device_used 预检有 TOCTOU，
# 并发同设备双注册可各得一次赠送；锁内复查命中则本次建号不赠送（账号照常创建）
cur = await conn.execute(
"SELECT COUNT(*) AS n FROM users WHERE trial_device = ? AND role != 'admin'",
(guard.device_id,))
row = await cur.fetchone()
if row and int(row['n']) > 0:
credits = 0
await conn.execute(
'INSERT INTO users (username, password_hash, salt, email, role, duration_seconds, '
'trial_device, register_ip, register_ua, last_active, created_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
(username, phash, '', email, role,
-1 if role == 'admin' else int(credits),
guard.device_id, guard.ip, guard.ua, now, now))
return await get_user_by_name(username)
def get_user_by_name(username: str) -> sqlite3.Row | None:
rows = _q('SELECT * FROM users WHERE username = ?', (username,))
async def get_user_by_name(username: str) -> sqlite3.Row | None:
rows = await _q('SELECT * FROM users WHERE username = ?', (username,))
return rows[0] if rows else None
def get_user_by_email(email: str) -> sqlite3.Row | None:
async def get_user_by_email(email: str) -> sqlite3.Row | None:
"""按邮箱查用户（注册/建号邮箱查重的友好提示；库侧另有唯一索引兜底并发）"""
email = _normalize_email(email)
if not email:
return None
rows = _q('SELECT * FROM users WHERE email = ? LIMIT 1', (email,))
rows = await _q('SELECT * FROM users WHERE email = ? LIMIT 1', (email,))
return rows[0] if rows else None
def register_ip_exceeded(ip: str, limit: int = 3, window: int = 86400) -> bool:
async def register_ip_exceeded(ip: str, limit: int = 3, window: int = 86400) -> bool:
"""同 IP 24h 内注册数超限（默认 3 次）——防 IP 轮换刷免费赠送次数；
设备指纹可被客户端伪造，IP 维度是第二道闸。家庭 NAT 场景超限可由管理员建号兜底。"""
if not ip:
return False
rows = _q('SELECT COUNT(*) AS n FROM users WHERE register_ip = ? AND created_at > ?',
(ip, int(time.time() - window)))
rows = await _q('SELECT COUNT(*) AS n FROM users WHERE register_ip = ? AND created_at > ?',
(ip, int(time.time() - window)))
return bool(rows and rows[0]['n'] >= limit)
def get_user_by_id(uid: int) -> sqlite3.Row | None:
rows = _q('SELECT * FROM users WHERE id = ?', (uid,))
async def get_user_by_id(uid: int) -> sqlite3.Row | None:
rows = await _q('SELECT * FROM users WHERE id = ?', (uid,))
return rows[0] if rows else None
def verify_login(username: str, password: str) -> sqlite3.Row | None:
async def verify_login(username: str, password: str) -> sqlite3.Row | None:
"""校验用户名密码。密码正确即返回用户行（含被禁用/封禁账号——
状态语义由登录路由判断：禁用显示"已被禁用"、封禁显示封禁原因。
原实现把禁用账号提前过滤成 None，路由的"账号已被禁用"分支永不可达）"""
user = get_user_by_name(username)
user = await get_user_by_name(username)
if user:
ok, needs_upgrade = _verify_password(user['password_hash'], password, user['salt'])
# PBKDF2 校验是 CPU 密集（~50ms），卸载到线程避免阻塞事件循环（正当 CPU 卸载）
ok, needs_upgrade = await asyncio.to_thread(
_verify_password, user['password_hash'], password, user['salt'])
if ok:
if needs_upgrade:
# 存量 SHA-256 哈希在登录成功时透明升级为 PBKDF2
_x('UPDATE users SET password_hash = ?, salt = \'\' WHERE id = ?',
(_hash_new(password), user['id']))
new_hash = await asyncio.to_thread(_hash_new, password)
await _x('UPDATE users SET password_hash = ?, salt = \'\' WHERE id = ?',
(new_hash, user['id']))
return user
return None
def list_users() -> list[dict[str, Any]]:
rows = _q('SELECT * FROM users ORDER BY id DESC')
async def list_users() -> list[dict[str, Any]]:
rows = await _q('SELECT * FROM users ORDER BY id DESC')
return [_user_public(r, full=True) for r in rows]
def delete_user(uid: int) -> bool:
async def delete_user(uid: int) -> bool:
"""删除用户（卡密 used_by 为用户名字符串无外键——删除时同步
标注"（已注销#id）"，管理端卡密列表不再显示悬空的已删用户名）"""
u = get_user_by_id(uid)
u = await get_user_by_id(uid)
if not u or u['role'] == 'admin':
return False
with _DB_LOCK:
conn = get_conn()
try:
name = u['username']
conn.execute('DELETE FROM users WHERE id = ? AND role != ?', (uid, 'admin'))
conn.execute("UPDATE cards SET used_by = used_by || ? WHERE used_by = ?",
(f'（已注销#{uid}）', name))
conn.commit()
finally:
conn.close()
async with write_transaction() as conn:
name = u['username']
await conn.execute('DELETE FROM users WHERE id = ? AND role != ?', (uid, 'admin'))
await conn.execute("UPDATE cards SET used_by = used_by || ? WHERE used_by = ?",
(f'（已注销#{uid}）', name))
return True
def reset_password(uid: int, new_password: str) -> bool:
async def reset_password(uid: int, new_password: str) -> bool:
"""重置密码（同时递增 login_seq，让所有已签发会话立即失效）"""
with _DB_LOCK:
conn = get_conn()
try:
conn.execute('UPDATE users SET password_hash = ?, salt = \'\', login_seq = COALESCE(login_seq, 0) + 1 WHERE id = ?',
(_hash_new(new_password), uid))
conn.commit()
return True
finally:
conn.close()
new_hash = await asyncio.to_thread(_hash_new, new_password)
await _x('UPDATE users SET password_hash = ?, salt = \'\', '
'login_seq = COALESCE(login_seq, 0) + 1 WHERE id = ?',
(new_hash, uid))
return True
def set_user_status(uid: int, status: int) -> bool:
async def set_user_status(uid: int, status: int) -> bool:
"""启用 / 禁用账号（1 启用，0 禁用）；管理员账号不可禁用"""
if status not in (0, 1):
return False
row = _q('SELECT role FROM users WHERE id = ?', (uid,))
row = await _q('SELECT role FROM users WHERE id = ?', (uid,))
if row and row[0]['role'] == 'admin':
return False
return bool(_x('UPDATE users SET status = ? WHERE id = ?', (status, uid)))
return bool(await _x('UPDATE users SET status = ? WHERE id = ?', (status, uid)))
def ban_user(uid: int, reason: str = '') -> bool:
async def ban_user(uid: int, reason: str = '') -> bool:
"""封禁账号（管理员不可封）"""
row = _q('SELECT role FROM users WHERE id = ?', (uid,))
row = await _q('SELECT role FROM users WHERE id = ?', (uid,))
if not row or row[0]['role'] == 'admin':
return False
return bool(_x('UPDATE users SET status = 2, ban_reason = ? WHERE id = ?',
(str(reason or '')[:200], uid)))
return bool(await _x('UPDATE users SET status = 2, ban_reason = ? WHERE id = ?',
(str(reason or '')[:200], uid)))
def unban_user(uid: int) -> bool:
async def unban_user(uid: int) -> bool:
"""解封账号，恢复正常（清空触发封禁的省份历史，否则下次登录立即再封）"""
return bool(_x('UPDATE users SET status = 1, ban_reason = \'\', recent_provinces = \'[]\' WHERE id = ?', (uid,)))
return bool(await _x('UPDATE users SET status = 1, ban_reason = \'\', recent_provinces = \'[]\' WHERE id = ?', (uid,)))
def set_user_credits(uid: int, count: int) -> None:
async def set_user_credits(uid: int, count: int) -> None:
"""设置用户剩余次数（-1 表示不限）"""
count = -1 if count < 0 else int(count)
_x('UPDATE users SET duration_seconds = ? WHERE id = ?', (count, uid))
await _x('UPDATE users SET duration_seconds = ? WHERE id = ?', (count, uid))
def set_user_password(uid: int, password: str) -> None:
async def set_user_password(uid: int, password: str) -> None:
# 新哈希 + 递增 login_seq（所有旧会话立即失效）
with _DB_LOCK:
conn = get_conn()
try:
conn.execute('UPDATE users SET password_hash = ?, salt = \'\', '
'login_seq = COALESCE(login_seq, 0) + 1 WHERE id = ?',
(_hash_new(password), uid))
conn.commit()
finally:
conn.close()
new_hash = await asyncio.to_thread(_hash_new, password)
await _x('UPDATE users SET password_hash = ?, salt = \'\', '
'login_seq = COALESCE(login_seq, 0) + 1 WHERE id = ?',
(new_hash, uid))
def set_user_netease_cookie(username: str, cookie: str) -> None:
_x('UPDATE users SET netease_cookie = ? WHERE username = ?', (cookie, username))
async def set_user_netease_cookie(username: str, cookie: str) -> None:
await _x('UPDATE users SET netease_cookie = ? WHERE username = ?', (cookie, username))
def _user_public(r: sqlite3.Row, full: bool = False) -> dict[str, Any]:
@@ -249,8 +241,8 @@ def _last_province(r: sqlite3.Row) -> str:
# ============ 免费体验（设备识别） ============
def device_used(device_id: str) -> bool:
async def device_used(device_id: str) -> bool:
if not device_id:
return False
rows = _q("SELECT id FROM users WHERE trial_device = ? AND role != 'admin' LIMIT 1", (device_id,))
rows = await _q("SELECT id FROM users WHERE trial_device = ? AND role != 'admin' LIMIT 1", (device_id,))
return bool(rows)