AGENT SKILL

alpha-dqn-sdk

Turnkey operational guide and architectural standards for integrating, deploying, and building downstream reinforcement learning applications using the Alpha-DQN core SDK without modifying internal core code.

License 许可
MIT
最近更新
2026-11-11
Stars
★ 3
Forks
0

01 前言 Front Matter

name
alpha-dqn-sdk
description
Turnkey operational guide and architectural standards for integrating, deploying, and building downstream reinforcement learning applications using the Alpha-DQN core SDK without modifying internal core code.

03 SKILL.md 正文

Alpha-DQN Core SDK: Turnkey Integration & Operational Guide

This skill provides comprehensive instructions for building downstream reinforcement learning (RL) projects on top of the alpha-dqn production engine.

[!IMPORTANT] The Immutability Contract (Open-Closed Principle): src/core/ is strictly immutable. Downstream projects must NEVER modify or alter files in src/core/. All custom behaviors, domain adaptation, and hyperparameter tuning must be achieved via Configuration Objects, Environment Adapters, or Pluggable Hooks.


1. Architectural Overview & Component Topology

Alpha-DQN is a production-grade Double-DQN framework featuring an evolved algorithmic triad:

  1. Curiosity Novelty Engine (src/core/curiosity.py): Orthogonal QR RND projections, online IDM Noisy-TV filter, and episodic Euclidean frontier expansion boost.
  2. Adaptive Loss Formulation (src/core/loss.py): $C^\infty$-smooth Pseudo-Huber loss with asymmetric expectile weighting ($\tau=0.598$), robust MAD scale estimation, and dynamic EMA $\beta$ adaptation.
  3. Novel Directed Epistemic Exploration (src/core/exploration.py): 3–5 step sticky-action momentum ($12.6\times$ diffusion speed), continuous collision sensing, 3-tier epistemic count UCB, and confident Q-margin exploitation gating.
Downstream Project (Your Application)
   │
   ├── 1. Environment Adapter (Transforms domain data into float vectors)
   ├── 2. Configuration (DQNConfig, CuriosityConfig, etc.)
   └── 3. Orchestrator (In-process Python import OR FastAPI HTTP REST client)
         │
         ▼
┌───────────────────────────────────────────────────────────┐
│              ALPHA-DQN CORE SDK (IMMUTABLE)               │
│                                                           │
│  src/schema/       Data contracts & Pydantic models       │
│  src/core/agent    DQNAgent (Double-DQN + Vectorized MLP) │
│  src/core/loss     AdaptiveLoss (Pseudo-Huber Expectile)  │
│  src/core/explore  ExplorationPolicy (Sticky + UCB)       │
│  src/core/curiosity CuriosityEngine (RND + IDM + Frontier)│
│  src/core/buffer   High-throughput Circular Replay Buffer │
│  src/core/persist  JSON Manifest & PyTorch Checkpoints    │
└───────────────────────────────────────────────────────────┘

2. Standard 5-Step Integration Workflow

Whenever you build a new application using Alpha-DQN, follow these five sequential steps:

Step 1: Map Domain State to a Continuous Vector (state_size)

  • Extract numeric signals from your domain (e.g. coordinates, velocities, sensor readings, technical indicators).
  • Flatten and normalize observations into a 1D Python list of floats List[float].
  • Define state_size = len(observation_vector).

Step 2: Define Discrete Action Space (action_size)

  • Map discrete agent decisions into contiguous integer indices: 0, 1, ..., action_size - 1.
  • Example for trading: 0: HOLD, 1: BUY, 2: SELL $\to$ action_size = 3.
  • Example for grid/navigation: 0: UP, 1: DOWN, 2: LEFT, 3: RIGHT $\to$ action_size = 4.

Step 3: Instantiate Configuration

Construct a typed DQNConfig specifying network capacity, training hyperparameters, and desired modules:

from src.schema.dqn import (
    DQNConfig,
    CuriosityConfig,
    AdaptiveLossConfig,
    ExplorationConfig,
)

config = DQNConfig(
    state_size=16,
    action_size=4,
    layers=[128, 128],                  # Configurable MLP architecture
    dueling=True,                       # Separate Value and Advantage streams
    learning_rate=0.0005,
    gamma=0.99,
    batch_size=32,
    memory_capacity=10000,
    train_interval=1,                   # Train after every N steps
    target_update_interval=100,         # Sync target model every N updates
    
    # Evolved Triad Modules (all enabled by default or configurable)
    curiosity=CuriosityConfig(enabled=True, intrinsic_reward_scale=0.1),
    adaptive_loss=AdaptiveLossConfig(enabled=True, initial_beta=0.5, tau=0.598),
    exploration=ExplorationConfig(enabled=True, sticky_min=3, sticky_max=5),
)

Step 4: Initialize Agent

from src.core.agent import DQNAgent

# Device is detected automatically (CUDA if available, otherwise CPU)
agent = DQNAgent(config=config)
agent.init()

Step 5: Execute Standard Episode Loop

Interact with your environment using the standardized 4-call execution cycle:

for episode in range(total_episodes):
    state = env.reset()
    done = False
    
    while not done:
        # 1. Action Inference
        action = agent.act(state)
        
        # 2. Environment Step
        next_state, reward, done, _ = env.step(action)
        
        # 3. Store Transition & Auto-Compute Curiosity / Exploration
        agent.remember(state, action, reward, next_state, done)
        
        # 4. Vectorized Mini-batch Gradient Descent
        agent.train_batch()
        
        state = next_state
        
    # Reset episodic buffers (directional frontier, sticky counters)
    agent.reset_episode()

3. Turnkey Integration Recipes

Recipe A: Direct Python In-Process Implementation (Minimal Template)

Use this complete script pattern for standalone Python bots, simulators, and data pipelines:

import random
from src.schema.dqn import DQNConfig
from src.core.agent import DQNAgent

def run_application():
    # 1. Setup Configuration
    config = DQNConfig(
        state_size=4,
        action_size=2,
        batch_size=16,
        memory_capacity=5000,
    )
    
    # 2. Instantiate Agent
    agent = DQNAgent(config=config)
    agent.init()
    
    # 3. Execution Loop
    for ep in range(10):
        state = [random.uniform(-1.0, 1.0) for _ in range(config.state_size)]
        
        for step in range(50):
            action = agent.act(state)
            next_state = [s + random.gauss(0, 0.05) for s in state]
            reward = 1.0 if abs(next_state[0]) < 0.5 else -0.5
            done = step == 49
            
            agent.remember(state, action, reward, next_state, done)
            agent.train_batch()
            state = next_state
            
        agent.reset_episode()
        print(f"Episode {ep} finished. Loss: {agent.training_loss:.4f}, Epsilon: {agent.epsilon:.3f}")

    # 4. Save Weights
    agent.save("models/project_weights.json")

if __name__ == "__main__":
    run_application()

Recipe B: Gymnasium / OpenAI Gym Adapter

To connect any standard Gymnasium environment (e.g. CartPole-v1, LunarLander-v2):

import gymnasium as gym
from src.schema.dqn import DQNConfig
from src.core.agent import DQNAgent

def train_gym(env_name: str = "CartPole-v1", episodes: int = 50):
    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n
    
    config = DQNConfig(
        state_size=state_dim,
        action_size=action_dim,
        batch_size=64,
        learning_rate=0.001,
    )
    agent = DQNAgent(config=config)
    agent.init()
    
    for ep in range(episodes):
        obs, _ = env.reset()
        state = obs.tolist()
        total_reward = 0.0
        done = False
        
        while not done:
            action = agent.act(state)
            next_obs, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated
            
            agent.remember(state, action, float(reward), next_obs.tolist(), done)
            agent.train_batch()
            
            state = next_obs.tolist()
            total_reward += reward
            
        agent.reset_episode()
        print(f"Episode {ep+1}: Total Reward = {total_reward}")
        
    env.close()

Recipe C: REST API / Microservice Integration (Decoupled Clients)

When the environment is running in a browser, separate microservice, or different programming language:

  1. Start the API Server:

    uvicorn main:app --host 0.0.0.0 --port 8000
    
  2. Client Interaction (Python httpx / JavaScript fetch):

    import httpx
    
    API_BASE = "http://localhost:8000/api/dqn"
    
    # 1. Initialize custom agent session
    httpx.post(f"{API_BASE}/init", json={
        "session_id": "crypto_trader_01",
        "config": {
            "state_size": 10,
            "action_size": 3,
            "dueling": True,
            "curiosity": {"enabled": True}
        }
    })
    
    # 2. Get action
    act_resp = httpx.post(f"{API_BASE}/act", json={
        "session_id": "crypto_trader_01",
        "state": [0.12, 0.45, -0.3, 0.8, 1.2, -0.4, 0.0, 0.2, 0.5, 0.1]
    }).json()
    action = act_resp["action"]
    
    # 3. Send step transition
    step_resp = httpx.post(f"{API_BASE}/step", json={
        "session_id": "crypto_trader_01",
        "state": [0.12, 0.45, -0.3, 0.8, 1.2, -0.4, 0.0, 0.2, 0.5, 0.1],
        "action": action,
        "reward": 1.5,
        "next_state": [0.15, 0.48, -0.2, 0.9, 1.1, -0.3, 0.1, 0.3, 0.4, 0.2],
        "done": False,
        "auto_train": True
    }).json()
    
    # 4. Check real-time metrics
    metrics = httpx.get(f"{API_BASE}/metrics?session_id=crypto_trader_01").json()
    print("Current loss:", metrics["training_loss"], "Epsilon:", metrics["epsilon"])
    

Recipe D: DQfD Expert Policy Blending (Guided Exploration)

To accelerate training using human rules, heuristics, or a legacy algorithmic controller:

class HeuristicTradingExpert:
    """Provides estimated Q-values based on domain heuristics."""
    def get_q_values(self, state: list[float]) -> list[float]:
        # state[0] = RSI indicator
        rsi = state[0]
        if rsi < 30.0:
            return [0.1, 1.0, 0.0]  # Strong BUY preference
        elif rsi > 70.0:
            return [0.1, 0.0, 1.0]  # Strong SELL preference
        return [1.0, 0.1, 0.1]      # Strong HOLD preference

# Register expert with agent
expert = HeuristicTradingExpert()
agent.set_expert_policy(expert)

4. Operational Best Practices & Troubleshooting

Topic Best Practice Rationale
Path Operations Never use from pathlib import Path outside src/config/. Use src.config exports (exists, read_text, etc.). Preserves project-wide path sandboxing and architectural compliance.
Logging Never use raw print() statements in production code. Use logger = setup_logger(Settings.LOG_DIR / "layer.log"). Ensures clean structured logging and zero linter violations.
State Normalization Normalize all state variables to $[-1.0, 1.0]$ or $[0.0, 1.0]$ before passing to act() or remember(). Prevents exploding gradients in early exploration phases.
Reward Clamping Use reward_scale and reward_clamp inside DQNConfig for unstable environments. Stabilizes Bellman target calculations.
Episodic Resets Always call agent.reset_episode() when done=True. Clears trajectory-specific curiosity frontiers, sticky counters, and collision buffers.
Inference Mode Use agent.act(state, force_pure_neural=True) for evaluation/production benchmarks. Completely bypasses exploration, expert blending, and randomness for 100% deterministic evaluation.

5. Verification Checklist for New Implementations

Before deploying any new project built on Alpha-DQN, verify:

  • [ ] src/core/ files remain 100% untouched.
  • [ ] state_size and action_size match the downstream environment specification.
  • [ ] agent.reset_episode() is called at the boundary of each episode.
  • [ ] Tests run and pass cleanly via pytest tests/ -v.
  • [ ] Linter reports 0 violations via human-skills '{"tool_name": "linter", ...}'.

04 同仓库更多 Skills

mdnaimul22
mdnaimul22/human-skills

system-lifecycle-designer

Universal architectural methodology for designing granular, atomic, and sub-atomic lifecycles and end-to-end workflows for single functionality integrations across any domain (daemons, batch jobs, request transactions, resource pools). Provides patterns for deterministic identity, pre-commit validation, differential reconciliation, sticky leasing, state machines, transactional persistence, and closed-loop feedback.

★ 3
mdnaimul22
mdnaimul22/human-skills

style-sampling

Make a new episode in a proven video style - Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime - on the person's topic. Use for "an episode of", "in the style of", "sample this style", 「用这个风格做一集」. One video to copy is reverse-any-video.

★ 3
mdnaimul22
mdnaimul22/human-skills

reverse-any-video

Rebuild any video file as an editable HyperFrames project - motion graphics, talking heads, live action or AI-generated footage - checked against the original frame by frame. Use to reverse, recreate, remake, replicate or copy a video, or 「复刻这个视频」「照着这个视频做一个」.

★ 3
mdnaimul22
mdnaimul22/human-skills

typescript-patterns

TypeScript patterns for strict, type-safe, maintainable code in modern TypeScript and JavaScript applications. Covers strict compiler configuration, type design, narrowing, discriminated unions, generics, async and error handling, module boundaries, dependency hygiene, and type-level testing. Use when writing or reviewing TypeScript or JavaScript code.

★ 3
mdnaimul22
mdnaimul22/human-skills

test-collaboration

Inventory and govern a project's test assets. Connect requirements, business rules, risks, bugs, and cross-boundary contracts to stable TEST-IDs and executable evidence in the repository's existing test registry or TESTS.md. Use for test inventory, test gap analysis, unit/integration/contract/E2E/smoke classification, bug-to-test regression protection, consumer/provider contract testing, test-necessity review, and pre-delivery evidence review.

★ 3
mdnaimul22
mdnaimul22/human-skills

story-lifecycle

Manage a file-based epic → story → sprint development loop without requiring an external project management tool. Creates and tracks epics, breaks them into user stories, assigns them to sprints, and drives implementation story by story. Use when starting a new feature initiative or when the team has no Jira/Linear/Taiga but needs structured delivery tracking.

★ 3