# Flyff Universe Leveling Bot — Project Overview

## Goal
Automate gameplay in **Flyff Universe** (browser MMORPG) with a **modular,
multi-mode bot**. First mode: autonomously target and attack nearby monsters one
after another, and heal via a food item (keyboard shortcut, default `"1"`) when
HP drops low. Later modes (e.g. a ringmaster/assist healer that follows, buffs,
and heals a party) plug into the same core.

## ⚠️ Risk posture (read first)
Automating gameplay **violates Flyff Universe's Terms of Service**; Gala Lab bans
for botting. Personal/educational project:
- Run only on an account the owner is willing to lose. **Never on a main.**
- Prefer semi-attended use. Keep input human-like; add modest timing jitter.
- Avoid network-layer evasion. The higher the "superhuman" factor, the higher the
  ban risk — we deliberately favor trusted, human-like input.

## The core technical constraint
Flyff Universe is **not a normal web page**:
- Renders entirely to **canvas / WebGL** — there is **no HTML/DOM** game state.
- Game logic runs in **WebAssembly (WASM)**.
- Client⇄server traffic is a **binary WebSocket**.
- **Canvas/WASM games typically ignore *synthetic* input** (`isTrusted === false`).
  This single fact drives the connection-layer decision below.

Consequence: the bot operates like a human — **sense** via screen pixels (and
optionally decoded WebSocket frames), **act** via **trusted** input events.

## Connection layer — DECISION: Playwright (Python) over CDP

We drive a real Chromium via **Playwright** (Chrome DevTools Protocol). Rationale:

| Approach | Acts (input accepted)? | Senses? | Verdict |
|---|---|---|---|
| Chrome extension / userscript | ❌ synthetic events often ignored | WS hook ok; pixels awkward | No — control problem |
| OS-level (`mss` + `pydirectinput`) | ✅ real OS input | ✅ screen capture | Works, but hijacks physical mouse/kbd + needs foreground focus. **Fallback only.** |
| **Playwright / Puppeteer (CDP)** | ✅ **trusted** input | ✅ screenshot + JS eval + WS hook | ✅ **Chosen** |
| Custom browser (CEF/Electron) | ✅ | ✅ | Overkill; Playwright already gives programmatic Chromium |
| AWS / cloud backend | same, remote | same | Deferred (latency, GPU/WebGL cost, datacenter-IP ban risk) |

**Why Playwright wins:** CDP dispatches input at the *browser* level
(`Input.dispatchKeyEvent` / `dispatchMouseEvent`), so events are **trusted** and
the game accepts them — the thing extensions can't do. It also gives screenshots
(→ OpenCV sensing), `page.evaluate()` (→ optional WebSocket state hook), and
network interception, all in one Python process alongside our CV stack. This
replaces `mss` (use `page.screenshot()`) and `pydirectinput` (use
`page.keyboard`/`page.mouse`).

**Connecting to the live game — two supported ways:**
- **Launch + persistent profile:** Playwright launches Chromium with a persistent
  `user_data_dir`; log in once, session persists across runs.
- **Attach to existing Chrome:** start Chrome with `--remote-debugging-port=9222`,
  then `playwright.chromium.connect_over_cdp("http://localhost:9222")` to drive
  the already-open window.

Must run **headed** (WebGL); keep `deviceScaleFactor = 1` so screenshot pixels map
1:1 to mouse CSS coordinates.

### Deferred ideas (designed-for, not built)
- **Custom browser:** only justified to patch the client/WASM — not needed for
  input+pixels automation. Skip.
- **AWS backend:** keep the bot **core UI-agnostic and config-driven** so it could
  later run on a cloud GPU VM for 24/7 farming, but don't build it now (latency in
  the see→act loop, WebGL needs a GPU instance, datacenter IPs raise ban risk).

## Sensing
- **Pixels (primary):** `page.screenshot()` → NumPy/OpenCV. Read HP/MP/EXP-bar pixels
  for triggers + the header; `cv2.inRange`/template-match to find mobs, the aggro bubble,
  the shop window, and vendor tags.
- **OCR (Tesseract via pytesseract):** exact HP/MP/Exp text (status header), and the item
  names + prices in private shops (ShopScanner). Heavy (~1 s for the status block) — kept off
  the hot loop. Bot still grinds without it (pixel reads).
- **WebSocket hook (optional/advanced):** override `window.WebSocket` via
  `page.add_init_script` to read **reliable** state (real HP, mob list) from
  incoming binary frames. The wall is decoding WASM-serialized binary — a
  days-to-weeks RE project, brittle across patches. Pursue only if explicitly
  wanted; even then, prefer reading state this way while still **acting** via
  trusted input (don't craft outgoing packets — most detectable/bannable).

## Architecture: pluggable modes on a shared core

```
BotRunner (main loop)
  ├─ always-on safety: panic/abort key, critical-HP behavior
  └─ active Mode.tick(ctx)         ctx = { Sensor, Actuator, Config }

Sensor    screenshot(), hp_fraction(), find_mobs(),
          [later] find_player(name), party_hp(), buff_timers()
Actuator  press(key), key_combo(), click(x,y), ctrl_click(x,y)   (all trusted/CDP)
Config    keybinds (food="1"), thresholds, cooldowns, per-mode params  (one file)
```

### Modes (all live; dispatched in `control.run_session`)
- **`grind_local` / `grind_selected`:** vision auto-attack — find mob → two-click engage →
  hold-until-dead → repeat; food-heal under a threshold; aggro-bubble fast-path.
- **`heal` / `heal+buff`:** a follower **Ringmaster** — spam the heal key on an interval, and
  (heal+buff) run a **two-class buff rotation** (each class on its own refresh timer, a pre-buff
  cast first, heals interspersed) + threshold/interval MP potions + anti-detect actions.
- **`shopscan` (ShopScanner):** walk a city, OCR open private-shop windows into a price database
  (`shop_db.json`), viewable/searchable/exportable in the panel.
- **`calibrate`:** reset the camera to the tuned reference view.

Adding a mode = a new branch in `run_session` (+ a sub-panel) reusing the Sensor/Actuator/Config
primitives — not a rewrite. Panel params live in `window.__grind/__heal/__shop`.

### Key Flyff facts that shape the design
- **`Ctrl + Click` on a monster = walk to it and auto-attack until it dies.** Once
  targeted this way, no attack-spam needed.
- **No native "target nearest monster" hotkey** → the attack loop needs vision.
- `Tab` de-selects current target.
- Action-bar slots: `Ctrl+1..0` and `Alt+1..0`, plus single keys. Food default `1`.

## Targeting strategies (grind mode), easiest → hardest
1. **Stationary click-scan:** stand in a mob-dense spot; `Ctrl+Click` a few fixed
   screen points in a cycle. No vision. Crude but AFK-capable. *First milestone.*
2. **Color/nameplate detection:** `cv2.inRange` on nameplate color → cluster →
   `Ctrl+Click` nearest → wait until its nameplate/target bar vanishes → repeat.
   *The real "attack nearby mobs in sequence."*
3. **Template matching:** `cv2.matchTemplate` on saved monster screenshots. Precise, brittle.

## Confirmed combat mechanics (verified live)
- **Attack = two separate left-clicks** on the mob (select, then attack); character
  then **auto-attacks to the death**. ~0.28s between clicks. **Never re-click
  mid-fight** (cancels). `dblclick`/`Ctrl+Click` do NOT attack (select/NPC only).
- **Fight-live signal:** top-center target frame present, detected by its **RED HP / CYAN-BLUE MP bar**
  (`target_bar_run` — a wide band; the MP bar is HP-independent so it holds to 0% HP). Replaced the old
  gold-name signal, which yellowish terrain false-fired. Gone = dead, confirmed by N consecutive absent
  polls (`DEATH_CONFIRM_CONSECUTIVE_ABSENT`) so a flicker doesn't drop a live mob.
- Detected blue-blob centroid sits **on the body** → click it directly (dy=0).

## Roadmap / milestones
- **M0 — Connection:** ✅ CDP input trusted; DPR=1, coords map 1:1.
- **M1 — Heal loop:** ✅ HP-bar pixel; heals on `1` at <50%.
- **M2 — Working grind (aibats):** ✅ blue-colour detect → two-click engage →
  hold-until-dead → chain kills. (Original `grind.py`/`find_aibat.py`.)
- **M3 — Control panel + modes:** ✅ In-game panel (`control.py`) with
  **grind_selected / grind_local / grind_aibat / calibrate / stop**, live status,
  and **hot-reload** of `grind_local.py`. Detection generalised beyond one colour.
- **M4 — Robust detection:** ✅(ish) **name-tag detection** (colour-agnostic),
  **`calibrate_view`** (consistent zoom + top-down), **selection-arrows** attack
  point, UI/player exclusion. Hard case remaining: red mobs on tan ground.
- **M5 — Multi-instance launcher:** ✅ `flyffbot.bat`/`flyffbot.py` — one window +
  profile + panel per account; run several in parallel.
- **M6 — Polish/robustness (ACTIVE — current focus):** reliable engage on spread-out/
  low-contrast mobs; safety stops (flee/logout at critical HP); timing jitter; logging.
  **This is where we are now** — refining `grind_local.py` directly (see M7 note).
- **M7 — Memory-reading sensing (TRIED, parked):** read the game's already-decrypted
  state from WASM linear memory (`HEAPU8`) — mob ids/world positions are ROCK-SOLID.
  BUT turning a memory mob position into an accurate on-screen click needs a reliable
  world→screen projection, and **that bridge was never made to work** (matrix-scan,
  homography, vision-snap all had too much error → clicks hit ground → character walked).
  Built `grind_local_2.py` (memory sense + vision-nameplate clicks + memory death-detect);
  it ≈ vision targeting and still underperforms. **Conclusion: the limiter is engage
  accuracy, not sensing — so we pivoted back to M6 (refine the vision bot).** Memory data
  + WASM-RE remain available for later (HP-from-memory still blocked by ECS realloc; vision
  HP works). Tools: `memread.py`, `hpoverlay.py`, `grind_local_2.py`.
- **M8 — `assist`/ringmaster mode:** ✅ **built** — `heal` (heal-spam) + `heal+buff` (two-class
  buff rotation, pre-buff, quick-buff, threshold MP potions, anti-detect) follower support modes.
- **M9 — Tabbed control panel + scanners:** ✅ per-script sub-panels; **Status Scanner** (HP/MP/EXP
  calibration → `scanner_calib.json`; OCR + PIXEL header rows; EXP/min); **EXP graphs**; grind-region
  calibration dashboard.
- **M10 — ShopScanner + price DB:** ✅ template-located shop window, white-isolation OCR of item
  names/prices → `shop_db.json`; **Shop DB** tab (search, per-item history, count, CSV export);
  best-effort auto-roam.
- **M11 — Distribution:** ✅ one-click **`install.bat`** (Python/venv/deps/Chromium/Tesseract),
  **`prune_for_share.bat`** (clean shareable copy), `requirements.txt`, `START_HERE.txt`.
- **M12 — Grind loop speed:** ✅ tuned (2026-06-28) — faster search (rank once, engage only when a
  mob is visible, lower `CYCLE`), faster fight-end (`FIGHT_POLL` + N-absent death confirm), faster
  engage (reuse the IDLE frame, lower `ENGAGE_SETTLE`/`SELECT_GAP`); fight-timing tunables exposed in
  the panel's ⚙️ Advanced dropdown.
- **M13+ —** harder-mob engage ACCURACY (ongoing), faster screenshots (JPEG/clip — needs OCR/threshold
  validation), then (optional) cloud.

## Network / WASM reverse-engineering (ACTIVE — `reverseEngineeringWASM/`)
Commands are **built and encrypted inside the WASM** (8.5 MB, 11,581 functions, stripped) and
sent via embind — plaintext never reaches JS. Rather than fight that blindly, we **disassembled
the WASM**, split it per-function, built a mechanical call graph + import-name map, and have
AI agents analyse functions outward from the captured `ws.send` anchor.
**Live data-collection (`packet_log.py` → `packet_logs/`):** while a script runs, every WS frame
**and** every bot action are logged to a per-session timeline. Since the body is XOR-encrypted but
direction/length/opcode/header/**timing** are visible, correlating "action at T" with "SENT frame
just after T" maps each action to its opcode WITHOUT decrypting — the practical path to identifying
attack/heal/buff packets (move opcode 1054 is already known and validates the method).
**Progress:** the outgoing send path is mapped and the **cipher is found** — func **2779**:
XOR every payload byte with a **32-byte key at mem 338112**, then a **CRC-32** (table @338144)
written at packet offset +5 (this is the "bytes that change every message" we saw, not a nonce).
The full outgoing **frame format is solved** (header `1f 37`+length+CRC32, body XOR-encrypted;
move opcode = 1054, serializer = func 4112). Only the live 32-byte key (mem 338112) is needed to
craft a packet.
**Memory-read sensor (M7 — HP + MP SOLVED 2026-07-01):** `controller = *(mem 1230588)` (stable) and IS
the player's entity. **HP/MP are anti-cheat XOR-obfuscated**, so `curHP = i32@(controller+2768) ^
i32@(controller+2744)` (key@+2744), `curMP = +2772 ^ K` — exact, vision-free, confirmed live (315→147
on damage). maxHP/MP = base stat formula (misses gear/buff) → peak-track. This is why value-scans
failed (obfuscated + entity reallocates); the stable controller sidesteps both. The old "+308/+316 HP
ratio" was WRONG (func 11585 = 3D-model deserializer). Mobs: same read on registry entities (@362748;
also plaintext pos/kind/id). **Exp parked** (no stable address). Read via `re_sense.py`; shown in the
panel header **MEM row**. Tools: `re_sense.py`, `reverseEngineeringWASM/{schema,findprop}.js`,
`re/grab_wasm.py`, `memread.py`.
**Docs:** `reverseEngineeringWASM/reverseEngineeringWasm_Overview.md` (plain-English status) ·
`reverseEngineeringWASM/wasm_overview.md` (technical per-function lookup + memory map) ·
`ReverseEngineerWasm_PLAN.md` (strategy/method/phases).

### Current files
`flyffbot.py`+`.bat` (launcher) · `control.py` (panel supervisor + `CONTROL_HTML` UI + heal/
heal+buff/quickbuff loops + `update_*` dashboards, hot-reload) · `grind_local.py` (vision bot:
detection, modes, calibration, engage, fight loop, status/OCR, aggro) · `shop_scanner.py` +
`shop_probe.py` (ShopScanner + price DB, hot-reloaded) · `grind_local_2.py` (parked memory hybrid) ·
`scanner_calib.json` (HP/MP/EXP bar + grind-region calibration) · `shop_db.json` (price DB) ·
`templates/` (menu + `shop_anchor.png` match templates) · `session.py`/`host.py` (dev CDP harness) ·
**distribution:** `install.bat`, `prune_for_share.bat`, `requirements.txt`, `START_HERE.txt`,
`.gitignore` · `grind.py`/`find_aibat.py` (original aibat grinder) · `shots.py` + `screenshots/` ·
probes (`shot.py`, `engage_watch.py`, `hp_diag.py`, …) · `re/main-wasm32.{js,wasm}` + 
`reverseEngineeringWASM/` (parked WASM RE).

## Environment (confirmed)
- Windows 11 Pro; PowerShell primary (Bash tool available).
- Python 3.12.10, pip 25.0.1; Node v22.15.0.
- Display 1920×1200. **All screen coordinates are resolution/UI-scale specific —
  always calibrate; never hard-code blindly.**
- Project root: `C:\Users\Aaron\Desktop\flyffBot` (not a git repo yet).
- Flyff runs in a Chrome window.

## Source references
- Flyff hotkeys: https://madrigalinside.com/guides/beginners-guide/hotkey-assignment/
- Flyff controls (Gala Lab): https://www.en-galalab.flyff.com/game-control
- Flyff WebSockets & WASM (community): https://www.elitepvpers.com/forum/flyff/5310484-flyff-universe-websockets-wasm.html
- Playwright (Python): https://playwright.dev/python/
- Monster DBs (for M3 mob database/seeding): Flyffipedia https://flyffipedia.com/monsters ·
  Madrigal Inside https://madrigalinside.com/game-data/monsters/ · flyff.me https://www.flyff.me/monsters
