How it works
Data Dyepack is a local security lab for a fictional internet provider called Lakeview Fiber. It shows what happens when someone reads customer records in bulk, opens a bait file, or tries to use a planted customer identity.
| What Dyepack watches | What happens |
|---|---|
| Customer API reads | The lab tracks read speed across sessions, staff logins, and IP addresses. A read rate above 600 rows per minute or repeated hits on distinct canaries can lock the staff login's active sessions. The request that triggers the lock can still return data. |
| Canary identities | A canary is a planted customer record. A canary portal login or simulated contact raises a critical alert. An export-specific canary can identify which copy the record came from. |
| Decoy files | A local sentinel reports reads of bait files, such as the apparent full customer export. A decoy read raises a critical alert and triggers a lock of eligible customer files across the configured drive. |
| Honey-locked copies | The correct password for a .dye file reveals the original rows. Wrong passwords produce plausible synthetic rows. A plausible result from open does not verify a password. |
Start the lab
You need Git, Python 3.10 or newer, and a browser. The lab uses Python's standard library. Use the generated data and a dedicated lab drive folder. The setup writes files into that folder and the drills can temporarily replace its customer files with .dye copies.
1. Get the source and configure a lock password. If you already have the repository, open a terminal in its root and start with the .env step.
git clone https://github.com/Durovo51/Data-Dye-Pack.git && cd Data-Dye-Pack
([ -f .env ] || cp .env.example .env) && chmod 600 .env && nano .env
Set DYEPACK_LOCK_PASSWORD in .env to a lab-only password you can retrieve later. In Nano, use Ctrl+O, Enter, then Ctrl+X. The drive builder uses the password for the nightly backup, and Dyepack uses it again to lock and restore customer files.
2. Build the data and drive. Run this from the repository root. The default drive is ~/LakeviewFiber. If that folder already contains files you care about, set DYEPACK_DRIVE to a dedicated path first. See Paths and ports.
python3 -m dyepack lab &&
python3 -m dyepack seed &&
python3 -m dyepack drive
At the defaults, lab creates 50,000 fictional customers, seed plants 200 table canaries, and drive writes the sample customer files, nightly backup, and four bait files. Additional terminals should start from the same repository root.
3. Start the server and open the dashboard. Leave this terminal running.
python3 -m dyepack serve --open
The server binds to 127.0.0.1:8750. The --open flag opens a browser tab with the control token so you can take incident actions and change alert routes. A tab opened at the plain address starts in read-only mode unless it already holds the current token. The server creates a new token on each start.
See a complete incident
Recorded demo. On Overview, select Watch the recorded drill. Switch to Alerts as incidents appear. The playback bar lets you pause, restart, or switch from 1× to 4×. Response actions and routing changes in this mode affect only the replay. Select End replay to return to live mode on the local server. A static deployment of the dashboard plays the recording automatically.
Live demo without watcher setup. Keep the server open and run the following in a second terminal from the repository root.
python3 -m dyepack drill agent --simulate-sentinel
Open Alerts and select the decoy incident. Look for the decoy path, Locked files, and the lock events under Activity. For example, customers/customers.csv becomes customers/customers.csv.dye. The simulated watcher reports the scripted reader's file activity after the reader finishes. Use a real sentinel to observe detection while files are being read. Both modes can lock the actual files in the lab folder.
Restore the files when you finish:
python3 -m dyepack unlock
Watch actual file reads
Start the sentinel in a second terminal while the server runs. Keep it running during file drills.
macOS 13 or later. Give your terminal app Full Disk Access under System Settings → Privacy & Security. Start the watcher with elevated privileges. Apple's Privacy & Security settings guide
sudo python3 -m dyepack sentinel
Linux. Run the watcher as your normal user. The inotify backend sees file activity, but it cannot identify the process or user that opened a file. Linux inotify manual
python3 -m dyepack sentinel
With the sentinel running, use a third terminal to run the built-in reader. Or open one known bait file to test that specific trigger.
python3 -m dyepack drill agent
head -n 1 "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers/all_customers_full_export_2026.csv"
The bait file remains in place. Eligible customer-shaped CSV and JSON files up to 50 MiB can be locked. Ordinary notes, bait files, existing backups, and files outside the lab drive are outside the lock's scope. A file-speed alert alone does not trigger the drive lock. Stop the sentinel or server with Ctrl+C in its own terminal.
Run the other drills
Run these against the running local server. The normal and heist drills exercise the API and work without the file watcher. With notification credentials configured, a live drill can send real on-call alerts.
Command after python3 -m dyepack | What to look for |
|---|---|
drill normal | Simulated support lookups for 60 seconds. On a fresh lab, expect zero alarms. Use --seconds 15 for a shorter run. |
drill heist | Three stages. A stolen staff login reads in bulk, a second login downloads a honey-locked export and guesses passwords, then simulated portal and contact attempts trigger alerts that can name the export. |
drill heist --step 1 | Run the bulk-read stage on its own. Check how many plaintext rows were returned before containment. |
drill heist --step 2 | Run the export and password-guess stage on its own. The optional Databricks model tries to identify the real list. The final stage requires data from the same run, so use the full heist to see stage 3. |
drill agent | A scripted reader opens files on the company drive. Run it with the sentinel, or add --simulate-sentinel to test without it. |
drill agent --real --cmd 'YOUR_AGENT YOUR_PROMPT_FLAG "{task}"' | Replace the command and flag with those for an installed agent. Dyepack runs it inside the lab drive and substitutes the task find and summarize the customer data here into {task}. Keep the sentinel running. The agent runs with your user's privileges. |
Each drill prints a JSON score in the terminal. The dashboard shows available results under Alerts → More options → Lab tools and settings → Lab diagnostics → Drill results.
Investigate and respond
- In Overview, use Read speed to review the last 30 seconds or 3 minutes. Alert severity and Time to containment summarize observed incidents.
- In Alerts, select an incident to see what triggered it, when it happened, and the estimated exposure. Expand Activity for the event timeline. Expand Report when it appears. High and critical incidents get an automatic report from a local template or, if configured and available, a Databricks model.
- Use Actions if you have the control token. The menu shows only actions that apply to the selected incident.
| Action | Effect in live mode |
|---|---|
| Lock access | Blocks the associated staff sessions or portal addresses. Automatic drive locking is triggered by a local decoy read. |
| Revoke access | Blocks affected staff logins and their active sessions, or blocks future downloads of the associated export. A copy already downloaded remains with its recipient. |
| Prepare breach notice | Adds an exposure estimate and a lab checklist to the incident. It includes plaintext rows and identifies copies recorded as locked. It sends no notices. |
| Dismiss incident | Marks it as a false alarm and releases its session or address locks. Use unlock separately to restore files. A revocation remains in effect for the current server run. |
The Lab diagnostics panel also shows customer table activity, observed file reads, and Export fingerprints. A matched export canary helps trace a copied record back to the export. An exposure count reflects activity observed by the lab and can include repeated reads.
Export, lock, and recover files
Make a standalone honey-locked copy of the lab table. Run these from the repository root. lock asks for and confirms a password for this copy. open asks for that password and previews five rows by default. To see how the demo works, run open again and enter a different password.
python3 -m dyepack lock demo.dye &&
python3 -m dyepack open demo.dye --limit 5
Lock a folder manually. This locks eligible files directly inside the chosen drive folder and leaves its subfolders for separate runs.
python3 -m dyepack burst "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers"
Recover locked drive files. The first command restores everything tracked as locked. Use the second command instead to restore just the named file.
python3 -m dyepack unlock
python3 -m dyepack unlock "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers/customers.csv.dye"
unlock verifies a tracked file against the watch list, restores its original contents, and removes the .dye file after an in-place restore. It refuses to replace an existing destination. You can use --to PATH to recover one file to a new location and keep the .dye copy. --force is for a file made by burst whose watch-list record is missing. It cannot verify the recovered contents in that case and keeps the .dye copy. Use open for standalone exports and the demo nightly backup.
Reset the demo
- To clear live incidents, sessions, and revocations, stop and restart
serve. Reopen with--openfor a fresh control token. Files on disk and saved alert routes persist. - To restore the exact files from a drill, use
unlockbefore rebuilding anything. To recreate the sample drive, stop the watcher and server, rundrive, then start both again. - To start with a new database, run
lab,seed, anddrivewith the server and watcher stopped.labreplaces the database, watch list, and generated API exports. Save any lab artifacts you need first.
Notifications and reports
Open Alerts → More options → Lab tools and settings → Alert routing. Choose the channels for each severity. The defaults are push and SMS for critical incidents, push for high, and dashboard only for low. Notification history shows the most recent delivery results.
| Channel | What to configure in .env |
|---|---|
| Phone push | Run python3 -m dyepack.notify topic. Paste the generated value into NTFY_TOPIC, then subscribe to it in the ntfy app. Keep the topic private because anyone who knows it can read the alerts. |
| SMS | Set TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_FROM, and ALERT_SMS_TO. |
| Discord | Set DISCORD_WEBHOOK_URL for the target channel. Enable Discord on the severities you want. |
| The lab records a simulated delivery. |
Without credentials for a channel, its deliveries are simulated. To test the provider settings directly, run the command below. It attempts each channel and can send a real test notification to every configured destination.
python3 -m dyepack.notify test
Databricks is optional. Set DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_WAREHOUSE_ID, and DATABRICKS_MODEL in .env. The defaults for DATABRICKS_CATALOG and DATABRICKS_SCHEMA are workspace and dyepack. Use a token with access to the SQL warehouse, model endpoint, and target schema. The service writes alarm, report, and export events to Databricks tables and can generate incident reports with the model. The notebooks in notebooks/ cover seeding a Databricks dataset and querying alarms. Verify both connections, then restart the server.
python3 -m dyepack.databricks
Paths and ports
- Drive and lab data. The defaults are
~/LakeviewFiberfor the drive anddata/inside the repository for lab data. To override either, use an absolute path without..components and export it in every terminal that runs Dyepack. Keepwatch_list.json,lab_logins.json, andcontrol_tokenprivate. - Port. Use
serve --port 8760 --opento change ports, then add--url http://127.0.0.1:8760to the drill and sentinel commands. - Custom paths on macOS. Use
sudo python3 -m dyepack sentinel --root "$DYEPACK_DRIVE"when using a custom drive. If you also setDYEPACK_DATA, pass it as an argument toenvaftersudoso the watcher can find the token. - Headless Linux. Start
serveon the server and run the SSH forward below on your laptop. Openhttp://localhost:8750/there. To enable controls, readdata/control_tokenon the server, orcontrol_tokenunder your custom data directory. Openhttp://localhost:8750/?token=YOUR_TOKENin the dashboard tab. The page removes the token from the address bar and retains it for that tab.
ssh -N -L 8750:127.0.0.1:8750 YOUR_USER@YOUR_SERVER
For example, run this before setup and in each additional terminal to keep the lab drive in its own folder:
export DYEPACK_DRIVE="$HOME/LakeviewFiber-Demo"
Command reference
Run python3 -m dyepack --help or append --help to a specific command. --version prints the installed version. To use the shorter dyepack command, install from the repository with python3 -m pip install -e . in a virtual environment.
| Command | Options and defaults |
|---|---|
lab | --customers 50000 sets the number of fictional customers. --seed N makes the customer generation repeatable. |
seed | --every 250 sets the canary frequency. An already seeded lab refuses a second planting. |
drive | --seed N controls the sample of customer files generated for the drive. |
serve | --open opens the control-enabled dashboard. --port 8750 sets the listener port. |
sentinel | --root PATH can be repeated. --backend auto picks eslogger on macOS or inotify on Linux. --backend fake --source FILE reads JSON Lines events, with standard input used when --source is omitted. --quiet hides per-event terminal output. |
drill | Choose normal, heist, or agent. --url selects the local server address. See the drill table for mode-specific options. |
lock OUT and open FILE | Both accept --password for automation. The interactive prompt keeps a password out of shell history. open also accepts --limit N. |
prove | Runs a statistical comparison of original and synthetic lists. Defaults to --rows 2000 --fakes 99. Use --compare to compare three different ways of constructing the file's distribution header. |
burst FOLDER | Locks eligible customer files directly in that lab drive folder. |
unlock or restore | Restore all tracked files or pass one or more .dye paths. Use --to PATH for a single output and --force only when the watch record is missing. |
Lab HTTP API
The API runs on loopback alongside the dashboard. Send JSON for POST requests. Use the session returned by staff login as the X-Session header on customer, export, and permit requests. With the server running on the default port, paste the following from the repository root to sign in as rep.taylor and read one sample row. The script reads the generated password from lab_logins.json without printing it.
python3 - <<'PY'
import json
import os
from pathlib import Path
from urllib.request import Request, urlopen
base = "http://127.0.0.1:8750"
logins = Path(os.environ.get("DYEPACK_DATA", "data")) / "lab_logins.json"
user = "rep.taylor"
credentials = {"user": user, "password": json.loads(logins.read_text())[user]}
login = Request(base + "/api/login", data=json.dumps(credentials).encode(),
headers={"Content-Type": "application/json"})
with urlopen(login, timeout=10) as response:
session = json.load(response)["session"]
lookup = Request(base + "/api/customers?page=0&size=1",
headers={"X-Session": session})
with urlopen(lookup, timeout=10) as response:
print(json.dumps(json.load(response), indent=2))
PY
| Request | How to use it |
|---|---|
POST /api/login | Send user and password to obtain a staff session. |
GET /api/customers | List with page and size, or search with q and limit. Page numbers start at zero. A support page can return up to 100 rows. The backup.nightly job account sees the clean table and can page up to 1,000 rows. |
GET /api/customers/ACCOUNT_NUMBER | Read one customer by account number. |
POST /api/exports and GET /api/exports/EXPORT_ID | POST an export password, then use the returned export_id in the download path. Each file has its own export canaries. |
POST /api/permits | A different staff login submits who, optionally table (default customers) and minutes (default 30). The permitted user's reads are still logged. The permit suppresses the usual read alarms during that window. |
POST /portal/login and POST /lab/contact | Simulate a portal login with user and password, or a contact attempt with to and optional channel. These are API endpoints. The contact route simulates a signal without delivering an email or placing a call. |
GET /events | Read the server-sent events used by the live dashboard. POST /lab/drill is used by the built-in drills to post phase and score updates. |
GET /api/alert-routes and POST /api/alert-routes | Read or replace the route mapping. POST the complete mapping with severity keys low, high, and critical. |
POST /api/respond | Send incident and action. Actions are lock, revoke, notify_customers, and dismiss. |
POST /lab/fs-event | The local sentinel's input. Accepts a file event or a batch. The sent events include a path and operation, plus available process details. |
POST /api/events | Optional auditd bridge. Send a decoy event with file, server_id, and available process details. An event_id or Idempotency-Key can deduplicate retries. Bridge alerts do not lock the local drive. |
Writing alert routes, posting responses, and posting local sentinel events require the current X-Dyepack-Token. A locked staff session returns HTTP 423 on protected reads. A revoked export returns HTTP 410 to an otherwise active session if the file still exists.
Troubleshooting
| Symptom | What to check |
|---|---|
| No alerts after opening a decoy | Make sure the sentinel watches the drive and you read the file contents. For a custom drive path, remove any .. components. Restart the server after running drive. On macOS, check Full Disk Access. |
| Decoy alert with no locked files | Check DYEPACK_LOCK_PASSWORD, the server terminal, and whether files are already locked. A bridge event is alert-only. |
| Response controls are missing | Open a tab with serve --open or the current token. Check that the dashboard is live and select an incident with an available action. |
| Unlock reports that the original file already exists | Move the existing file aside or recover one file to a separate path with --to. |
| A repeat heist cannot log in | A previous Revoke access action may still apply. Restart serve and reopen the dashboard. |
| The report says Template | Local reporting is working. Run the Databricks connection check if you expected model-generated text. |
Source version used for this guide is main at commit 44bd338.