Data Dyepack user guide

Set up the Lakeview Fiber lab, run a security drill, and investigate what happened.

How it works

Data Dyepack is a local security lab for a fictional internet provider called Lakeview Fiber. It shows what happens when someone reads customer records in bulk, opens a bait file, or tries to use a planted customer identity.

What Dyepack watchesWhat happens
Customer API readsThe lab tracks read speed across sessions, staff logins, and IP addresses. A read rate above 600 rows per minute or repeated hits on distinct canaries can lock the staff login's active sessions. The request that triggers the lock can still return data.
Canary identitiesA canary is a planted customer record. A canary portal login or simulated contact raises a critical alert. An export-specific canary can identify which copy the record came from.
Decoy filesA local sentinel reports reads of bait files, such as the apparent full customer export. A decoy read raises a critical alert and triggers a lock of eligible customer files across the configured drive.
Honey-locked copiesThe correct password for a .dye file reveals the original rows. Wrong passwords produce plausible synthetic rows. A plausible result from open does not verify a password.

Start the lab

You need Git, Python 3.10 or newer, and a browser. The lab uses Python's standard library. Use the generated data and a dedicated lab drive folder. The setup writes files into that folder and the drills can temporarily replace its customer files with .dye copies.

1. Get the source and configure a lock password. If you already have the repository, open a terminal in its root and start with the .env step.

git clone https://github.com/Durovo51/Data-Dye-Pack.git && cd Data-Dye-Pack
([ -f .env ] || cp .env.example .env) && chmod 600 .env && nano .env

Set DYEPACK_LOCK_PASSWORD in .env to a lab-only password you can retrieve later. In Nano, use Ctrl+O, Enter, then Ctrl+X. The drive builder uses the password for the nightly backup, and Dyepack uses it again to lock and restore customer files.

2. Build the data and drive. Run this from the repository root. The default drive is ~/LakeviewFiber. If that folder already contains files you care about, set DYEPACK_DRIVE to a dedicated path first. See Paths and ports.

python3 -m dyepack lab &&
python3 -m dyepack seed &&
python3 -m dyepack drive

At the defaults, lab creates 50,000 fictional customers, seed plants 200 table canaries, and drive writes the sample customer files, nightly backup, and four bait files. Additional terminals should start from the same repository root.

3. Start the server and open the dashboard. Leave this terminal running.

python3 -m dyepack serve --open

The server binds to 127.0.0.1:8750. The --open flag opens a browser tab with the control token so you can take incident actions and change alert routes. A tab opened at the plain address starts in read-only mode unless it already holds the current token. The server creates a new token on each start.

See a complete incident

Recorded demo. On Overview, select Watch the recorded drill. Switch to Alerts as incidents appear. The playback bar lets you pause, restart, or switch from 1× to 4×. Response actions and routing changes in this mode affect only the replay. Select End replay to return to live mode on the local server. A static deployment of the dashboard plays the recording automatically.

Live demo without watcher setup. Keep the server open and run the following in a second terminal from the repository root.

python3 -m dyepack drill agent --simulate-sentinel

Open Alerts and select the decoy incident. Look for the decoy path, Locked files, and the lock events under Activity. For example, customers/customers.csv becomes customers/customers.csv.dye. The simulated watcher reports the scripted reader's file activity after the reader finishes. Use a real sentinel to observe detection while files are being read. Both modes can lock the actual files in the lab folder.

Restore the files when you finish:

python3 -m dyepack unlock

Watch actual file reads

Start the sentinel in a second terminal while the server runs. Keep it running during file drills.

macOS 13 or later. Give your terminal app Full Disk Access under System Settings → Privacy & Security. Start the watcher with elevated privileges. Apple's Privacy & Security settings guide

sudo python3 -m dyepack sentinel

Linux. Run the watcher as your normal user. The inotify backend sees file activity, but it cannot identify the process or user that opened a file. Linux inotify manual

python3 -m dyepack sentinel

With the sentinel running, use a third terminal to run the built-in reader. Or open one known bait file to test that specific trigger.

python3 -m dyepack drill agent
head -n 1 "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers/all_customers_full_export_2026.csv"

The bait file remains in place. Eligible customer-shaped CSV and JSON files up to 50 MiB can be locked. Ordinary notes, bait files, existing backups, and files outside the lab drive are outside the lock's scope. A file-speed alert alone does not trigger the drive lock. Stop the sentinel or server with Ctrl+C in its own terminal.

Run the other drills

Run these against the running local server. The normal and heist drills exercise the API and work without the file watcher. With notification credentials configured, a live drill can send real on-call alerts.

Command after python3 -m dyepackWhat to look for
drill normalSimulated support lookups for 60 seconds. On a fresh lab, expect zero alarms. Use --seconds 15 for a shorter run.
drill heistThree stages. A stolen staff login reads in bulk, a second login downloads a honey-locked export and guesses passwords, then simulated portal and contact attempts trigger alerts that can name the export.
drill heist --step 1Run the bulk-read stage on its own. Check how many plaintext rows were returned before containment.
drill heist --step 2Run the export and password-guess stage on its own. The optional Databricks model tries to identify the real list. The final stage requires data from the same run, so use the full heist to see stage 3.
drill agentA scripted reader opens files on the company drive. Run it with the sentinel, or add --simulate-sentinel to test without it.
drill agent --real --cmd 'YOUR_AGENT YOUR_PROMPT_FLAG "{task}"'Replace the command and flag with those for an installed agent. Dyepack runs it inside the lab drive and substitutes the task find and summarize the customer data here into {task}. Keep the sentinel running. The agent runs with your user's privileges.

Each drill prints a JSON score in the terminal. The dashboard shows available results under Alerts → More options → Lab tools and settings → Lab diagnostics → Drill results.

Investigate and respond

  1. In Overview, use Read speed to review the last 30 seconds or 3 minutes. Alert severity and Time to containment summarize observed incidents.
  2. In Alerts, select an incident to see what triggered it, when it happened, and the estimated exposure. Expand Activity for the event timeline. Expand Report when it appears. High and critical incidents get an automatic report from a local template or, if configured and available, a Databricks model.
  3. Use Actions if you have the control token. The menu shows only actions that apply to the selected incident.
ActionEffect in live mode
Lock accessBlocks the associated staff sessions or portal addresses. Automatic drive locking is triggered by a local decoy read.
Revoke accessBlocks affected staff logins and their active sessions, or blocks future downloads of the associated export. A copy already downloaded remains with its recipient.
Prepare breach noticeAdds an exposure estimate and a lab checklist to the incident. It includes plaintext rows and identifies copies recorded as locked. It sends no notices.
Dismiss incidentMarks it as a false alarm and releases its session or address locks. Use unlock separately to restore files. A revocation remains in effect for the current server run.

The Lab diagnostics panel also shows customer table activity, observed file reads, and Export fingerprints. A matched export canary helps trace a copied record back to the export. An exposure count reflects activity observed by the lab and can include repeated reads.

Export, lock, and recover files

Make a standalone honey-locked copy of the lab table. Run these from the repository root. lock asks for and confirms a password for this copy. open asks for that password and previews five rows by default. To see how the demo works, run open again and enter a different password.

python3 -m dyepack lock demo.dye &&
python3 -m dyepack open demo.dye --limit 5

Lock a folder manually. This locks eligible files directly inside the chosen drive folder and leaves its subfolders for separate runs.

python3 -m dyepack burst "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers"

Recover locked drive files. The first command restores everything tracked as locked. Use the second command instead to restore just the named file.

python3 -m dyepack unlock
python3 -m dyepack unlock "${DYEPACK_DRIVE:-$HOME/LakeviewFiber}/customers/customers.csv.dye"

unlock verifies a tracked file against the watch list, restores its original contents, and removes the .dye file after an in-place restore. It refuses to replace an existing destination. You can use --to PATH to recover one file to a new location and keep the .dye copy. --force is for a file made by burst whose watch-list record is missing. It cannot verify the recovered contents in that case and keeps the .dye copy. Use open for standalone exports and the demo nightly backup.

Reset the demo

  • To clear live incidents, sessions, and revocations, stop and restart serve. Reopen with --open for a fresh control token. Files on disk and saved alert routes persist.
  • To restore the exact files from a drill, use unlock before rebuilding anything. To recreate the sample drive, stop the watcher and server, run drive, then start both again.
  • To start with a new database, run lab, seed, and drive with the server and watcher stopped. lab replaces the database, watch list, and generated API exports. Save any lab artifacts you need first.

Notifications and reports

Open Alerts → More options → Lab tools and settings → Alert routing. Choose the channels for each severity. The defaults are push and SMS for critical incidents, push for high, and dashboard only for low. Notification history shows the most recent delivery results.

ChannelWhat to configure in .env
Phone pushRun python3 -m dyepack.notify topic. Paste the generated value into NTFY_TOPIC, then subscribe to it in the ntfy app. Keep the topic private because anyone who knows it can read the alerts.
SMSSet TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_FROM, and ALERT_SMS_TO.
DiscordSet DISCORD_WEBHOOK_URL for the target channel. Enable Discord on the severities you want.
EmailThe lab records a simulated delivery.

Without credentials for a channel, its deliveries are simulated. To test the provider settings directly, run the command below. It attempts each channel and can send a real test notification to every configured destination.

python3 -m dyepack.notify test

Databricks is optional. Set DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_WAREHOUSE_ID, and DATABRICKS_MODEL in .env. The defaults for DATABRICKS_CATALOG and DATABRICKS_SCHEMA are workspace and dyepack. Use a token with access to the SQL warehouse, model endpoint, and target schema. The service writes alarm, report, and export events to Databricks tables and can generate incident reports with the model. The notebooks in notebooks/ cover seeding a Databricks dataset and querying alarms. Verify both connections, then restart the server.

python3 -m dyepack.databricks

Paths and ports

  • Drive and lab data. The defaults are ~/LakeviewFiber for the drive and data/ inside the repository for lab data. To override either, use an absolute path without .. components and export it in every terminal that runs Dyepack. Keep watch_list.json, lab_logins.json, and control_token private.
  • Port. Use serve --port 8760 --open to change ports, then add --url http://127.0.0.1:8760 to the drill and sentinel commands.
  • Custom paths on macOS. Use sudo python3 -m dyepack sentinel --root "$DYEPACK_DRIVE" when using a custom drive. If you also set DYEPACK_DATA, pass it as an argument to env after sudo so the watcher can find the token.
  • Headless Linux. Start serve on the server and run the SSH forward below on your laptop. Open http://localhost:8750/ there. To enable controls, read data/control_token on the server, or control_token under your custom data directory. Open http://localhost:8750/?token=YOUR_TOKEN in the dashboard tab. The page removes the token from the address bar and retains it for that tab.
ssh -N -L 8750:127.0.0.1:8750 YOUR_USER@YOUR_SERVER

For example, run this before setup and in each additional terminal to keep the lab drive in its own folder:

export DYEPACK_DRIVE="$HOME/LakeviewFiber-Demo"

Command reference

Run python3 -m dyepack --help or append --help to a specific command. --version prints the installed version. To use the shorter dyepack command, install from the repository with python3 -m pip install -e . in a virtual environment.

CommandOptions and defaults
lab--customers 50000 sets the number of fictional customers. --seed N makes the customer generation repeatable.
seed--every 250 sets the canary frequency. An already seeded lab refuses a second planting.
drive--seed N controls the sample of customer files generated for the drive.
serve--open opens the control-enabled dashboard. --port 8750 sets the listener port.
sentinel--root PATH can be repeated. --backend auto picks eslogger on macOS or inotify on Linux. --backend fake --source FILE reads JSON Lines events, with standard input used when --source is omitted. --quiet hides per-event terminal output.
drillChoose normal, heist, or agent. --url selects the local server address. See the drill table for mode-specific options.
lock OUT and open FILEBoth accept --password for automation. The interactive prompt keeps a password out of shell history. open also accepts --limit N.
proveRuns a statistical comparison of original and synthetic lists. Defaults to --rows 2000 --fakes 99. Use --compare to compare three different ways of constructing the file's distribution header.
burst FOLDERLocks eligible customer files directly in that lab drive folder.
unlock or restoreRestore all tracked files or pass one or more .dye paths. Use --to PATH for a single output and --force only when the watch record is missing.

Lab HTTP API

The API runs on loopback alongside the dashboard. Send JSON for POST requests. Use the session returned by staff login as the X-Session header on customer, export, and permit requests. With the server running on the default port, paste the following from the repository root to sign in as rep.taylor and read one sample row. The script reads the generated password from lab_logins.json without printing it.

python3 - <<'PY'
import json
import os
from pathlib import Path
from urllib.request import Request, urlopen

base = "http://127.0.0.1:8750"
logins = Path(os.environ.get("DYEPACK_DATA", "data")) / "lab_logins.json"
user = "rep.taylor"
credentials = {"user": user, "password": json.loads(logins.read_text())[user]}
login = Request(base + "/api/login", data=json.dumps(credentials).encode(),
                headers={"Content-Type": "application/json"})
with urlopen(login, timeout=10) as response:
    session = json.load(response)["session"]
lookup = Request(base + "/api/customers?page=0&size=1",
                 headers={"X-Session": session})
with urlopen(lookup, timeout=10) as response:
    print(json.dumps(json.load(response), indent=2))
PY
RequestHow to use it
POST /api/loginSend user and password to obtain a staff session.
GET /api/customersList with page and size, or search with q and limit. Page numbers start at zero. A support page can return up to 100 rows. The backup.nightly job account sees the clean table and can page up to 1,000 rows.
GET /api/customers/ACCOUNT_NUMBERRead one customer by account number.
POST /api/exports and GET /api/exports/EXPORT_IDPOST an export password, then use the returned export_id in the download path. Each file has its own export canaries.
POST /api/permitsA different staff login submits who, optionally table (default customers) and minutes (default 30). The permitted user's reads are still logged. The permit suppresses the usual read alarms during that window.
POST /portal/login and POST /lab/contactSimulate a portal login with user and password, or a contact attempt with to and optional channel. These are API endpoints. The contact route simulates a signal without delivering an email or placing a call.
GET /eventsRead the server-sent events used by the live dashboard. POST /lab/drill is used by the built-in drills to post phase and score updates.
GET /api/alert-routes and POST /api/alert-routesRead or replace the route mapping. POST the complete mapping with severity keys low, high, and critical.
POST /api/respondSend incident and action. Actions are lock, revoke, notify_customers, and dismiss.
POST /lab/fs-eventThe local sentinel's input. Accepts a file event or a batch. The sent events include a path and operation, plus available process details.
POST /api/eventsOptional auditd bridge. Send a decoy event with file, server_id, and available process details. An event_id or Idempotency-Key can deduplicate retries. Bridge alerts do not lock the local drive.

Writing alert routes, posting responses, and posting local sentinel events require the current X-Dyepack-Token. A locked staff session returns HTTP 423 on protected reads. A revoked export returns HTTP 410 to an otherwise active session if the file still exists.

Troubleshooting

SymptomWhat to check
No alerts after opening a decoyMake sure the sentinel watches the drive and you read the file contents. For a custom drive path, remove any .. components. Restart the server after running drive. On macOS, check Full Disk Access.
Decoy alert with no locked filesCheck DYEPACK_LOCK_PASSWORD, the server terminal, and whether files are already locked. A bridge event is alert-only.
Response controls are missingOpen a tab with serve --open or the current token. Check that the dashboard is live and select an incident with an available action.
Unlock reports that the original file already existsMove the existing file aside or recover one file to a separate path with --to.
A repeat heist cannot log inA previous Revoke access action may still apply. Restart serve and reopen the dashboard.
The report says TemplateLocal reporting is working. Run the Databricks connection check if you expected model-generated text.

Source version used for this guide is main at commit 44bd338.