# C/C++ Web Archaeology — instructions for AI agents You are adding entries to the C/C++ Web Archaeology catalog: a hand-picked list of **2000–2010 programming material that teaches the reader to build something**, still live on its original website. Your job is to find new entries, prove each one, and propose them in a pull request (PR) that the owner reviews. Read this whole file before you start. It is complete: you need nothing else from the owner. If anything in the repository disagrees with this file, this file wins. - Live site: https://paoloanzn.github.io/cpp-web-archaeology/ - Repository: https://github.com/paoloanzn/paoloanzn.github.io (branch `main`) - Catalog data: `cpp-web-archaeology/data/projects.json` (a JSON list of entries) **Done means:** a PR against `main` that adds new entries which all pass the inclusion test (section 1) and the hard requirements (section 2), with a filled-in PR description (section 8). You never merge it. The owner does. ## 1. The inclusion test — read this twice An entry belongs in the catalog only if **the page teaches the reader to build something themselves**. That is the whole point of the catalog: someone opens the link today and can follow it to build a shell, a kernel, a compiler pass, an emulator, as a student or hobbyist did back then. Apply these three questions to every candidate. All three must be "yes": 1. **What does the reader build?** You can name it in one short phrase ("a user-level thread library", "an x86 code generator for expressions"). If the honest answer is "nothing, they read about it", reject. 2. **Does this page guide the building?** It is either - an **assignment, lab or project spec** (requirements, starter code, steps, deliverables, due dates), or a course page that indexes such projects; or - a **tutorial or build-along** that goes step by step from little or nothing to a working result, explains *why* each step is done, and gives code to write, run and change. 3. **Is the reader the one writing the code — and does that code implement the mechanism the page is about?** In C or C++ (or, for the Lisp wing, a Lisp dialect — see section 2). Writing plug-ins, callbacks or scripts for an existing framework whose core mechanism stays hidden from the reader does not count (see "Tool exercises" below). Reject these even when they are old, live, technical and well written: - **Reference docs and manuals of existing software** (API manuals, user guides, programmer's guides). They describe a tool; they do not teach building. - **Project pages, source releases and design docs of a finished system.** - **A developer's blog posts about their own project** (progress reports, "how my code works", bug hunts, opinions). - **Benchmarks and comparisons.** - **Research papers, slide decks, talk videos.** - **Course pages with no buildable projects** (syllabus only, reading list, slides). These are real verdicts from the owner. Calibrate on them. Utah CS 5460 Fall 2009 HW1 — generate x86 machine code at run time https://users.cs.utah.edu/~mflatt/past-courses/cs5460/hw1.html Peter Kankowski — Code generator: an x86 expression compiler in C++ https://strchr.com/code_generator DotGNU libjit tutorials — https://www.gnu.org/software/dotgnu/libjit-doc/libjit_3.html Graham Toal — Emulator Writer's HOWTO for Static Binary Translation — http://www.gtoal.com/sbt/ Darek Mihocka — NO EXECUTE! Part 25: Predicting Interpreters — http://www.emulators.com/docs/nx25_nostradamus.htm DynamoRIO 0.9.4 API manual; Intel Pin 2.0 User Guide; Dyninst Programmer's Guide Strongtalk downloads page; YARV homepage; Parrot JIT subsystem doc Factor "polymorphic inline caching" post; Daedalus "Dynarec Improvements" post "Simple test: libjit vs LLVM" (Habr, 2010) Borderline cases, decided: - **Hands-on technique articles count.** A page where the reader writes and runs small programs to see a mechanism work (run generated machine code from a buffer, watch how the stack or the NX bit behaves, measure a cache) teaches building, even if no big program results. Size does not matter; the reader must write and run the code. - **Tool exercises do not count.** Workbooks, labs or hand-outs whose exercises are "write a tool *for* framework X" (instrumentation plug-ins, analysis callbacks, mutator scripts) teach the framework, not the mechanism: reject. Contrast: using a code-generation library to build your own compiler or JIT (the reader designs the translation) → accept. - **Blog posts:** ask *who builds*. If the post walks the reader through writing the code step by step (they could follow along and end with a working result), it is a build-along → accept. If it reports what the author built or changed in their own project, it is a project blog → reject. - **Side tools are fine.** An assignment that uses a framework only as a helper (e.g. a tracer that produces input files) while the reader writes the main mechanism → accept. - **Mixed pages** are judged by their main body. A tutorial with a short benchmark at the end still counts; a benchmark with a little code does not. - A book chapter or tutorial series that builds a program (Practical Common Lisp, Bran's kernel tutorial, NeHe lessons, Lex & YACC HOWTO) → accept. - A HOWTO that walks you through writing code (GCC Frontend HOWTO, Linux Kernel Module Programming Guide) → accept. One that is mainly reference → reject. - A library or project homepage → reject, unless the page itself walks through how the code is built (Picol's "550 lines explained" article style). - A short teaching note on one technique with code to try → accept only if you can answer question 1 with something real the reader writes. ## 2. Hard requirements Each one has a reason. Use the reason to decide edge cases. | Requirement | Rule | Why | |---|---|---| | Period | The material is from 2000–2010, proven **on the page** (a date, a semester, a year in the URL path — short codes like `fa07`, `s04`, `2003sp` count — or a period toolchain: VC6 `.dsw`, gcc 2.95/3.x, CVS, Fedora Core, Bochs, Nachos, Pintos, SDL 1.2, DrScheme...). Quote the strongest proof verbatim; a URL code may be the quote. A modern copyright footer or a search snippet is not proof. | The catalog is archaeology: the reader should see what was there then. | | Live | The URL loads today on its original or long-running site. Never `web.archive.org` or other mirrors. No GitHub/GitLab-only sources, Medium, Dev.to, modern reposts. | The catalog is about what survived on the live web. | | Language | The reader writes C or C++. Lisp wing: Scheme, Common Lisp, Emacs Lisp, Clojure, AutoLISP or Racket, with a `Lisp / ...` category. A course where students write Java/ML/Python is out, even if the target is C. | Scope of the catalog. | | New | Not already in the catalog; at most 2 semesters of one course; at most 2 entries per site in one PR. | Breadth beats ten copies of one course. | | Safe | No exploit or attack labs (shellcode, buffer-overflow exploitation, malware). Defensive and crypto-implementation work is fine. | Owner's policy. | When the inclusion test or a requirement is uncertain after you have looked at the page, reject the candidate and add it to `leads.md` with a one-line note. A missed entry costs nothing; a bad entry costs the owner a review. ## 3. Your limits - Change **only** `cpp-web-archaeology/data/projects.json` and `.claude/skills/cpp-web-archaeology/references/leads.md`. Never edit the website, scripts, workflows, or anything else: the repository is also the owner's blog. - **Never push to `main` and never merge.** Work on a branch, open a PR, stop. - Only append entries. Never edit, reorder or delete existing entries. If an existing entry looks dead or fails the inclusion test, list it in the PR instead. - Keep scratch files outside the repository (for example `/tmp/cwa`). - If your environment cannot reach the live web, say so at the start and stop: you cannot verify anything. Never claim a check passed when it did not run. - If you cannot push to the repository (a fork you can push to counts as access): finish steps 1–6 with `append_catalog.py --dry-run`, then put `merged.json` and your `leads.md` additions in your final answer. ## 4. Workflow Copy this checklist into your notes and tick each step. Do not skip the review step: it is where most bad entries are caught. ``` - [ ] 1. Set up and read the gaps - [ ] 2. Research candidates (apply section 1 to each, write batch files) - [ ] 3. Merge, then review every entry with the inclusion test - [ ] 4. Verify every URL is live - [ ] 5. Append and validate (repeat until validation passes) - [ ] 6. Update leads.md - [ ] 7. Commit, push the branch, open the PR ``` ### Step 1 — Set up ```bash git clone https://github.com/paoloanzn/paoloanzn.github.io && cd paoloanzn.github.io git checkout -b catalog/$(date +%Y-%m-%d)- REPO=$(pwd); SKILL=$REPO/.claude/skills/cpp-web-archaeology; WORK=/tmp/cwa; mkdir -p $WORK/batch python3 $SKILL/scripts/catalog_stats.py --urls-out $WORK/existing_urls.txt python3 $SKILL/scripts/catalog_stats.py --categories ``` Shell variables are often lost between commands in agent sandboxes: repeat the `REPO/SKILL/WORK` line at the top of each command. The scripts are Python 3, standard library only; each has `--help`. Read `$SKILL/references/leads.md` now: it lists open leads (check them first) and dead ends (do not spend time on them). If the owner gave you a focus (for example "compilers"), stay on it. Otherwise pick the smallest areas and thinnest years from the stats. ### Step 2 — Research - If you can run parallel sub-agents, give each one a separate domain, this file, and its own output file in `$WORK/batch/`. Otherwise work domain by domain. - Search with period fingerprints, not generic phrases: `site:edu "project 3" "Fall 2004" "Makefile"`, `site:edu inurl:fa05 "you will implement"`, `"Visual C++ 6.0" ".dsw" assignment`, `"starter code" "due" filesystem C 2003`. - **Follow directories.** When a page is strong, open its course index, sibling projects, sibling semesters and the instructor's other course archives. This finds more than searching. Hosts with live year-coded archives include `cs.princeton.edu/courses/archive`, `cs.cornell.edu/courses`, `pages.cs.wisc.edu`, `classes.cs.uchicago.edu/archive`, `cseweb.ucsd.edu/classes`, `cs.cmu.edu`. Many are already well covered: prefer new institutions and new sites (`grep -i HOST $WORK/existing_urls.txt`). - Open every candidate yourself. Read enough of it to answer the three questions of section 1 and to find the date proof. Never accept from a search snippet. - Web search budgets are limited. When yours runs low, follow directories with curl. - Write one batch file per domain: ```json {"accepted": [ ], "rejected": [ {"url": "...", "reason": "short reason"} ], "borderline_leads": [ {"url": "...", "note": "why it is worth a later look"} ], "investigated_count": 0} ``` ### Step 3 — Merge and review ```bash python3 $SKILL/scripts/merge_batch.py --batch-dir $WORK/batch --out $WORK/merged.json --report $WORK/report.json ``` The script checks format, duplicates and the date quote. It cannot judge teaching value: you must. Read every merged entry and ask the three questions again, as if you were the owner. Remove failures by re-running the command with `--drop URL [URL ...]`. Also drop near-duplicates and any entry whose date proof is weak. Then re-open at least 5 pages and confirm the quoted date proof is there: `curl -sSL URL | grep -i -o -m3 -E "due[^<]{0,30}|(fall|spring|winter) 20[01][0-9]"`. Finish all `--drop` runs before step 4 (re-merging overwrites the step 4 fixes). ### Step 4 — Verify live ```bash python3 $SKILL/scripts/check_urls.py $WORK/merged.json --fix ``` `--fix` swaps http/https when only the other works. `REDIRECT` means the page now lands on a login wall, a front page or another site: drop it. Drop anything that still fails, with one exception: some servers send an incomplete TLS chain or block scripts, so curl fails while a browser works. Never disable TLS verification. Check those pages with a browser tool or WebFetch, keep them only if they load with their date proof, and list them in the PR. ### Step 5 — Append and validate ```bash python3 $SKILL/scripts/append_catalog.py $WORK/merged.json python3 $SKILL/scripts/validate_catalog.py # must print 0 errors python3 $SKILL/scripts/catalog_stats.py # read "Records matching NO intent tile" ``` If validation reports errors, fix the entries in `merged.json`, reset the catalog with `git checkout cpp-web-archaeology/data/projects.json`, and append again. **Only go on when validation passes.** If a new entry matches no website tile, improve its tags and notes with the words a reader would search for; do not edit the website. ### Step 6 — Update leads.md Remove leads you added or proved dead. Add new leads from `report.json` (`leads`) under the matching domain heading, and new dead ends under "Known dead ends". Format: `- URL — one-line note`. ### Step 7 — Commit, push, open the PR ```bash git add cpp-web-archaeology/data/projects.json .claude/skills/cpp-web-archaeology/references/leads.md git status --short # only these two files may be listed git commit -m "Add N curated entries" git fetch origin main && git rebase origin/main && git push -u origin HEAD ``` Open a PR against `main` with the description from section 8, then stop. If the rebase conflicts on `projects.json` (a weekly bot re-sorts it), keep `main`'s version and append again. During a rebase, "ours" is `origin/main`: ```bash git checkout --ours cpp-web-archaeology/data/projects.json python3 $SKILL/scripts/append_catalog.py $WORK/merged.json && python3 $SKILL/scripts/validate_catalog.py git add cpp-web-archaeology/data/projects.json && GIT_EDITOR=true git rebase --continue ``` ## 5. Entry format Each new entry has exactly these 12 fields. No others. ```json { "title": "Utah CS 5460 (Fall 2009) HW1 — Generating x86 machine code at run time", "url": "https://users.cs.utah.edu/~mflatt/past-courses/cs5460/hw1.html", "year": 2009, "year_label": "Fall 2009", "language": ["C"], "category": "Compilers / JIT", "kind": "project assignment", "site": "users.cs.utah.edu", "score": 94, "evidence": [ "\"Due: Tuesday, September 8th, 2009\"", "Students write C that emits x86 machine code into an mprotect-ed buffer", "Starter code and test harness provided" ], "tags": ["jit", "x86", "code generation", "mprotect", "assignment"], "notes": "Write a C function that generates x86 machine code at run time, makes the buffer executable and calls it." } ``` | Field | Rule | |---|---| | `title` | "Institution COURSE (Term) — What you build" or "Author — Article title". | | `url` | The exact live URL you loaded. https if it works. No tracking parameters, no `#fragment`. | | `year`, `year_label` | Integer 2000–2010; the most precise date the page shows. | | `language` | List with `"C"` and/or `"C++"`, or a Lisp dialect (`"Scheme"`, `"Common Lisp"`, `"Emacs Lisp"`, `"Clojure"`, `"AutoLISP"`, `"Racket"`). | | `category` | Reuse one from `catalog_stats.py --categories`. Lisp-only entries must start with `Lisp / `. Create a new category only if it will hold several entries. | | `kind` | Exactly one of: `project assignment`, `lab assignment`, `project sequence`, `course project sequence`, `lab sequence`, `tutorial`, `tutorial series`, `project tutorial`, `build-along article`, `book chapter`. | | `site` | The URL's host without `www.`. Must match the URL. | | `score` | Section 6. | | `evidence` | 3–6 facts you saw on the page. The first is a **verbatim quote of the date proof in double quotes**. One should say what the reader builds or is given (starter code, steps). | | `tags` | 3–7 words a reader would search for, including the project type ("kernel", "compiler", "filesystem", "B+ tree", "jit"). The site's "What do you want to build?" tiles match on title, category, notes and tags. | | `notes` | One or two plain sentences starting with a verb: what the reader builds. | ## 6. Score The score is your confidence that the entry fits the catalog: period proof and teaching value together. | Score | Meaning | |---|---| | 98–100 | Original dated assignment or build-along with a complete path from start to working result. | | 95–97 | Very strong date proof and a clearly buildable project. | | 90–94 | Good, with slightly weaker date proof or a smaller project. | | 85–89 | Add only with a reason you state in the PR. | | below 85 | Never add. | Caps: proof is only a period toolchain (no date or semester) → at most 94. Date comes only from a linked file or directory listing → at most 92. Score honestly; never raise a score to avoid explaining it. ## 7. When things go wrong - **A host resets curl or returns 403/503 but loads in a browser:** use a browser tool or WebFetch, and write "checked with WebFetch" in the evidence. - **The page is a course index whose project pages are dead:** reject; the reader cannot build from it. - **Two candidates have the same text on two hosts:** keep the original host only. - **You find good material from 2011 or later:** out of range. Do not add it. - **You are unsure whether something teaches building:** it goes to `leads.md`. ## 8. PR description Title: `Add N curated entries` ``` Focus / domains researched: Pages investigated: Accepted: Rejected: For each accepted entry, one line: title — what the reader builds — URL Entries scored 85–89 and why: Pages checked with WebFetch/browser instead of curl: New categories (if any): Entries matching no website tile (if any): Leads added / removed: Notable dead ends: Existing entries that look dead or fail the inclusion test: Agent / model: ``` ## 9. Final check before opening the PR - [ ] Every new entry passes the three questions of section 1. You could tell the owner, for each one, what the reader builds. - [ ] Every entry has a verbatim date quote; you re-opened at least 5 pages. - [ ] No reference manuals, project pages, own-project blog posts, benchmarks, papers or slides. - [ ] No `web.archive.org`, no GitHub-only sources, no exploit labs, no duplicates. - [ ] `validate_catalog.py` prints 0 errors. - [ ] Only `projects.json` and `leads.md` changed. - [ ] The work is on a branch with an open PR. You did not push to `main` or merge.