GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add three courses: GitHub Copilot, LLM features in Python, security review of AI-written code by xxg1413 · Pull Request #107 · flypythoncom/python

Add three courses: GitHub Copilot, LLM features in Python, security review of AI-written code #107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cursor merged 6 commits into master from cursor/three-new-courses-21e9
Oct 11, 2026

Uh oh!

There was an error while loading. Please reload this page.
Merged

Add three courses: GitHub Copilot, LLM features in Python, security review of AI-written code #107

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a8d24ac
courses: add Hands-on with GitHub Copilot (report-tool core group)
cursoragent Oct 11, 2026
272f863
courses: add Building LLM features in Python (validate, allowlist too…
cursoragent Oct 11, 2026
42638c7
courses: add Security review of AI-written Python code (five flaw cla…
cursoragent Oct 11, 2026
75f2f8b
courses: state the real verify.py exit rule in the Copilot course
cursoragent Oct 11, 2026
784ffc7
courses: combine nested with-statements in the new course tests (ruff…
cursoragent Oct 11, 2026
639126d
index: list the three new courses in llms.txt, READMEs, the foundatio…
cursoragent Oct 11, 2026
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view
Prev Previous commit
Next Next commit
courses: add Building LLM features in Python (validate, allowlist too… 
…ls, retry, honest eval)

Co-authored-by: P小二 <[email protected]>
  • Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

@cursoragent @xxg1413
cursoragent and xxg1413 committed Oct 11, 2026
commit 272f863859d40aff2024e104ee74be7976fc6622

There are no files selected for viewing

117 changes: 117 additions & 0 deletions 117 courses/llm-features-in-python/COURSE.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
---
id: course-llm-features
type: course
title: "Build LLM features in Python: structured output, tool calls, and evals"
summary: Turn a demo-day model call into a feature you can ship — validate structured output, allowlist tool calls, retry with the error fed back, fall back to a human instead of guessing, and measure it with an offline eval that divides by every case.
lang: en-US
content_version: 1
status: reviewed
reviewed_on: 2026-10-11
badge:
id: course-llm-features
name_en: Model output is untrusted input
name_zh: 模型输出是不可信输入
requires: All five checkpoints claimed (L01–L05)
course_id: course-llm-features
---

# Build LLM features in Python: structured output, tool calls, and evals

> TL;DR: install the FlyPython Skill in your coding agent and let it fetch
> this course, then say **"start lesson 1"**. (Lesson files arrive via the
> Skill — you download nothing by hand.) You finish with `ticket_triage.py`:
> a support-ticket triage feature that treats every model reply as
> untrusted input, plus an offline eval with a pass/fail threshold. No API
> key, no network, no third-party packages — a replay client stands in for
> the model.

## What you build

A triage feature for support tickets: the model reads a ticket and replies
with a category, a priority, a one-line summary, and whether a human must
look at it. It may call a lookup tool first. Three scenario "skins" ship
with the course, each with labelled cases and recorded model replies:

| Skin | You are… | Tool the model may call |
| --- | --- | --- |
| `scenario/shop-support/` | routing an online shop's support inbox | `lookup_order(order_id)` |
| `scenario/it-helpdesk/` | sorting internal IT requests | `lookup_device(asset_tag)` |
| `scenario/saas-feedback/` | classifying in-app product feedback | none |

The recorded replies are hand-written fixtures that imitate failure shapes
models really produce — JSON in a Markdown fence, a category that is not on
the list, `"priority": "high"`, an extra field, a tool that does not exist,
a tool argument of the wrong type, chatty prose instead of JSON. They are
**not** captured from any model, and the course makes no claim about how
often any real model produces them.

## Teaching contract (read this first, agent)

- **Audience:** a Python developer who has called a model API in a script
or notebook and now has to put that call inside a product — where a
malformed reply, a hallucinated tool call, or an eval that looks better
than it is becomes somebody's incident. Not a prompt-engineering course
and not an introduction to Python.
- **Prerequisites:** Python 3.11+ on PATH and any coding agent that can run
commands. Standard library only; no API key and no network access are
needed for the exercise.
- **Tool:** tool-agnostic. The course has not yet been taught in a tool;
`REVIEW.md` records the mechanical verification and the pending
agent-taught run-through, which will name the exact tool and version.
- **Lesson order:** L01 → L02 → L03 → L04 → L05, one lesson per session.
Never skip the checkpoint at the end of a lesson.
- **Teaching style:** work from the files in this folder, not from general
knowledge about any provider's API. Quote the `TASK.md` line you are
satisfying. Make the smallest change that moves a test from red to
green. Never add dependencies, never edit `solution/`, `tests/`, or
`scenario/`, and ask before touching any file not named in the current
lesson. Never weaken validation to make a recorded reply pass.
- **When to stop:** a lesson is done when its checkpoint command runs and
the learner can say, in their own words, what failed and why. If the
learner cannot, re-teach from the failing test, not from the solution.
- **How to use `verify.py`:** the learner's command is plain
`python verify.py` — it runs the suite against `starter/`, prints
per-checkpoint status and earned claim codes, and exits non-zero until
every checkpoint is passed (l03, l04) or attested (l01, l02, l05). `starter --expect-failure`
and `solution` remain maintainer checks.
- **Honesty rules:** an offline eval over a handful of recorded replies
says nothing about how a live model will behave on live traffic — say so.
Do not claim the feature is production-ready. Do not promise learning or
career outcomes.

## What this course does NOT cover

Any specific provider SDK or its structured-output mode, API keys and
secrets handling, prompt wording, retrieval (RAG) and embeddings,
fine-tuning, agent frameworks, streaming, cost and latency budgets, or
online evaluation on production traffic. Lesson 5 shows where a real
client plugs in; the companion repository has guides for the rest — link,
don't improvise.

## Badge contract

- Badge: **Model output is untrusted input Badge** (badge id `course-llm-features`) — earned by claiming all five checkpoints.
- Challenges: L01–L05 checkpoints, 10 points each; +50 course-badge bonus when all five are claimed on flypython.com.
- Evidence: `python verify.py` — L03 (validation, tool calls, retries) and L04 (honest eval) are objectively gated by the suite; L01/L02/L05 are learner-attested.
- Submission: each test-passed checkpoint prints a deterministic claim code; a reflection checkpoint prints one only after you answer its questions and run `python verify.py --attest ID`; record it on flypython.com against your account. Self-reported evidence, never a certificate.

## Folder map

```
COURSE.md / COURSE_cn.md this file (EN / 中文)
lessons/L01.md … L05.md lessons (each has an _cn.md pair)
scenario/<skin>/ scenario.json, cases.json, recordings.json, lookup data
TASK.md / TASK_cn.md the task contract the change must satisfy
starter/ticket_triage.py the demo-day implementation
solution/ticket_triage.py the reviewed solution (do not copy in lesson 3)
tests/test_ticket_triage.py the contract suite (15 tests, read-only)
verify.py objective pass/fail evidence
REVIEW.md maintainer run-through record
```

## Evidence and licensing

`REVIEW.md` records the run-through state with dates, tool versions, and
observed deviations. Code in this folder is MIT-licensed; lesson prose is
CC BY 4.0 (see repository `LICENSE`). Report teaching drift or unclear
lessons via the repository's `course-feedback` issue form.
103 changes: 103 additions & 0 deletions 103 courses/llm-features-in-python/COURSE_cn.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
id: course-llm-features
type: course
title: "用 Python 构建 LLM 功能：结构化输出、工具调用与评测"
summary: "把演示日的一次模型调用变成能上线的功能——校验结构化输出、给工具调用设白名单、带着错误信息重试、拿不准就转人工而不是瞎猜，并用分母覆盖全部用例的离线评测来度量。"
lang: zh-CN
content_version: 1
status: reviewed
reviewed_on: 2026-10-11
badge:
id: course-llm-features
name_en: Model output is untrusted input
name_zh: 模型输出是不可信输入
requires: 认领全部五个检查点（L01–L05）
course_id: course-llm-features
---

# 用 Python 构建 LLM 功能：结构化输出、工具调用与评测

> 一句话：在你的编码 Agent 里装上 FlyPython Skill，让它取回本课程，然后
> 说**「开始第 1 课」**。（课程文件经 Skill 送达——你不手动下载任何东西。）
> 完成时你会得到 `ticket_triage.py`：一个把每条模型回复都当作不可信输入的
> 工单分诊功能，外加一个带通过/失败阈值的离线评测。不需要 API key、不需要
> 联网、不需要第三方包——由一个回放 client 代替模型。

## 你要做的东西

一个工单分诊功能：模型读一张工单，回复分类、优先级、一句话摘要，以及
是否必须由人来看。它可以先调用一个查询工具。课程自带三个「场景皮肤」，
每个都有带标注的用例和录好的模型回复：

| 皮肤 | 你是…… | 模型可调用的工具 |
| --- | --- | --- |
| `scenario/shop-support/` | 给网店客服收件箱分流 | `lookup_order(order_id)` |
| `scenario/it-helpdesk/` | 整理公司内部 IT 请求 | `lookup_device(asset_tag)` |
| `scenario/saas-feedback/` | 给应用内产品反馈分类 | 无 |

录好的回复是手写的练习数据，模仿模型真实会出现的失败形状——包在
Markdown 围栏里的 JSON、不在列表里的分类、`"priority": "high"`、多出来的
字段、不存在的工具、类型不对的工具参数、用闲聊散文代替 JSON。它们**不是**
从任何模型采集的，课程也不对任何真实模型出现这些问题的频率做任何声明。

## 教学契约（先读，Agent）

- **受众：** 在脚本或 notebook 里调过模型 API、现在要把这个调用放进产品
的 Python 开发者——在产品里，一条格式错误的回复、一次幻觉出来的工具
调用、一个看起来比实际好的评测，都会变成某个人的事故。这不是提示词
工程课，也不是 Python 入门。
- **前置条件：** PATH 上有 Python 3.11+，以及任意一个能执行命令的编码
Agent。只用标准库；练习不需要 API key，也不需要联网。
- **工具：** 与工具无关。本课程尚未在任何工具里实际授课；`REVIEW.md`
记录机械验证结果和待完成的 Agent 授课实跑，实跑时会写明确切的工具与
版本。
- **课序：** L01 → L02 → L03 → L04 → L05，一次课一个会话。
绝不跳过课尾的检查点。
- **教学风格：** 从本文件夹里的文件出发，不要凭任何厂商 API 的通用知识
发挥。引用你正在满足的那条 `TASK.md` 原文。做让测试从红变绿的最小
改动。绝不加依赖，绝不改 `solution/`、`tests/` 或 `scenario/`，动当前
课程未点名的任何文件前先询问。绝不为了让某条录好的回复通过而放松校验。
- **何时停下：** 一节课的检查点命令跑通、且学习者能用自己的话说清
哪里坏了、为什么，这节课才算完。学习者说不出就从失败测试重新教，
不要从 solution 教。
- **如何使用 `verify.py`：** 学员命令就是 `python verify.py`——它针对
`starter/` 跑套件、打印逐检查点状态与已获认领码，直到每个检查点都
通过（l03、l04）或已自报确认（l01、l02、l05）之前都以非零退出。`starter --expect-failure` 与 `solution` 留作
维护者检查。
- **诚实规则：** 在少量录好的回复上跑的离线评测，说明不了真实模型在真实
流量上的表现——要讲明白。不要声称功能达到生产可用。不要承诺学习或
职业结果。

## 本课程不覆盖什么

任何具体厂商的 SDK 或其结构化输出模式、API key 与密钥管理、提示词措辞、
检索增强（RAG）与 embedding、微调、Agent 框架、流式输出、成本与延迟
预算，以及在生产流量上的在线评测。第 5 课展示真实 client 从哪里接入；
其余内容由配套仓库的指南承载——给链接，不即兴发挥。

## 徽章契约

- 徽章：**模型输出是不可信输入徽章**（badge id `course-llm-features`）——认领全部五个检查点获得。
- 挑战：L01–L05 检查点，各 10 分；在 flypython.com 上集齐五个再加 50 分课程徽章奖励。
- 证据：`python verify.py`——L03（校验、工具调用、重试）与 L04（诚实评测）由套件客观把关；L01/L02/L05 为学习者自我报告。
- 提交：测试通过的检查点打印确定性认领码；自报检查点要先回答课后问题，再运行 `python verify.py --attest ID` 才打印码；在 flypython.com 上记入你的账号。这是自我报告的证据，从来不是证书。

## 文件夹地图

```
COURSE.md / COURSE_cn.md 本文件（EN / 中文）
lessons/L01.md … L05.md 课文（每课都有一个 _cn.md 对）
scenario/<skin>/ scenario.json、cases.json、recordings.json、查询数据
TASK.md / TASK_cn.md 改动必须满足的任务契约
starter/ticket_triage.py 演示日版本的实现
solution/ticket_triage.py 评审过的解答（第 3 课不要抄）
tests/test_ticket_triage.py 契约套件（15 个测试，只读）
verify.py 客观的通过/失败证据
REVIEW.md 维护者实跑记录
```

## 证据与许可

`REVIEW.md` 记录实跑状态——日期、工具版本、观察到的偏差。文件夹内代码
采用 MIT 许可；课文文字采用 CC BY 4.0（见仓库 `LICENSE`）。发现教学漂移
或课程不清楚之处，请通过仓库的 `course-feedback` issue 表单反馈。
75 changes: 75 additions & 0 deletions 75 courses/llm-features-in-python/REVIEW.md
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Maintainer run-through record

## 2026-10-11 — contract verification (mechanical)

- Environment: Linux x86_64, Python 3.12.3, standard library only.
- The scenario recordings (`scenario/*/recordings.json`) are hand-written
fixtures that imitate common reply failures (Markdown fences, invented
categories, string priorities, prose, out-of-allowlist tool calls). They
were not captured from any model, and no model API was called while
authoring or verifying this course.
- Commands and results:
- `python verify.py starter --expect-failure` — exit 0; the twelve
expected failure names reproduced; three tests pass on the starter.
- `python verify.py solution` — exit 0; 15/15 tests pass.
- `python verify.py` — exit 1 on the shipped starter; l03/l04 `[open]`,
l01/l02/l05 `[pending]` with no code.
- Solution eval CLI:

| Skin | Output | Exit |
| --- | --- | --- |
| `shop-support` | `accuracy=0.750 correct=6/8 fallbacks=1 min_accuracy=0.800` | 1 |
| `it-helpdesk` | `accuracy=0.833 correct=5/6 fallbacks=1 min_accuracy=0.800` | 0 |
| `saas-feedback` | `accuracy=0.800 correct=4/5 fallbacks=0 min_accuracy=0.800` | 0 |

The starter prints `accuracy=1.000 correct=5/6` on `it-helpdesk`, the
contradiction Lesson 1 is built on.
- Lesson 3's failing-count table (12 → 9 → 7 → 3 → 0) was measured by
swapping the solution's functions into a scratch copy of the starter
one group at a time; `test_tool_result_is_fed_back_to_the_model` turns
red after the parser group, as the lesson says.
- The Lesson 3 exercise one-liner prints the retry feedback message
after the `triage` turn.
- `python tools/verify_courses.py` — contract satisfied.

## Pending before this course can be called taught-in-tool

- One full agent-taught run-through ("start lesson 1" through the Lesson 5
checkpoint) in a named coding agent and version, recording observed
deviations from `COURSE.md` here. The course is tool-agnostic; no
agent-taught session has happened yet.
- Lesson 5 asks learners to apply the pattern to their own model-backed
feature with a real provider. Whether the contract transfers cleanly to a
provider's native structured-output or tool-calling API is unverified
here.

## Deviation log

### 2026-10-11 — agent solvability run (challenge mode, not taught mode)

- Agent: a Cursor Cloud Agent subagent (Claude Opus 5.5), Linux, Python
3.12.3, standard library only.
- Method: worked in a copy of the course with `solution/` and this file
removed; implemented `starter/ticket_triage.py` from `TASK.md` and the
tests only, without access to the reference solution.
- Result: green after one implementation pass (14/14 on the revision it
was given); the eval CLI printed the same three lines as the solution
table above.
- Observed deviations and the changes they triggered:
- `TASK.md` did not define `attempts` in the fallback; the agent inferred
"number of rejected replies" from a test. `TASK.md`/`TASK_cn.md` now
state it.
- `TASK.md` did not say whether a tool call beyond `max_tool_calls` runs,
or whether refused calls count as `tool_calls`, and no test exercised
the budget. The contract now says only calls that ran are counted and
the over-budget call is not run; the new
`test_tool_call_budget_ends_in_the_fallback` enforces it (suite 14 → 15
tests, starter failures 11 → 12, lessons updated). The agent's
implementation passes the new test unchanged.
- `COURSE.md` said plain `python verify.py` exits non-zero "while l03 and
l04 are open"; it actually exits non-zero until l01/l02/l05 are also
attested. Wording corrected in `COURSE.md`/`COURSE_cn.md`.
- Limitations: challenge-mode run (TASK.md → tests), not the taught
COURSE.md walkthrough; teaching quality remains pending. The agent
implemented all four functions in one pass, so it did not walk Lesson 3's
per-turn sequence (that sequence was measured mechanically, above).
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.