Wrap-up

You built an AI agent from scratch, measured it, broke it and made it safer. Here’s what to take home.

What you built

• A chat client in a single HTTP request, working with any OpenAI-compatible provider
• An agent loop with five tools and a step limit
• An eval suite with tool checks, text checks, an LLM judge and repeated runs
• A security test suite built from real attacks
• Five layers of defense, with numbers that show they work

Five things to remember

1. An agent is a model in a loop with tools. Frameworks add convenience, but the core is the 20 lines you wrote.
2. Models are stateless. The conversation you send is all they know, including any text a tool brings in.
3. If you don’t measure it, you don’t know it. Run evals more than once, and before and after every change.
4. The model is not a security boundary. Put identity, permissions and approvals in your code.
5. Every bug and every attack becomes a test case. That’s how fixes stay fixed.

Keep going

• Put evals in CI. Run run_evals.py --min-pass-rate 0.9 on every pull request, against a cloud model or a self-hosted runner with Ollama.
• Add a real tool. Connect the agent to something you use: a calendar, a database, a weather API. Ask yourself what it should not be allowed to do, and write that test first.
• Try a framework or protocol. Explore MCP, LangChain or the Microsoft Foundry Toolkit. You’ll recognize the loop inside each of them.
• Read more about the risks in the OWASP Top 10 for LLM Applications.

Share what you built

Post a screenshot of your flags or your eval results, and share feedback or suggestions through GitHub issues.

Found a bug or have an idea for this workshop? Open an issue or see Contributing.

Thanks for building with us!