What Is a Skill Editor? Keep Your AI Skills Improving

What is a skill editor?

A skill editor is a skill whose only job is to maintain your other skills. A skill is a saved set of instructions for one recurring task, such as prepping a podcast guest or editing a video. A skill editor reads many of your past conversations, finds the corrections you keep making, and updates the right skill with those patterns. It does not patch a skill after every chat. Product manager and creator Peter Yang uses this idea to keep his skills clean as he keeps tweaking them.

Why do AI skills get worse the more you tweak them?

Most people fix a skill the moment it annoys them. The output has a problem, so you open the skill and add a line. This feels productive, but it has a cost.

Each patch reflects one situation. After a month you have a skill full of narrow rules. Some contradict each other. Some only applied to one project. The skill gets longer, the model has more to juggle, and the output gets less predictable.

Say you edit a video with an AI skill and one clip needed tighter pacing. You add "always cut pauses longer than half a second." The next video is a slow, reflective interview where pauses matter. The rule now works against you. You blame the model, but the skill is the problem.

This is overfitting. A rule that fit one conversation gets applied to every conversation.

How does a skill editor work?

The idea is to separate doing the work from maintaining the instructions. While you work, you just correct the output in the chat, as you normally would. Later, on a schedule or whenever you choose, you ask the skill editor to look at the pile of conversations together.

Looking at many threads at once changes what you can see. In a single chat, every correction looks equally important. Across a dozen chats, the picture sorts itself out:

• Corrections that appear repeatedly are real preferences.
• Corrections that appear once were probably about that specific job.
• Corrections that conflict show you where the skill is unclear and needs a decision from you.

The editor then proposes changes to the skill: add a rule, sharpen a vague one, remove one that no longer earns its place.

What does a skill editor look like in practice?

In the workflow Peter Yang walked through, the skill editor sits alongside his chief of staff, an advisor, a council of AI perspectives, and skills for podcast prep, titles and thumbnails, video editing, and newsletters. Those skills run across ChatGPT, Claude Code, and Grok Bot. The skill editor is the one that keeps the others tidy. He demonstrated it by analysing 13 rounds of feedback on a video, so the editor could see which notes kept coming back instead of reacting to the latest one.

I won't guess at what that review found, because I can't verify it. The useful part is the shape of the method. Thirteen rounds of feedback is a lot of signal. Any one round might say "make the intro shorter." Thirteen rounds can show you that the same three notes appear in nearly every round. Those three notes belong in the skill. Everything else probably does not.

Patching after every chat vs. reviewing in batches

Patching is fine for an obvious bug, like a skill that points to the wrong file. For preferences and style, batch review is better.

How do I set up a skill editor myself?

You don't need a special product. You need your skills, your past conversations, and a clear prompt. Here is a simple process.

1. Pick one skill. Choose something you run at least weekly, like a weekly report, a video edit, or a client email draft.
2. Collect the threads. Gather several weeks of conversations where you used that skill. Export them, or point your AI tool at your chat history if it can read it.
3. Extract your corrections. Ask the editor to list every time you pushed back, rewrote something, or said "no, more like this."
4. Group and count. Have it cluster the corrections by theme and say how many separate threads each one appeared in.
5. Propose edits. For each repeated theme, it drafts a specific change to the skill. One-offs go on a separate "not added" list.
6. Review and approve. You read the changes, keep what is right, and reject what is not.
7. Test on a real task. Run the updated skill and check whether the old corrections stop appearing.

Here is a prompt you can adapt:

You are my skill editor. Your job is to improve the skill below
using my past conversations. Do not edit it based on any single chat.

Skill to improve:
[paste the current skill]

Conversations:
[paste or attach several weeks of threads where I used this skill]

Do this:
1. List every correction, rewrite, or complaint I made.
2. Group them by theme. For each theme, say how many separate
   conversations it appeared in and quote one example.
3. Treat themes that appear in 3 or more separate conversations
   as candidates for the skill. List the rest as "one-offs, not added."
4. Flag any corrections that contradict each other and ask me
   which way I want it.
5. Propose a revised skill. Show every change as old text -> new text,
   with a one-line reason. Remove rules that duplicate each other.
6. Keep the skill short. Do not add a rule unless the evidence supports it.
Do not apply anything. I will approve changes.

The "3 or more" threshold is a rule of thumb I'm suggesting, not a fixed standard. Adjust it to how much data you have.

Is a skill editor worth the effort?

It depends on how often you use the skill. If you run a task once a quarter, you won't have enough threads to learn from, and a manual edit is fine. If you run it several times a week, the review pays off quickly, because every repeated correction you remove is time you stop spending.

There is also a durability argument. Yang's point in the same conversation was that tools and models change every few weeks, so what you build should travel with you. Skills are saved instructions, so they can move between tools. A skill that has been refined against real use is an asset you keep, whichever tool you use next.

A few cautions:

• Keep the editor on a leash. It should propose, not rewrite silently. A skill that quietly drifts is worse than one you edit by hand.
• Protect the parts that must stay yours. Yang keeps his goals document human-written and lets AI only suggest changes. The same principle applies to your core instructions.
• Watch for bloat. If the revised skill is longer than the original and you can't say why, cut it back.
• Start small. Yang's own advice for the larger system was to start with one recurring task that would save you time. The same applies here.

What does this mean for you?

If you use AI at work for the same tasks repeatedly, you are probably already correcting the same things over and over. Writing a weekly update, drafting recruiter outreach, summarizing meeting notes, formatting a report: all of these produce repeated feedback.

What you can do this week:

• Marketers and content teams: take your brand-voice or draft-review skill and run a review over a month of edits you made to its drafts.
• Recruiters and managers: review the threads where you rewrote outreach or feedback messages, and find the tone corrections you keep making.
• Analysts and accountants: look at the repeated formatting or checking steps you ask for after a first pass, and move them into the skill.
• Anyone: save your chats for a few weeks before you edit a skill. The waiting is what gives you the evidence.

Could I build my own skill feedback tracker?

You can run the review with just a chat window, but if you want a lasting record, a small tool helps. For example, a corrections log: a form where you note which skill you used and what you changed, plus a view that counts how many times each correction has come up. That turns the "3 or more" rule into something you can see instead of reconstruct from memory. Remy builds this kind of app from a plain-English description. It produces a deployed full-stack app with a database and sign-in, so you could describe the log and the repeat-count view and use it yourself or share it with a team instead of keeping it in a spreadsheet.