Most AI skills are a pile of prompts nobody has measured. We do not want to ship that. Every routine inside an MCAFax skill has to earn its place, and the Lab is where it does that. Our rule is simple: skills stay lean, and a new idea replaces what it beats. It does not get added on top.
How a test works
Every idea starts as a micro-skill: one narrow technique we can test, such as spotting a stacking clause in a cash-advance contract or using answer-first intros on guides. Each one goes through the same steps:
- Write it down first. The hypothesis, metric, minimum sample and stop rule are fixed before the test starts.
- Name the champion, the version we use today. If we have no measurement yet, we measure the current version first.
- Run the challenger fairly: same inputs or matched groups, one change at a time.
- Decide at the minimum sample, not before.
- The winner replaces the loser. Losers are cut, with the reason written down.
Where we use outside research, we read it, link it and treat it as a reason to test, not as proof. Plenty of published rules look strong in one study and fade in the next.
Champion vs challenger
The champion is what the skill does now. The challenger is the new idea. The challenger only wins if it beats the champion at the sample size we set in advance. If it fails, or the result is inconclusive, the skill stays as it is. Most ideas lose, and that is fine. A lab that never cuts anything is a marketing page.
Minimum samples
We set a minimum for each kind of test before it starts, and below that minimum a result is marked inconclusive, never a win. Some examples of our current rules:
- Paper trade desks: 30 closed trades per variant and 60 trading days.
- Page tests on MCAFax.com: matched page groups of 15 to 20 pages, run for 4 to 8 weeks.
- Social tests: 20 posts per variant, compared by median.
- Contract and calculator routines: 20 to 60 labeled examples.
These are our own rules, not statistical guarantees, and they can change as we learn.
Paper trading is hypothetical
Our trade desks run on paper: hypothetical trades with modeled fills. Real trading has slippage, taxes, gaps and emotions that paper does not. We count costs, delay entries by a day to see if the edge survives, and compare against random entries and against the S&P 500. A strategy that wins on paper for a few weeks has told us very little, and we say so.
Not financial or investment advice; paper trading results are hypothetical.
Tests run on our own sites and accounts
We do not test on your business. Search and AI-answer tests run on MCAFax.com: we split similar pages into groups, change one thing in one group, and compare against pages we left alone. Social tests run on our own @MCAFax account, changing one variable at a time, such as the first line or where the link goes. When data is missing, we write "no data", never zero.
Your AI keeps your own information on your side. MCAFax supplies public knowledge, and your AI applies it to your situation. We test our methods on our own data and our own properties, so we do not need your information to improve a skill.
Disclosures
MCAFax may receive compensation from some providers. It does not affect our grades or what we show you. Nothing in the Lab is legal, tax or investment advice.
