Chinese Open-Weight AI Models: How to Evaluate Them

This is general information about evaluating AI models. It is not legal, procurement, or export-control advice, and license terms change. Read the license attached to the specific model version you plan to run.

Most of them are not open source

The models people mean when they say open source are almost always open weight. You get the trained parameters. You can download them, run them on your own hardware, and fine-tune them.

You do not get the training data. You usually do not get the training code. You cannot reconstruct the model or audit what went into it. Open weight is a distribution choice, not a transparency guarantee.

The license often is not open source either. Several popular releases carry acceptable-use clauses, user-count thresholds, or restrictions on training competing models. Those are real terms, and the word on the announcement blog does not tell you what they say.

The adoption stat everyone repeats is wrong

You will see a claim that roughly 80 percent of startups pitching a16z run on Chinese open-source models, usually credited to Martin Casado.

Casado has publicly corrected it. The 80 percent described startups already building on open source, which is roughly a fifth to a quarter of the companies pitching. Four in five of everyone is a different claim from four in five of a subset.

The correction matters more than the number. If you are choosing a model because everyone else supposedly did, you are outsourcing the decision to a statistic that turned out to describe something else.

Four questions that decide fit

• What does the license permit for your use? Commercial use, redistribution, fine-tuning, output ownership. Read the license for the version you will run, not the family. Terms differ between releases from the same lab.
• Where will it run, and who sees the data? Self-hosted open weights mean your prompts never leave your infrastructure, which is the strongest privacy argument available. Calling the same model through somebody's hosted API gives that away entirely. The model did not change. The data path did.
• What does it cost you in full? Free weights are not a free system. GPU capacity, someone who can operate it, evaluation work, and the upgrade treadmill are the actual bill. Below meaningful volume, an API is usually cheaper.
• How does it score on your work? Benchmarks describe benchmarks. Build a set of thirty to fifty real examples from your own workload, with known good answers, and run every candidate against it. This is the single highest-value hour in the process and almost nobody spends it.

What the origin question is about

Country of origin is worth thinking about, and it is worth thinking about precisely, not vaguely.

Weights you download and run offline do not phone home. That is checkable, and you should check it, not trust it. The concerns that survive are different: whether your sector or your customers' contracts restrict it, whether export rules touch your situation, and what the model does with politically sensitive prompts, which is a capability question you can test in an afternoon.

Sector rules and procurement policy are the part that blocks deals, and they vary enough that a general article cannot answer it for you. That one goes to your compliance owner, not to a blog.

The complication worth sitting with

Open weights are usually argued as insurance. Nobody can deprecate your model, raise your price, or change the terms underneath you.

That holds, and it costs something. You inherit the maintenance, the security patching, the evaluation of every upgrade, and the decision about when to move. Teams that adopt open weights for independence and then never upgrade end up running an aging model badly, which is worse than the vendor lock-in they were avoiding.

The honest position is that neither choice is safe. One buys convenience and takes on dependency. The other buys control and takes on work. Pick the failure you would sooner own.

Where this leaves you

Model choice matters less than most teams assume, and evaluation discipline matters far more. A well-evaluated second-choice model beats a fashionable first choice nobody tested.

The durable skill is reading claims like the 80 percent one and asking what population it described before repeating it. Our AI Fluency Diagnostic ($79) grades that across a team and returns a verdict on where the gaps sit.

Frequently Asked Questions

Is Chinese open-source AI open source?

Usually it is open weight, which is different. You get the trained parameters and can run and fine-tune them, but not the training data or, generally, the training code, so the model cannot be reconstructed or audited. Licenses also vary and often carry acceptable-use clauses, user thresholds or restrictions on training competing models.

Do 80 percent of startups use Chinese open-source models?

No. That figure is a misreading of a remark by Martin Casado, who has publicly corrected it. The 80 percent described startups already building on open source, which is roughly a fifth to a quarter of those pitching. It was never a claim about four in five startups overall.

Does self-hosting an open-weight model protect my data?

Running the weights on your own infrastructure means prompts never leave it, which is the strongest privacy position available. Calling the same model through a third party's hosted API gives that up completely. The model is identical in both cases. What changes is the data path, and that is what determines the privacy outcome.

Is running open weights cheaper than an API?

Not automatically. Free weights are not a free system. The real cost is GPU capacity, an operator who can run it, evaluation work, and keeping current as new versions ship. Below meaningful volume, a hosted API is usually cheaper. The crossover depends on your throughput, so calculate it instead of assuming.

How should I compare candidate models?

Build a set of thirty to fifty real examples from your own workload with known good answers, then run every candidate against it and score the results. Public benchmarks describe public benchmarks. A private evaluation set is the only thing that tells you how a model performs on the work you do.