← AP News
NO. 48 · Insight

Swapping a model takes weeks — seven in ten companies that tried it said "weeks or more," Ramp counted "8% of businesses changed providers in September," and PayPal kept its model and lifted accuracy from 43% to 95%

Translated from the Korean original by AI; native-speaker review pending.

In an October survey published by VentureBeat Intelligence on November 12, 47% of companies said they had swapped the AI model in a production app within the past year, and 69% of those who had done so said it took "weeks or more" of engineering beyond changing the model name (the swap group is 45 respondents, so a small sample). 44% either had to rework the app after performance dropped or reverted to the original model. Ramp wrote on October 6 that "the share of businesses switching AI model providers hit a record 8% in September," and PayPal said at an event that it kept its model unchanged and raised its internal assistant's accuracy from 43% to 95% by placing business definitions, query rules and analysis procedures outside the model (a single VentureBeat source). The same week, Anthropic published a post saying old prompting habits such as "think hard" slow down Opus 5.5 — it is a post by the maker of the AI that wrote this article, so we add no assessment. We group the three under "what is tied to the model vs. what sits outside it," and close with two "in brief" items: MathArena's new rankings (a 4.8-point gap between first and third place, a 15x cost gap) and the eBay terms clause banning "automated ordering without human review."

This article was written with AI (Claude) and reviewed by an editor. Our company is a paying customer of both Anthropic and OpenAI, and we also use Google models in testing. Item 3 is a post written by the maker of the AI that wrote this article (Anthropic) about its own model; the Ramp post in item 2 contains a sentence saying Anthropic's share grew, the rankings table in "In brief" lists an Anthropic model in second place, and the assistant on PayPal's case-study page uses Claude as its front end. For that reason, we add no assessment to any side and only carry over the numbers and sentences. Figures announced by companies are marked as "announced by the company," and anything from a single source is noted with that source. This article is not investment advice on any specific company.

In NO.04 we wrote "five criteria a small team works by when models change every week," and in NO.46 we wrote about the speed at which the "daily driver" slot changed within two days. This week's numbers are the other side of that — not how fast models change, but what it costs us to change a model. We read one survey, one set of payment data, one company case, and one post in which a model maker says "drop this habit," dividing them into "tied to the model / kept outside the model." We write in three parts: facts, interpretation, and what we would do.

1. VB Pulse — 47% of companies swapped a production model, and 69% of those who did said "weeks or more" (facts)

VentureBeat's research arm, VentureBeat Intelligence, published its October survey ("VB Pulse — Agentic Orchestration") on November 12 as an article and a report page. The sample is small. 100 qualified respondents out of 106 (72% people doing AI work at organizations with 100 or more employees, 28% consultants serving such organizations; 88% came through a survey panel), organization size over 10,000 for 31% and 100–250 for 23%, multiple respondents from the same organization in 18 cases, and the report itself notes that "as a self-selected sample it is hard to generalize to all enterprises."

Here are the numbers. 47% of companies swapped the model in a production AI system within the past 12 months while keeping the app. Among the 45 respondents who had done so, the engineering required beyond changing the model name was — "almost none, a configuration change" 13%, "days" 9%, "weeks" 33%, "a month or more" 36%, "don't know" 9%. Weeks or more adds up to 69% (the article body rounds it to 70%; the report says 69% — this article uses the report's figure). Outcomes were "the new model was the same or better" 56%, "performance dropped and we reworked the app" 33%, "we reverted to the original model" 11% — reworked or reverted combined, 44% (the article says 45%). Compared with the September survey, "a month or more" rose from 12% to 36%, but the article said the respondent mix differed so read only the direction, and noted that the 44% is not a statistically significant difference from September (58%).

The survey did not ask "why did you switch" or "what did you fix." The article gives four examples. (1) Prompts, tools and tests are tuned to one model and have to be reworked (including the "harness" that determines which tools can be called). (2) Evaluation sets must be built from real requests, and gathering that data is the hardest part (Box's Ben Kus). (3) Fine-tuning does not carry over to the next model — PayPal's Len Zhang said "when a new model comes out, you have to fine-tune again on that model." (4) The same model yields very different results in different harnesses — in one benchmark cited by the article, Kimi K3's "median cost per passed task" was $18.34 in one coding harness and $3.28 in another, with the same 63.3% pass rate (an example from a single article). JPMorgan is described as putting each new Claude model through internal approval before employees use it, and Upwork's Sudhanshu Bist said that for a production system he would "probably keep the existing model." On the side of making swaps easier, the article lists gateways and routers (change one line, but check the app separately), keeping business knowledge outside the model (PayPal, in item 2), testing in the harness you will actually run, measuring by "cost per good outcome," and doing what T-Mobile did — running the old and new versions side by side on real calls and scaling only the better one.

2. Ramp: "8% switched providers in September"; PayPal: "same model, 43% to 95%"; Uber: "cost per cleanup down 90%" (facts)

Ramp (a US corporate card and spend management company) published a post from its data team on October 6 titled "Businesses are changing AI model providers more often" (by Ryan Stevens, Director of Applied Science). The share of businesses that switched AI model providers hit a record 8% in September, a monthly rate on a three-month average. The post describes two movements — Anthropic greatly increased its share after the Opus 4 and Sonnet 4 launches in May 2025 and switching declined for the following year, and OpenAI's price cut and new model releases in June 2026 broke that balance. It says there have been nine or more model release announcements from the two companies since June. The author wrote, as a possibility, that "this may be an early signal that models are becoming a commodity." The post does not state the sample size, the criterion for counting a "switch," or the sample composition (Ramp's November AI Index itself notes that "customers skew toward tech and finance" — NO.47). The sentence about Anthropic's share growing is carried over as Ramp wrote it, and we add no assessment.

PayPal's figures come from a single VentureBeat article dated November 10 — a write-up of remarks by two PayPal executives at VentureBeat events on October 8 and October 28, not an official PayPal post. Len Zhang (Vice President and Head of AI Product) said the internal accuracy of an internal data science assistant (one that answers business questions that used to go to the data science team) rose from 43% to 95% with "the LLM unchanged." What changed was the knowledge layer outside the model — "golden queries" that define which data to pull, and how, for which purpose; "analysis playbooks" that describe how to dig into a question, such as login failures or passkey usage; and business definitions such as "what is a new account," given as knowledge instead of leaving the model to infer them from data. How 43% and 95% were measured, and over what period, is not in the article (the 1,000+ questions over 10 weeks and the estimated 1,600 hours saved are a separate sentence, and the article itself added that "question volume does not prove productivity"). In the same article, PayPal said it tested four to five models for a customer-facing commerce agent — large models had latency of around 15 seconds, an 8-billion-parameter model was fast but less accurate, and fine-tuning improved it but not enough — and that it reduced its reliance on fine-tuning because of the cost of retraining every time the model changes (not abandoning it entirely). For reference, Google Cloud's PayPal customer case page (undated) describes the same kind of assistant — Looker's MCP server placed on an internal MCP hub, queried in natural language from Claude Desktop, roughly 5,000 users, with metric definitions fixed by Looker's semantic layer. That page's "nearly 100% accurate against defined metrics" is a vendor-side claim with no baseline or method, and the 43%→95% figure does not appear there. We also could not confirm whether the two articles refer to the same assistant.

Uber appears in a different VentureBeat article from November 10 (the VB Pulse infrastructure survey — 156 respondents, 38% "delayed, scaled back or canceled an AI project because of cost overruns" versus 12% for "compute shortages"). Uber's Uday Kiran Medisetty (Vice President and Distinguished Engineer) said that after benchmarking managed agents on real tasks to choose a model, "cost per cleanup" fell 90%, with quality measured by reverts and incident counts and cost measured "per completed review or cleanup." The article notes that the 90% is a figure specific to Uber's tasks.

3. Anthropic — "drop these seven old prompting habits," one of which comes from a 2022 paper (facts, conflict of interest)

On November 12, Anthropic posted an article on claude.com titled "Using Claude Opus 5.5 as your daily driver for knowledge work" (by Aleksandra Todorova; summarized by The Decoder on November 14). Because it is a post by the maker of the AI that wrote this article about its own model, we carry over only the list and the figures the company stated. Opus 5.5 is the default model on the Pro, Max, Ultra and Team plans, and the post names seven habits to change — (1) adding phrases like "think hard" (said to be unnecessary because this model always thinks); (2) starting the effort level at High (start at the default Medium and raise it only when it falls short); (3) giving instructions one step at a time (state the goal, the output format and when to ask back, all at once); (4) asking for a draft or outline before the finished version; (5) transcribing chart numbers by hand (attach the image or file instead); (6) vague prohibitions like "avoid generic design" (be specific about what to avoid); (7) splitting work so only complex tasks go to Opus and the rest to Sonnet (the post says running Sonnet at a high effort level brings cost per task close to Opus 5.5).

The figures the company stated are as follows, and all are the company's internal tests. Opus 5.5 at Medium matched or beat Opus 5 at High on its own coding and knowledge-work evaluations; removing the "think" phrase made responses start faster with no clear drop in quality; the lowest effort level read dense charts more accurately than Opus 5's highest level; 16 of 18 reports built from saved web snapshots passed an automated check for "fabricated figures and citations"; and a merger-model task took 63 minutes (Opus 5: 93 minutes) at half the cost. The statement from the outside company Factory that it "got the same results as Opus 5 High at Medium with 20–25% fewer output tokens" is also a quotation in the company's post. The price table in the post is Haiku 5.5 0.10/0.50, Sonnet 5.5 2/10, Opus 5.5 4/20, Fable 5.2 8/40 (input/output per million tokens).

We separately checked the origin of habit (1). According to the Wikipedia entry "Prompt engineering," "chain-of-thought," which has the model write out intermediate reasoning steps, was reported in 2022 by Google Brain as a few-shot technique with examples attached, and the finding that a single sentence, "Let's think step by step," is enough to have an effect comes from a paper the same year by researchers at Google and the University of Tokyo (Kojima et al., NeurIPS 2022). The same entry notes that a meta-analysis of some 100 papers found the gains concentrated in math, logic and symbolic reasoning, with small gains on other tasks and lower accuracy on some, and includes the sentence that "the marginal value of elaborate prompt construction is declining" as instruction-following and reasoning improve. In other words, a habit created by a 2022 paper is one the maker is telling us to "drop" in 2026 — but the only figures so far showing that dropping it helps are that maker's own tests.

Two in brief (facts)

MathArena. The math competition evaluation site run by ETH Zurich's SRI Lab and INSAIT added results for Gemini 4 Carbon and Claude Fable 5.2 on November 13 (we opened it ourselves on November 15, and The Decoder reported the same table the same day). The overall "expected performance" (the average normalized score across non-deprecated competitions) is Gemini 4 Carbon (high) 85.4%±3.0, Claude Fable 5.2 (max) 82.8%±5.3, GPT-6.1 Sol (max) 80.6%±2.7, and the highest open model is Kimi K3.1 Pro (max) at 54.3%±5.8. Cost is "the average cost of solving one problem once," at $15.70, $40.20, $1.07 and $3.70 respectively. The gap between first and third is 4.8 points and roughly 15x in cost, and the intervals of first and second (±3.0 and ±5.3) overlap. The Decoder wrote that Carbon's most common failure was "timeout," and that in one case it ran a script for more than 11 hours without checking progress (The Decoder's primary source is an evaluator's X post; we confirmed with the table on the site). Because the model in second place is Anthropic's, we add no assessment of the rankings.

eBay. We opened eBay's User Agreement directly (the current version took effect for existing users on June 28, 2026). Section 3 prohibits the use of robots, spiders, scrapers and other automated means "without eBay's express written permission," and gives as examples, in parentheses, "buy-for-me agents, LLM-driven bots, or any end-to-end flow that attempts to place orders without human review." According to PPC Land (November 14), this wording is not new but has been there since the January 20 revision (effective February 20), and the report that Meta's Muse access to eBay became unstable in early November is relayed from The Verge's November 10 story (which would not open on our end). The eBay spokesperson statement PPC Land carried is that "we have restricted agent access in some cases, including unauthorized scraping and training on our data," and no list of permitted agents or application process has been published. It is a sentence that sits in the same row as United (NO.45, 47).

Reading the three together (interpretation)

First, the cost of changing a model is not the cost of the model but the cost of everything around it. In a gateway, the model name changes in one line, yet seven in ten of those who swapped said "weeks or more," for exactly the reasons the article gives — prompts, tools, tests and fine-tuning are tuned to that model. So "the share of companies switching models often rose to 8%" (Ramp) and "a swap takes weeks" (VB) are not opposites; they are the same cost seen from both sides (interpretation). That said, both numbers have small samples or unstated compositions, so we use them to read "what is tied" rather than "how much."

Second, the only things that carry over to the next model are the ones kept outside it. What PayPal improved was not the model but "the definition of a new account" and "this is how you dig into this question," and those are assets that remain usable when the model changes. Fine-tuning, by contrast, is a cost that disappears along with the model, which is why PayPal reduced its reliance on it. In NO.05 we wrote that "instructions given to AI are a liability, not an asset," and with this week's numbers we cut that sentence once more — instructions with a model's name baked in are a liability; business definitions, worked answers and test sets are assets (interpretation). Uber's 90% is also a number that came not from "choosing the model well" but from having a benchmark made of real tasks (the caveat that it is specific to Uber is carried over as the article gave it).

Third, habits are tied to the model too. "Think step by step" is a sentence a 2022 paper created, and four years later the maker wrote that the sentence slows its model down. Whether that is true is something we have to measure ourselves — the figures in the maker's post are all the maker's tests, and that company is the company behind the AI that wrote this article. But the question itself — "which of the habits baked into our prompts came from which model?" — is valid regardless of the model (interpretation). MathArena's table is the cost side of that question — a 4.8-point score gap at 15x the cost, with first and second place overlapping in their intervals. NO.40's "a one-point gap is within the interval" and NO.46's "price per task, not price per token" show up in the same shape in math problems.

What we would do — three things for a team running apps, commerce and AI channels this week (in practice)

  1. Run one "model swap rehearsal" and write down the time it took. Take the three tests from NO.17 and NO.46 (ingredient-label judgment, app copy, Ria's script) and actually switch them from our current default model to one other model, then record the time in four columns: "changing the model name / fixing prompts / fixing tool calls / re-aligning tests." We do not have to go all the way — whichever column takes longest is our list of "what is tied." Next to the result, write "cost per good outcome" the way the VB article counts it.
  2. Build a box that sits outside the model. Separate SafeList's judgment criteria (say only as much as "no avoided items were found" — NO.03), our commerce terminology (what "new customer," "cancellation" and "return" mean), and 20 worked answers for ingredient-label judgment into separate files rather than prompts, and change the prompts so they only point to those files. Add one line to NO.05's "delete instructions every quarter" check — flag "any sentence that presupposes a model's name or a specific model's quirks," and delete those sentences first at swap time.
  3. Test the version with "think" phrases removed against our own numbers. A maker's recommendation is only a recommendation, so run the prompts for the three tests with and without "think hard"-type sentences at the same effort level, 20 runs each, and record accuracy, time to first token and output token count. Whichever way it comes out, keep it in our table as two rows: "announced by the company / confirmed by us." For commerce, just one line — after looking at the eBay and United wording, check whether our own terms have a clause on "end-to-end automated ordering without human review" and, if so, whether an application route for permission is written in, and attach it to the draft of the "why" column in NO.45.

What we did not pick this time

The three The Decoder posts of November 13–14 on Anthropic's IPO roadshow, CEO remarks and a former researcher's claims are investment-decision material with a conflict of interest, so we did not use them. Analysis of Nvidia's executive reshuffle (The Decoder, VentureBeat) is outside our topic. Runway's release of Praxis-1 weights (Apache 2.0, robot control) and OpenAI's robotics team reassignment come from The Decoder alone, so we leave only the headlines. Anthropic's Claude in Google Meet (VentureBeat, November 12) and self-hosted Claude Managed Agents (Help Net Security, November 12) are conflicts of interest, so we note only the headlines. Help Net Security's "a pinned MCP server may not be pinned" (November 13) and its "stolen Claude API keys" article are security incidents, so headlines only — once an official notice comes out we will cover them as a follow-up to NO.32 without describing the methods. PPC Land's report that "the EU proposed a fine of 6% of global revenue for paid AI recommendations" is on hold because we could not open the official document. The two Skift articles of November 11 (hotel chain MCR's CEO on "banning AI bot bookings" and Citi on "AI companies should pay for mistaken bookings") and the November 9 "check of AI crawler directives on 50 travel sites" are follow-up candidates for NO.37 and 45 as we did not open the full text. The title of another October VB Pulse report — "78% of agents' confident wrong answers came from missing or conflicting business definitions" — points the same way as item 2, but we did not open the full text, so we note only the title. As of November 15, Meta's newsroom still has no post on Muse's phone feature (NO.47).

What we could not confirm in this story

  • The VB Pulse swap group is 45 respondents. "Weeks or more 69%" and "a month or more 36%" are shares within those 45, and the report described the difference from September as "direction only" and "not significant." The article's 70%, 45% and 55% differ in rounding from the report's 69%, 44% and 56%.
  • The Kimi K3 example in which cost per task split into $18.34 and $3.28 depending on the harness is one benchmark cited in the VentureBeat article, and we did not open that benchmark.
  • Ramp's "8% in September" has no sample size, counting criterion or sample composition in the post. That Ramp's data consists of corporate card payment records is an explanation from other Ramp posts (NO.43, 47) and is not stated in this one.
  • PayPal's 43%→95% comes from a single article relaying remarks at VentureBeat events (no confirmation from other outlets apart from republishing sites), with no measurement method, period or test set. The Google Cloud case page's "nearly 100%" is a vendor-side claim, and we could not confirm that the two articles describe the same assistant.
  • The figures in Anthropic's post are all the company's internal tests with no independent confirmation. Factory's 20–25% is an outside company's statement quoted in the company's post.
  • We saw MathArena's table ourselves, but the details of which effort level and settings each model ran at, and the "11-hour timeout" case, are only in The Decoder (relaying an evaluator's X post).
  • We read only the first 100,000 characters of the eBay User Agreement (the remaining 27,000 unread), and the Muse access instability is PPC Land's relay of The Verge's report. A list of permitted agents exists nowhere.

What we read alongside

We opened and checked sources of different kinds directly. They are listed separately as company announcements and official documents, survey and evaluation organizations and data, media reports, wiki, and our earlier articles.

Company announcements and official documents:

Survey and evaluation organizations and data:

Media reports:

Wiki:

Our earlier articles:

← Weekly AI News — Google change…A broadcast with no host and n… →