# How Feedback Bench 1.1 works Window: every full Monday-to-Sunday week since 31 August 2026. It grows by one week each week. **Disclosure.** Feedback Bench is built and maintained by Enterpret. Enterpret sells feedback analytics to software companies. No vendor pays to be listed, removed or re-scored. ## 1. What this ranks Feedback Bench ranks AI agents by what their users say about them in public. Lab benchmarks test a model on fixed tasks. They do not show what it is like to pay for an agent, hit its limits and trust its work for a month. Users write about that in public every day. Each category, such as coding agents or personal agents, is ranked on its own: its agents are compared only with each other, on criteria written for that category. Every category uses the method on this page. Where a step differs by category, this page describes the category you are reading. Version 1.1 asks two questions about every agent and weighs them equally: - **Popularity**: how many people talk about the agent, on a log scale from the least-discussed agent (0) to the most-discussed (1). - **Customer love**: how well its users, speaking from their own use, rate it compared with the other agents, on the same criteria. The norm of its peers is 0.5. The **Feedback Score** puts both on a common scale (section 7). 50 is an agent that is average on both; one standard deviation is 10 points. Four rules shape the method: 1. Only stances from the author's own use count as ratings. 2. Each agent is compared with the other agents, never with its own users. 3. Popularity is on a log scale, and small samples are pulled toward the norm, so an agent needs volume to rank high. 4. One account counts once per agent, platform and week. A **post** is a Reddit post or comment, an X post, or a G2 or Trustpilot review. A **channel** is the platform it comes from. *Figure 1: From post to rank. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* ## 2. Collecting and attributing posts Each category lists the channels where its users post, and collects every agent through the same kinds of routes: the agent's own spaces, posts that name it, and reviews of the product itself, never of a parent brand. Every coding agent is collected by the same four routes: 1. **Its official subreddits**, all posts and comments. 2. **Reddit posts and comments whose own text names the agent**, from any subreddit we collect and from one Reddit search per agent. 3. **X posts that mention the agent's official product handles**, not staff members' personal handles. OpenAI Codex has no product handle, so it uses an X search for "OpenAI Codex", "Codex CLI" and "Codex app". 4. **G2 and Trustpilot reviews** of the product, not of a parent brand. Reddit and X carry almost all of the data. Some agents lack a route: Grok Build has no subreddit or product handle, Amp has no subreddit, and Pi, Zed and Conductor have no review pages. Devin includes r/windsurf and the Windsurf handle, because Windsurf is now Devin. Google Antigravity includes r/GeminiCLI and @geminicli, because Antigravity CLI replaced Gemini CLI. **Attribution.** A post in an agent's own subreddit counts for that agent. Any other post counts only when the labeller (section 3) confirms that it is about the agent. A name match is not enough: "cursor" can be a mouse cursor and "amp" an amplifier. A post that names two agents counts for both. ## 3. Labelling The **labeller**, a large language model, labels every post with its category's **codebook**, v1.0: a written list of criteria, each with a definition and a boundary with its nearest neighbours. One model labels each week. From the week of 12 October 2026 a newer model labels the new weeks; earlier weeks keep their labels, and a post that arrives late is labelled by its own week's model, so every agent in a week is labelled the same way. For each post and agent it records: - **About**: whether the post refers to this agent. - **Judges**: whether the post takes a stance on the agent. A question or a news link can be about an agent without judging it. - **Criteria and polarity**: which criteria the stance names, and for each one praise, complaint or mixed. This is aspect-based sentiment, applied to product mechanisms (Hu and Liu 2004). A second pass by a large language model reads every judging post and marks whether its stance comes from the author's own use of the agent. A user who reports what the agent did, what it cost or how it fits their work speaks first-hand. A reaction to an announcement, a benchmark or a price change, a plan to try the agent, a relayed report and a promotion do not. Only first-hand stances count as ratings. Review sites that accept reviews only from verified customers carry fewer promotional reviews (Mayzlin, Dover and Chevalier 2014). Large language models now label text at scale for social science, with a written codebook as the instrument (Ziems et al. 2024). A **criterion** names one product mechanism that one team could own and one post can show, such as the usage quota. On the site each has a short label that names the outcome users want, such as "Quota lasts". The criteria sit in **areas** ordered along the user's path: from paying and limits to account and support. Each category's criteria came from its own posts: a large language model listed every concrete thing a stratified sample of judging posts praised or complained about, with no categories given, and grouped the items into criteria, as in inductive codebook development (MacQueen et al. 1998). Two large language models then assigned every item to one criterion independently, and criteria they could not tell apart were merged or redrawn. The technical reference lists every criterion of the category, with its definition and boundary. ## 4. The unit: the user-week One frustrated user can write twenty comments in one thread, so posts are not independent, and the unit of analysis must be the person, not the post (Altman and Bland 1997). The unit of measurement is the **user-week**: one user, on one platform, talking about one agent, in one Monday-to-Sunday week. For each criterion and area, a user-week adds up that user's labels that week: +1 for praise, −1 for complaint, 0 for mixed. A positive sum makes a **positive** user-week, a negative sum a **negative** one, and zero is not rated. Positive and negative user-weeks are **rated user-weeks**. Customer love counts them, never raw posts. A **user** is a Reddit or X account, or a named G2 reviewer. A Trustpilot review counts as its own user, because the collected reviews carry no reviewer name. **Recency weight.** Every week since 31 August counts, but recent weeks count more, because opinion of a product drifts as it changes (Koren 2010). G2 also weights older reviews less (G2 2026). Each user-week carries a weight: ``` weight = 0.5 ^ (age / 4) age = weeks before the latest full week ``` The latest full week weighs 1, a week 4 weeks earlier 0.5, and a week 8 weeks earlier 0.25: the weight halves every 4 weeks. Customer love, the peer norms and the area weights add up these weights. Popularity counts each user with the weight of their most recent week. The counts the site shows are not weighted. ## 5. Popularity Popularity counts the accounts that post in or about an agent's spaces: every user with at least one post that counts for the agent. A complaint counts the same as praise. An agent's **share of voice** is its distinct users, each weighted by their most recent week, divided by the same sum over all agents. Popularity puts that share on a log scale, from the least-discussed agent to the most-discussed: ``` Popularity(a) = (log S(a) − log S_min) / (log S_max − log S_min) ``` The log scale gives every tenfold increase in people the same step. The formula is the income index of the Human Development Index (Klugman, Rodríguez and Choi 2011). Popularity measures how many accounts post about an agent on the collected channels, not how many people use it. ## 6. Customer love Customer love measures how well an agent's users rate it compared with the other agents. Complaints outnumber praise across the category, so each agent is compared with its peers, not judged on its raw share of praise. It is computed for each area, then combined. **Step 1: the peer norm.** For each agent and each area or criterion, the **peer norm** is the positive share of weighted rated user-weeks of all the other agents, across all channels, smoothed by one praise and one complaint: (positive + 1) / (positive + negative + 2), Laplace's rule of succession (Gelman et al. 2013). An agent's own users never set its own bar. This matters most for the largest agents, whose users would otherwise make up much of the norm they are compared with. **Step 2: shrinkage.** A share from a handful of users is not reliable. Five praises out of seven is not evidence of excellence. So before we compare, we add 200 imaginary user-weeks that rate the agent exactly at its peer norm: ``` p(a, c) = (positive + 200 × peer norm) / (positive + negative + 200) ``` Take a norm of 40% positive. An agent with 5 positive and 2 negative user-weeks has a raw share of 71%. With the 200 imaginary user-weeks, 80 positive and 120 negative, its share is (5 + 80) / (7 + 200) = 41%: close to the norm, because 7 user-weeks say little. An agent with 3,500 positive and 1,500 negative moves only from 70% to 69%, because 200 imaginary user-weeks barely move 5,000 real ones. At 200 real user-weeks, an agent's own evidence and the norm count equally. This is a beta prior worth 200 user-weeks, and shrinkage of this kind improves estimates for small samples (Efron and Morris 1975). *Figure 2: How shrinkage pulls small samples to the norm. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* **Step 3: the comparison.** The agent's shrunk share is compared with its peer norm on the log-odds scale: ``` lift(a, c) = logit p(a, c) − logit peer norm(a, c) love(a, c) = sigmoid(lift) ``` On the log-odds scale a move from 5% to 10% counts for more than a move from 45% to 50%. That matters, because area norms differ widely. A love of 0.5 means exactly the peer norm; above 0.5, users rate the agent above its peers. The site shows the same quantity for each criterion, for reading only. **Step 4: one number.** Customer love is the weighted mean of the area lifts, mapped back to 0 to 1: ``` Customer love(a) = sigmoid( Σ_area w_area × lift(a, area) ) w_area = the area's share of all weighted area ratings, all agents ``` The weights are the same for every agent, so no agent gains because its own users discuss its strengths. This is direct standardisation (Fleiss, Levin and Paik 2003). The weights follow how often users rate an area. The current weights and norms are in the technical reference at the end of this page. ## 7. The Feedback Score Each measure is first standardised across the agents in the category. A **z-score** is the number of standard deviations an agent sits above or below the average agent: ``` z_pop(a) = z-score of log share of voice z_love(a) = z-score of logit customer love Feedback Score(a) = 50 + 10 × z( ½ × z_pop(a) + ½ × z_love(a) ) ``` Popularity is standardised on the log scale (section 5), and customer love on the log-odds scale, where its lifts are measured (section 6). The average of the two z-scores is standardised once more, so the score has mean 50 and standard deviation 10 across the agents. Standardising first matters: a measure with a wider spread would otherwise dominate the composite at any nominal weight (Paruolo, Saisana and Saltelli 2013). Equal weights on standardised parts are the most common choice for a composite indicator when there is no basis for other weights (OECD and JRC 2008; Decancq and Lugo 2013), and unit weights on standardised measures are known to be robust (Dawes 1979). G2's software rankings also combine two components, a satisfaction score and a market presence score (G2 2026). Scores are relative to the category: 50 is its average agent, and each build standardises again. Every measured agent is ranked, with no minimum volume, once a full Monday-to-Sunday week after its launch is in the window. Until then a new agent is not shown, and its posts count toward no score, norm or weight, so its launch week moves no other agent. Posts from before an agent's launch do not count. A small agent needs volume to rank high: log popularity keeps it low, and shrinkage keeps its customer love near the norm until it has many ratings. *Figure 3: Popularity × customer love. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* ## 8. Worked example: the top two Figure 4 works through the two agents at the top of the current build. For each, it shows every area's weight times lift, the sum that gives customer love, both z-scores and the Feedback Score. It also shows the customer love at which the two would swap places, with every other agent held fixed. *Figure 4: The top two, area by area. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* ## 9. How firmly each position holds Every customer love value and every score carries a 95% **interval**, and every agent has a **rank range**. They show how much each result rests on a few voices. An agent whose customer love comes from many people holds its position firmly. **Bootstrap.** We resample each agent's users with replacement, 1,000 times, with seed 20260927 (Efron and Tibshirani 1993). A user's weeks move together, as in a cluster bootstrap (Field and Welsh 2007). Each resample recomputes norms, area weights, customer love, the standardisation and the score. Popularity counts every collected account, so it is held fixed. An agent's rank range runs from the 2.5th to the 97.5th percentile of its rank across the resamples, as Goldstein and Spiegelhalter (1996) recommend for league tables. The board shows plain positions; the rank ranges are in the standings table at the end of this page and in the data files. **Readings.** A criterion or area reads as the agent's **position**: its place among the agents with at least 30 rated user-weeks there, ordered by customer love, such as "3rd of 12". Its plausible places run from 1 plus the number of agents whose interval lies wholly above its own, to the field size minus the number whose interval lies wholly below. Requiring intervals not to overlap is a conservative test (Schenker and Gentleman 2001). It **ranks high** when both its position and the middle of that span are in the top half, and **ranks low** when both are in the bottom half. With fewer than 4 agents in the field it ranks neither. Under 30 rated user-weeks, unweighted, it reads **too few posts**. No correction for multiple comparisons is applied. ## 10. Sensitivity We rerun the full ranking under each variant below, each changing one choice: a one-at-a-time version of the sensitivity analysis Saisana, Saltelli and Tarantola (2005) propose for composite indicators. - **Combination**: min-max scaling, mean rank, a standardised geometric mean. - **Parameters**: shrinkage strength 50, 100 or 400; equal area weights; area weights without paying and limits; a recency half-life of 2 or 8 weeks, or no recency weight. - **Rules**: popularity from posts marked as about the agent only; a norm that includes the scored agent's own users. - **Sources**: Devin and Google Antigravity without their earlier sources. - **Bootstrap**: two bootstraps that also resample popularity. - **Weights**: popularity alone, customer love alone, and the version 1.0 formula. Figure 5 shows each agent's rank under every variant, and Kendall's tau for each variant against the published order (Kendall 1938), where 1 means identical. *Figure 5: Rank under each variant. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* ## 11. Other outputs **Weekly scores.** Each week is also scored alone with the same method, without recency weights, on that week's users, norms, weights and standardisation. An agent with no posts in a week has no score that week. Figure 6 marks the model releases in the window. *Figure 6: Week by week, with model releases. Drawn on the [Method page](https://feedbackbench.com/coding-agents/#/method).* **Requests.** A separate pass by a large language model reads every counted post and returns each explicit ask to add or change something, in a neutral wording of at most 12 words, much as feature requests are mined from app reviews (Maalej and Nabil 2015). The wordings are grouped into themes within each criterion; a second pass then merges themes that ask for the same change under different criteria, so one ask counts once. A theme shows once at least 2 user-weeks ask for it. Requests do not enter the score. **Failure modes.** For each criterion with at least 30 complaining user-weeks, a large language model reads 150 complaint posts drawn at random and names 4 to 8 failure modes: recurring situations in which an agent fails its users there. A second model then puts every complaint post on that criterion in one mode, or in Other. Modes stay fixed once named; each week's new posts are sorted into them. A mode counts distinct user-weeks, per agent. Failure modes do not enter the score. **Quotes.** Pages quote whole public posts that clearly praise, or clearly complain about, the agent on one criterion. A second large language model confirms each quote before it shows. Links and personal details are removed and marked, such as [link]. **Briefs.** Each agent, area and criterion page carries a brief written by a large language model and labelled as interpretation. On an agent page, code first computes the findings: each area's and criterion's share of the agent's ratings against other agents, its position, the head-to-head counts, the weekly series and the levers. A published rule picks the reasons on each side, and their figures. For each reason, code gathers first-hand posts on that criterion; the model writes from these findings and posts only, a second pass edits the draft, and code rejects any text that cites a post outside its reason's set, quotes a passage that is not in the post, or uses a number that is not in the findings. Area and criterion pages follow the same pattern. Code computes the standing, every agent's position, lift and counts, the failure modes and the requests. The position rule picks the agents that rank high and low. The model quotes only praise from the agents that rank high and complaints from those that rank low, each confirmed by the quote check, and the same checks reject numbers outside the findings. ## 12. Limitations 1. **Labels.** Every label comes from a large language model applying one codebook. The model changed once, from the week of 12 October 2026. The newer model marks fewer posts as judging an agent, so weekly counts before and after that week differ in part because of the labeller. 2. **Who posts.** People who post are not a random sample of users (Ruths and Pfeffer 2014). Strong opinions and heavy users are over-represented (Hu, Pavlou and Zhang 2017), and early adopters tend to rate a new tool more warmly than later users (Li and Hitt 2008). 3. **Collection.** A Reddit search returns only its newest 1,000 or so results, and X feeds may be capped per day (Morstatter et al. 2013). Both bias popularity down for the busiest agents. 4. **Coordinated accounts.** Each account counts once per week, but many accounts acting together would each count (Luca and Zervas 2016). On Reddit, AutoModerator, deleted accounts and names ending in "bot" are dropped. Accounts that post far more than others about one agent, repeat the same text, or write in a product team's voice are reviewed by a large language model, and only those it marks as users count; vendor staff, automated accounts and promotion are left out. Report a suspected campaign with the challenge form at the end of this page. ## 13. Data and citation Every view on the site has a machine-readable twin: [llms.txt](llms.txt) for agents and scripts; [llms-full.txt](llms-full.txt) with the ranking, method and every agent page; [method.md](method.md) with every rule quoted from the build file; the full build as JSON, linked from llms.txt; and [rankings.csv](data/rankings.csv), [criteria.csv](data/criteria.csv) and [requests.csv](data/requests.csv). In the data files, popularity is `reach`, customer love is `regard` and user-weeks are `authorWeeks`. Scores change with each build, so cite the version and the build date shown in the technical reference: > Feedback Bench 1.1, Coding agents, built YYYY-MM-DD, https://feedbackbench.com/coding-agents/. Built by [Enterpret](https://www.enterpret.com). ## References - Altman, D. G. and Bland, J. M. (1997). Statistics notes: Units of analysis. *BMJ* 314(7098), 1874. - Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. *American Psychologist* 34(7), 571–582. - Decancq, K. and Lugo, M. A. (2013). Weights in multidimensional indices of wellbeing: An overview. *Econometric Reviews* 32(1), 7–34. - Efron, B. and Morris, C. (1975). Data analysis using Stein's estimator and its generalizations. *Journal of the American Statistical Association* 70(350), 311–319. - Efron, B. and Tibshirani, R. J. (1993). *An Introduction to the Bootstrap*. Chapman and Hall. - Field, C. A. and Welsh, A. H. (2007). Bootstrapping clustered data. *Journal of the Royal Statistical Society, Series B* 69(3), 369–390. - Fleiss, J. L., Levin, B. and Paik, M. C. (2003). *Statistical Methods for Rates and Proportions*, 3rd ed. Wiley. - G2 (2026). Research scoring methodologies. - Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. and Rubin, D. B. (2013). *Bayesian Data Analysis*, 3rd ed. CRC Press. - Goldstein, H. and Spiegelhalter, D. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. *Journal of the Royal Statistical Society, Series A* 159(3), 385–443. - Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. *Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 168–177. - Hu, N., Pavlou, P. A. and Zhang, J. (2017). On self-selection biases in online product reviews. *MIS Quarterly* 41(2), 449–471. - Kendall, M. G. (1938). A new measure of rank correlation. *Biometrika* 30(1–2), 81–93. - Klugman, J., Rodríguez, F. and Choi, H.-J. (2011). The HDI 2010: New controversies, old critiques. *Journal of Economic Inequality* 9(2), 249–288. - Koren, Y. (2010). Collaborative filtering with temporal dynamics. *Communications of the ACM* 53(4), 89–97. - Li, X. and Hitt, L. M. (2008). Self-selection and information role of online product reviews. *Information Systems Research* 19(4), 456–474. - Luca, M. and Zervas, G. (2016). Fake it till you make it: Reputation, competition, and Yelp review fraud. *Management Science* 62(12), 3412–3427. - Maalej, W. and Nabil, H. (2015). Bug report, feature request, or simply praise? On automatically classifying app reviews. *2015 IEEE 23rd International Requirements Engineering Conference*, 116–125. - MacQueen, K. M., McLellan, E., Kay, K. and Milstein, B. (1998). Codebook development for team-based qualitative analysis. *CAM Journal* 10(2), 31–36. - Mayzlin, D., Dover, Y. and Chevalier, J. (2014). Promotional reviews: An empirical investigation of online review manipulation. *American Economic Review* 104(8), 2421–2455. - Morstatter, F., Pfeffer, J., Liu, H. and Carley, K. M. (2013). Is the sample good enough? Comparing data from Twitter's Streaming API with Twitter's Firehose. *Proceedings of the International AAAI Conference on Web and Social Media* 7(1), 400–408. - OECD and JRC (2008). *Handbook on Constructing Composite Indicators: Methodology and User Guide*. OECD Publishing. - Paruolo, P., Saisana, M. and Saltelli, A. (2013). Ratings and rankings: Voodoo or science? *Journal of the Royal Statistical Society, Series A* 176(3), 609–634. - Ruths, D. and Pfeffer, J. (2014). Social media for large studies of behavior. *Science* 346(6213), 1063–1064. - Saisana, M., Saltelli, A. and Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. *Journal of the Royal Statistical Society, Series A* 168(2), 307–323. - Schenker, N. and Gentleman, J. F. (2001). On judging the significance of differences by examining the overlap between confidence intervals. *The American Statistician* 55(3), 182–186. - Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z. and Yang, D. (2024). Can large language models transform computational social science? *Computational Linguistics* 50(1), 237–291. ## Earlier version Feedback Bench 1.0, the version published until 16 October 2026, used the same posts, labels and bootstrap. It combined the two measures as 100 × √(popularity × customer love), with popularity = log(1 + share / 0.5%) / log(1 + leader's share / 0.5%). Its customer love counted every stance and compared each agent with channel norms built from all agents, its own users included. --- ## Appendix A: formulas For agent `a`, criterion or area `c`: ``` User-week value v(u, w, c) = Σ over u's first-hand judging posts in week w of (+1 praise, −1 complaint, 0 mixed) on c positive if v > 0, negative if v < 0, not rated if v = 0 (a criterion's value also adds to its area's value) Recency weight r(w) = 0.5 ^ (age(w) / h) h = 4 weeks; age(w) = weeks from w to the latest full week pos and neg below sum r(w) over positive and negative user-weeks; n = pos + neg Share of voice S(a) = Σ_u r(latest week of u for a) / Σ_b Σ_u r(latest week of u for b) u = a user who counts for the agent Popularity R(a) = (log S(a) − log S_min) / (log S_max − log S_min) Peer norm π(a, c) = (pos(c) − pos(a, c) + 1) / (n(c) − n(a, c) + 2) all other agents, all channels Shrunk share p(a, c) = (pos(a, c) + m · π(a, c)) / (pos(a, c) + neg(a, c) + m) Lift L(a, c) = logit p(a, c) − logit π(a, c) Love on c sigmoid(L(a, c)) Area weight w(c) = n(c) / Σ_c' n(c') n = weighted rated user-weeks; over the areas, all agents Customer love G(a) = sigmoid( Σ_area w(area) · L(a, area) ) z-score z(x)(a) = (x(a) − mean of x over agents) / standard deviation of x over agents (divided by the number of agents) Feedback Score F(a) = 50 + 10 · z( ½ · z(log S)(a) + ½ · z(logit G)(a) ) Bootstrap for b in 1..1000: resample each agent's users with replacement (all their weeks together), recompute π, w, L, G, the z-scores and F with S fixed; rank the agents by F Intervals 2.5th and 97.5th percentiles of G, love on c and F across resamples; rank range: the order statistics at the 2.5th and 97.5th percentiles of rank Positions agents ordered by F; rank ranges are reported in the standings table and the data files ``` ## Appendix B: parameters | Parameter | Value | Reason | |---|---|---| | Window | From 2026-08-31 to the latest full Monday-to-Sunday week | Measurement began on 2026-08-31. Every full week since then counts. | | Unit | User-week, per platform | One account counts once per week. | | Excluded accounts | Reddit AutoModerator, deleted accounts, names ending in "bot"; reviewed accounts marked as vendor staff, automated or promotion; posts that share a referral, invitation or redeem code | Removes automated, unattributable and promotional posts, and posts by the vendor's own staff. | | Author review | 40 or more first-hand ratings of one agent, the same text 3 or more times, or 2 or more posts in a product team's voice | Heavy and repeated posters are checked before they count. | | Recency half-life | 4 weeks | Recent weeks decide most of the rank, while every week counts. | | New-agent embargo | 1 full week after launch | A launch week draws early enthusiasts and launch offers; a new agent is ranked only once a full week has passed. | | m, shrinkage strength | 200 weighted user-weeks | An agent's own evidence carries half the weight at 200 weighted rated user-weeks. | | Smoothing of norms | One praise and one complaint added | No norm of exactly 0% or 100%. | | Area weights | Each area's share of all weighted area ratings | The same for every agent. | | Combination | Equal weights on the z-scores of log share of voice and logit customer love | Section 7. | | Score scale | Mean 50, standard deviation 10 | 50 is an agent that is average on both measures. | | Bootstrap | 1,000 resamples of users, seed 20260927 | The fixed seed makes intervals reproduce exactly. | | Minimum for a reading | 30 rated user-weeks, unweighted | Below it the site shows "too few posts". | ## Technical reference (built 2026-10-19) Each rule below is quoted verbatim from the build file. Names: popularity is `reach` and customer love is `regard` in the data files (coding.json, rankings.csv, criteria.csv). ### Sources 2026-08-31 to the Sunday before 2026-10-19. Four channels, collected the same way for every agent: (1) the agent's official subreddits, posts and comments; (2) Reddit posts and comments whose own text names the agent, from any subreddit we collect and from a Reddit search per agent; (3) posts on X that mention the agent's official handles (for OpenAI Codex, which has no product handle, an X search for "OpenAI Codex", "Codex CLI" and "Codex app"); (4) G2 and Trustpilot reviews of the product. Parent-brand review pages and staff handles are excluded. Each post counts once per agent. ### Attribution A post in an agent's own subreddit counts for that agent. Any other post counts for an agent only when the classifier confirms it is about that agent; a post can count for several agents. ### Labels Every post is labelled by a large language model with the criteria codebook v1.0 (63 criteria in 9 areas): whether it is about the agent, whether it judges it, which criteria its stance names, and praise, complaint or mixed for each. On a pilot of 308 posts, a second large language model labelling independently agreed at kappa 0.75 on whether a post judges the agent, 0.73 overlap on its criteria, 0.78 on its areas, and 98% on polarity. Human annotators have not yet validated v1.0. 3 post(s) the classifier declined to label are left out. ### Unit A user-week: one user on one platform, about one agent, in one week. For each criterion it is positive if that user's praise outweighs their complaints that week, negative if the reverse. On Reddit, AutoModerator, [deleted] and names ending in 'bot' are not counted. Each review counts as its own user when the platform gives no name. ### Recency Every week in the window counts, and recent weeks count more. Each user-week is weighted 0.5^(age / 4), where age is the number of weeks before the latest full week (the week of 2026-10-12): that week weighs 1, 4 weeks earlier 0.5, 8 weeks earlier 0.25. Customer love, category baselines and area weights use weighted user-weeks; popularity counts each user with the weight of their most recent week. Counts shown (users, user-weeks, praise, complaint) are not weighted. Weekly scores score each week on its own, without weights. ### Popularity Share of voice = the agent's users across all four channels, each weighted by recency, / the sum over all agents. Popularity places share of voice on a log scale: (log share − log of the smallest share) / (log of the largest share − log of the smallest share), so the least-discussed agent is 0 and the most-discussed is 1. Popularity counts every user in the window, so it has no sampling interval. ### Criterion love For each criterion: the agent's peer norm is (positive + 1) / (positive + negative + 2) recency-weighted user-weeks of all other agents, across all channels. The agent's positive share is shrunk toward its peer norm with 200 imaginary weighted user-weeks and compared with it on the log-odds scale. Only first-hand stances count: a separate model pass marks whether each judging post speaks from the author's own use, and a stance that does not, or that the pass refused, is not a rating. 0.5 is the category norm. ### Customer love Customer love = the customer love scores of the 9 areas, averaged on the log-odds scale with weights equal to each area's share of the category's recency-weighted rated user-weeks. ### Feedback Score Feedback Bench 1.1: popularity and customer love count equally. Each is standardised across the agents in the category (z-score of log share of voice; z-score of customer love on the log-odds scale), the two z-scores are averaged, and the average is restated as a standard score: 50 is the average agent and 10 points is one standard deviation. ### Intervals 95% intervals and rank ranges from 1000 bootstrap resamples of users. Popularity is held fixed; customer love and the standardisation are recomputed in each resample. ### Eligibility An agent is ranked once 1 full week (Monday to Sunday) after its launch is in the window. Until then it is not shown, and its posts count toward no score, norm or weight, so its arrival moves no other agent. Posts written before an agent's launch do not count, and its weekly scores start in the week it is first ranked. Shrinkage keeps the customer love of agents with few rated user-weeks near the norm. ### Receipts Up to 5 praise and 5 complaint posts per area and 3 per criterion. Each is a whole post of 60 to 450 characters (700 for areas) with at least 8 words beyond handles and links, that judges the criterion, whose labels are all praise or all complaint, and that contains no slur. Posts that touch the fewest criteria come first, then posts of at most 280 characters, then the newest. A second large language model reads the first 3 candidates per slot and keeps a post only if it clearly praises, or clearly complains about, the agent on that criterion. Links and personal details removed by the source show as marks such as [link]. ### Caveat A subreddit is a proxy for the product. X volume is capped at 1,000 posts a day per feed; no feed in the window reached it on its median day. Labels are automatic. Criteria seen in few posts have wide intervals; those under 30 user-weeks for an agent are shown as too few posts. ### Authors An author is reviewed when, for one agent, they post 40 or more first-hand ratings in the window, post the same text 3 or more times, or write 2 or more posts in a product team's voice ("we shipped", "our users"). A large language model reads a sample of their posts and marks the author as a user, vendor staff, an automated account, or promotion (referral codes, ads for the author's own product, copy-pasted praise). Only users count: staff and promotion posts are left out for that agent, and an automated account is left out for every agent. An author whose posts the model declines to review is left out for that agent, as declined labels are. A post that shares a referral, invitation or redeem code is promotion and is left out, whoever wrote it. ### Levers For each area where an agent's customer love is below the category norm: the Feedback Score and rank it would have if that one area were at the norm (0.5) and everything else stayed the same, computed with the same formula as the ranking. An area's pull follows its weight, so an area many users of the category rate moves the score more than one they rarely mention. ### Head to head Posts that judge two agents. In each post, the agent whose stance (praise minus complaint across all its labels, the overall stance included) is higher is ahead; ties are left out. Shares count the posts where one is ahead, with a 95% Wilson interval. Where: the agent's own subreddit, the other's, or anywhere else. Pairs with at least 30 such posts are shown. ### Failure modes A separate pass sorts complaints into failure modes. For each criterion with at least 30 complaining user-weeks, a large language model reads 150 complaint posts drawn at random and names 4 to 8 failure modes; a second large language model then puts every complaint post on that criterion in one mode, or in Other when it fits none. Modes are fixed once named, and new posts are sorted into them each week. A mode counts distinct user-weeks with a complaint in it, per agent; across agents, one user-week per agent. Modes do not change the labels or the Feedback Score. Examples: up to 3 posts per mode from different users, chosen and checked as receipts are. ### Requests A separate pass by a large language model reads every counted post and extracts what the user asks the agent or its vendor to add or change, with the criteria it maps to and a short normalised wording; it does not touch the labels or the Feedback Score. A large language model groups the wordings within each criterion (the first criterion the request maps to) into themes, and a second pass merges themes that ask for the same change under different criteria; a merged theme sits under the criterion most of its requests map to, and code counts it once. A theme counts distinct user-weeks that ask for it, per agent; across agents, one user-week per agent. Themes asked in fewer than 2 user-weeks, and requests that share no theme, are not shown. Examples: up to 3 posts per theme from different users, without slurs: whole posts of 60 to 450 characters with at least 8 words beyond handles and links, posts of at most 280 characters first, then the newest. ### Display rules - Reading of a criterion or area: its position. Position on a criterion: its place among the agents with at least 30 rated user-weeks there, ordered by customer love. It ranks high when both its position and its 95% interval put it in the top half, low when both put it in the bottom half; neither otherwise, or when fewer than 4 agents qualify. Too few posts under 30 rated user-weeks. - Agent place: its position on Popularity and on Customer love among all 17 agents. The Feedback Score weighs the two equally. - Top quadrant: popularity of at least 0.5 and customer love of at least 0.5; at the edge when the customer love 95% interval still includes 0.5. - Rank is the plain position by Feedback Score. How sure each position is: the rank range, reported in the method and the data files. ### Parameters | Parameter | Value | |---|---| | scoreMean | 50 | | scoreSd | 10 | | priorStrength | 200 | | halfLifeWeeks | 4 | | embargoWeeks | 1 | | bootstrap | 1000 | | seed | 20260927 | ### Area weights in Customer love Each area's share of the category's rated user-weeks. | Area | Weight | |---|---| | Paying and limits | 0.306 | | Setting up and connecting | 0.052 | | Choosing models | 0.104 | | Instructing and context | 0.062 | | Doing the work | 0.247 | | Checking and finishing | 0.024 | | Interface and sessions | 0.065 | | Reliability and speed | 0.106 | | Account and support | 0.034 | ### Category baselines (positive share of user-weeks, %) | Code | Name | Baseline % | |---|---|---| | paying | Paying and limits | 25.6 | | setup | Setting up and connecting | 39.4 | | models | Choosing models | 26.4 | | context | Instructing and context | 36.5 | | work | Doing the work | 51.4 | | checking | Checking and finishing | 50.1 | | interface | Interface and sessions | 40.4 | | reliability | Reliability and speed | 16.3 | | account | Account and support | 10.8 | | limits.plan_value | How much use a plan's price buys | 40.2 | | limits.window_interrupts_work | Short rolling usage window blocks or interrupts work | 14.2 | | limits.burn_rate | Single prompt, model or effort level consumes disproportionate quota | 18.4 | | limits.allowance_change | Price, allowance or plan terms changed | 4.8 | | limits.reset_schedule | Quota reset timing and bonus or banked resets | 17.9 | | limits.usage_meter | Usage meter visibility and accuracy | 8.5 | | limits.prompt_cache | Prompt cache hits, misses and invalidation | 29.1 | | billing.overage_charges | Pay-as-you-go overage, fallback billing and spend caps | 5.7 | | billing.pricing_clarity | Pricing and plan terms stated clearly and consistently | 4.7 | | billing.free_tier | Free tier and free model availability and limits | 58.9 | | billing.subscription_portability | Using an existing subscription across tools | 41.0 | | setup.install_signin | Install, launch and sign-in | 18.5 | | setup.provider_byok_local | Connecting own API keys, local models and custom endpoints | 51.0 | | setup.extensions_mcp | MCP servers, plugins, skills and hooks | 49.9 | | setup.onboarding_docs | Onboarding, discoverability and documentation | 20.7 | | setup.ide_integration | IDE and editor integration | 50.1 | | models.catalog_access | Which models are offered on a plan and when | 24.2 | | models.routing_auto | Automatic model routing and fallback | 25.2 | | models.effort_control | Reasoning effort setting and its defaults | 47.2 | | models.quality_drift | Quality got worse or better over time | 24.5 | | context.instruction_files | Persistent project rules files are read and obeyed | 54.2 | | context.instruction_following | Direct in-prompt instructions and caps are followed | 26.0 | | context.clarifying_questions | Asks the user versus guessing | 39.9 | | context.long_context_decay | Output degrades as the context window fills | 17.4 | | context.compaction | Context compaction keeps what matters, cheaply and quickly | 36.8 | | context.session_memory | Memory and state carried across sessions | 44.6 | | context.codebase_retrieval | Finding the right files in the codebase | 44.6 | | context.attachments | Images, PDFs and file attachments as input | 36.3 | | work.capability | Can do the user's kind of task | 64.5 | | work.frontend_ui | Frontend and visual UI output | 44.5 | | work.bug_diagnosis | Diagnosing and fixing reported bugs | 68.1 | | work.regressions_introduced | Breaks existing code or reintroduces bugs | 6.8 | | work.scope_overreach | Does unrequested work or over-engineers | 5.3 | | work.stuck_loops | Spins, loops or gets stuck without progress | 4.3 | | work.premature_stop | Stops mid-task or answers instead of acting | 7.9 | | work.long_running_autonomy | Long unattended runs and goal/loop mode | 77.4 | | work.multi_agent_orchestration | Subagents, parallel agents and orchestrators | 64.8 | | work.reward_hacking | Games checks instead of fixing the problem | 2.2 | | work.destructive_actions | Risky or irreversible actions without confirmation | 22.3 | | work.git_workflow | Git commits, branches and sync | 34.5 | | work.computer_browser_use | Computer use and browser control | 52.9 | | work.safety_refusals | Safety filters block legitimate coding tasks | 15.3 | | work.permission_prompts | Tool approval prompts and autonomy modes | 23.0 | | work.plan_mode | Plan-before-edit mode | 60.2 | | work.response_verbosity | Length and clarity of replies, summaries and comments | 25.1 | | work.sycophancy_pushback | Caves to or argues with the user's judgement | 19.5 | | verify.false_completion | Claims work is done or fixed when it is not | 3.7 | | verify.self_testing | Builds, tests or runs its own changes | 62.9 | | verify.agent_code_review | Agent-performed code review finds real issues | 78.2 | | verify.change_review_ui | Reviewing and approving the agent's changes | 42.8 | | ui.display_settings | How the interface shows work, and what the user can configure | 34.3 | | ui.session_history | Saving, switching, resuming and rewinding sessions | 26.5 | | ui.interrupt_steer | Stopping and steering a running agent | 42.7 | | surfaces.remote_mobile | Mobile, remote-control and voice access | 56.6 | | surfaces.cloud_sessions | Cloud and remote sandbox execution | 55.4 | | rel.service_errors | Outages, server errors and capacity or rate errors | 6.8 | | rel.response_speed | Latency, throughput and fast mode | 26.7 | | rel.client_failures | Client crashes, freezes and failed tool execution | 7.2 | | rel.update_breakage | Updates break working setups | 11.8 | | account.support | Support, refunds and issue handling | 12.9 | | account.billing_errors | Wrong charges, failed payments and plan provisioning | 1.6 | | account.bans_restrictions | Account bans and access restrictions | 8.3 | | account.data_privacy | Data retention, training use and deployment isolation | 18.4 | ### Criteria 63 criteria in 9 areas. Codebook: every criterion names one product mechanism, visible in a single post. #### Paying and limits (`paying`) What do you pay, and how far does it get you? - [`limits.plan_value`](https://feedbackbench.com/coding-agents/criteria/limits.plan_value.md) **How much use a plan's price buys.** Whether a plan's weekly or monthly allowance covers the user's normal work, and how its price compares with other plans or agents for the use it gives. Covers running out days early, never hitting the cap, missing tiers and value per dollar. Boundary: Not this: see limits.window_interrupts_work when the short multi-hour window blocks work. Not this: see limits.burn_rate when one prompt, model or effort level drained the quota. Not this: see limits.allowance_change when price or allowance changed over time. Not this: see billing.pricing_clarity when the information itself is unclear. - [`limits.window_interrupts_work`](https://feedbackbench.com/coding-agents/criteria/limits.window_interrupts_work.md) **Short rolling usage window blocks or interrupts work.** The short rolling usage window (a cap of a few hours) blocks the user mid-task or mid-session: forced waits, killed sessions, or a long allowance left unused because the short window ran out. The post is about the window itself, not about what consumed it. Boundary: Not this: see limits.burn_rate when the point is that one prompt, model or effort level used a large share. Not this: see limits.plan_value for the weekly or monthly total. - [`limits.burn_rate`](https://feedbackbench.com/coding-agents/criteria/limits.burn_rate.md) **Single prompt, model or effort level consumes disproportionate quota.** How much quota a specific prompt, task, model version, effort level or feature consumes for the work done. Includes new versions using more than earlier ones and praise for efficient ones. The post names what consumed the quota. Boundary: Not this: see limits.window_interrupts_work when the post is about being blocked by the short window, not about the consumer. Not this: see limits.prompt_cache for cache misses. Not this: see work.multi_agent_orchestration when runaway subagents cause the burn. - [`limits.allowance_change`](https://feedbackbench.com/coding-agents/criteria/limits.allowance_change.md) **Price, allowance or plan terms changed.** The vendor changes a plan's price, quota, multipliers, included models or tiers over time: cuts, raises, removals, silent changes and changes that differ from what was announced. Boundary: Not this: see limits.plan_value for the current level with no change named. Not this: see limits.reset_schedule for when resets happen. - [`limits.reset_schedule`](https://feedbackbench.com/coding-agents/criteria/limits.reset_schedule.md) **Quota reset timing and bonus or banked resets.** When and how usage resets happen: scheduled reset dates, surprise or bonus resets, banked resets, and paid resets. Covers resets that wipe saved allowance or land uselessly close to the normal reset. Boundary: Not this: see limits.allowance_change for changes to quota size. Not this: see limits.usage_meter when the reset is only displayed wrongly. - [`limits.usage_meter`](https://feedbackbench.com/coding-agents/criteria/limits.usage_meter.md) **Usage meter visibility and accuracy.** Whether the product shows how much quota and tokens were used and how much remains, per task and per model, and whether that figure is accurate. Covers meters that climb while idle and hidden token counts. Boundary: Not this: see billing.overage_charges for money actually charged. Not this: see limits.burn_rate when the meter is believed and the complaint is the amount consumed. - [`limits.prompt_cache`](https://feedbackbench.com/coding-agents/criteria/limits.prompt_cache.md) **Prompt cache hits, misses and invalidation.** Whether prompt caching is kept across turns, resumes and provider routing, and how cache reads are priced. Covers cache drops that inflate usage. Boundary: Not this: see limits.burn_rate for consumption not attributed to caching. - [`billing.overage_charges`](https://feedbackbench.com/coding-agents/criteria/billing.overage_charges.md) **Pay-as-you-go overage, fallback billing and spend caps.** Usage spills from the plan into metered, on-demand or API billing, and whether spend caps and alerts work. Covers overage turning on silently and credit top-ups. Boundary: Not this: see account.billing_errors for wrong charges on the subscription itself. Not this: see models.routing_auto when the cause is auto-selection of a pricier model. - [`billing.pricing_clarity`](https://feedbackbench.com/coding-agents/criteria/billing.pricing_clarity.md) **Pricing and plan terms stated clearly and consistently.** Whether pricing pages, docs and plan descriptions clearly and consistently state costs, limits and what 'unlimited' means. Boundary: Not this: see limits.allowance_change when terms actually changed. Not this: see limits.usage_meter for in-product usage display. - [`billing.free_tier`](https://feedbackbench.com/coding-agents/criteria/billing.free_tier.md) **Free tier and free model availability and limits.** Whether free models, free tiers and trial or promo access exist, how long they stay available, and whether their limits are usable. Boundary: Not this: see models.catalog_access for paid model availability. Not this: see rel.service_errors when free models return errors. - [`billing.subscription_portability`](https://feedbackbench.com/coding-agents/criteria/billing.subscription_portability.md) **Using an existing subscription across tools.** Whether a paid subscription from one vendor can be used inside another agent or harness, or used as an API outside the vendor's own app. Boundary: Not this: see setup.provider_byok_local for API keys, local models and custom endpoints. #### Setting up and connecting (`setup`) How hard is it to install and connect? - [`setup.install_signin`](https://feedbackbench.com/coding-agents/criteria/setup.install_signin.md) **Install, launch and sign-in.** Getting the agent installed, started and signed in on the user's platform: installers, updates on first run, login flows, browser redirects and switching accounts. Boundary: Not this: see setup.provider_byok_local for connecting own keys or local models. Not this: see rel.update_breakage when a later update breaks a working setup. - [`setup.provider_byok_local`](https://feedbackbench.com/coding-agents/criteria/setup.provider_byok_local.md) **Connecting own API keys, local models and custom endpoints.** Whether user-supplied providers work: bring-your-own-key, OpenAI-compatible endpoints, OpenRouter, and local servers such as Ollama or LM Studio. Boundary: Not this: see billing.subscription_portability for reusing a paid subscription. Not this: see rel.tool_call_errors for tool-format failures once connected. - [`setup.extensions_mcp`](https://feedbackbench.com/coding-agents/criteria/setup.extensions_mcp.md) **MCP servers, plugins, skills and hooks.** Whether external tools and extension points load, authenticate and refresh correctly: MCP servers, plugins, skills, hooks and third-party integrations. Boundary: Not this: see setup.ide_integration for editor extensions. Not this: see context.instruction_files for rules files. - [`setup.onboarding_docs`](https://feedbackbench.com/coding-agents/criteria/setup.onboarding_docs.md) **Onboarding, discoverability and documentation.** How easily a new user learns the product: quick-start guides, docs explaining features and modes, and how findable features are. Boundary: Not this: see billing.pricing_clarity for pricing docs. Not this: see ui.customization for settings complexity. - [`setup.ide_integration`](https://feedbackbench.com/coding-agents/criteria/setup.ide_integration.md) **IDE and editor integration.** How the agent works inside an editor or IDE, including extension support, editor-native features, autocomplete and feature parity across IDEs. Boundary: Not this: see verify.change_review_ui for diff and approval views. Not this: see ui.terminal_display for CLI rendering. #### Choosing models (`models`) Which models do you get, and do they hold up? - [`models.catalog_access`](https://feedbackbench.com/coding-agents/criteria/models.catalog_access.md) **Which models are offered on a plan and when.** Whether models are available on the user's plan: same-day support for new releases, deprecations and removals, multi-provider choice, and models missing from a surface. Boundary: Not this: see billing.free_tier for free-model availability. Not this: see models.routing_auto for which model actually runs. - [`models.routing_auto`](https://feedbackbench.com/coding-agents/criteria/models.routing_auto.md) **Automatic model routing and fallback.** Auto mode, routers or fallbacks pick, switch or hide the model used. This includes subagents running on a model other than the one requested, silent downgrades, and inability to exclude models. Boundary: Not this: see models.effort_control for reasoning level. Not this: see billing.overage_charges for the resulting charge alone. - [`models.effort_control`](https://feedbackbench.com/coding-agents/criteria/models.effort_control.md) **Reasoning effort setting and its defaults.** How the reasoning or effort level can be set, whether it stays set, and how outcomes differ by level, such as overthinking at high effort or failing at low effort. Boundary: Not this: see limits.burn_rate when the point is the quota share an effort level consumed. Not this: see work.scope_overreach for over-engineered output. - [`models.quality_drift`](https://feedbackbench.com/coding-agents/criteria/models.quality_drift.md) **Quality got worse or better over time.** The post compares the agent or one of its models with an earlier time and says it got worse or better: 'nerfed', 'dumber since last week', 'better than at launch'. An explicit comparison with the past is required. Boundary: Not this: see general.unspecific for a verdict with no comparison over time. Not this: see work.capability for what it can do now. Not this: see limits.allowance_change for changed quotas or prices. #### Instructing and context (`context`) Does it follow your instructions and keep the right context? - [`context.instruction_files`](https://feedbackbench.com/coding-agents/criteria/context.instruction_files.md) **Persistent project rules files are read and obeyed.** Whether the agent reads and follows persistent project instruction files (rules or agent markdown files) across turns. Boundary: Not this: see context.instruction_following for instructions in the current prompt. Not this: see context.session_memory for auto-written memory. - [`context.instruction_following`](https://feedbackbench.com/coding-agents/criteria/context.instruction_following.md) **Direct in-prompt instructions and caps are followed.** Whether the agent executes explicit instructions, prohibitions and budgets given in the prompt, such as iteration caps and token caps. Boundary: Not this: see context.instruction_files for rules files. Not this: see work.sycophancy_pushback for evaluating the user's claims or opinions. Not this: see work.scope_overreach for unrequested extras. - [`context.clarifying_questions`](https://feedbackbench.com/coding-agents/criteria/context.clarifying_questions.md) **Asks the user versus guessing.** Whether the agent stops to ask clarifying questions when it is blocked or the request is ambiguous, instead of inventing assumptions or workarounds. Boundary: Not this: see work.stuck_loops for repetitive failure without any question. - [`context.long_context_decay`](https://feedbackbench.com/coding-agents/criteria/context.long_context_decay.md) **Output degrades as the context window fills.** The agent forgets details, rules or its own statements as the session grows long. Boundary: Not this: see context.compaction for losses caused by summarisation. Not this: see context.session_memory for loss across separate sessions. - [`context.compaction`](https://feedbackbench.com/coding-agents/criteria/context.compaction.md) **Context compaction keeps what matters, cheaply and quickly.** How automatic or manual compaction summarises history: what it drops, how long it takes, what it costs, and whether it is visible. Boundary: Not this: see context.long_context_decay for degradation without compaction. - [`context.session_memory`](https://feedbackbench.com/coding-agents/criteria/context.session_memory.md) **Memory and state carried across sessions.** Whether knowledge persists correctly between sessions: memory files, re-discovery cost at session start, stale memories, and leakage between sessions. Boundary: Not this: see ui.session_history for saving and resuming chat transcripts. Not this: see context.instruction_files for user-written rules. - [`context.codebase_retrieval`](https://feedbackbench.com/coding-agents/criteria/context.codebase_retrieval.md) **Finding the right files in the codebase.** How the agent searches and indexes a repository and pulls in relevant files. Covers over-reading on trivial edits and missing files in large or multi-repo codebases. Boundary: Not this: see limits.burn_rate for consumption not tied to file reading. - [`context.attachments`](https://feedbackbench.com/coding-agents/criteria/context.attachments.md) **Images, PDFs and file attachments as input.** Whether attached images, screenshots, PDFs and other files are accepted and read. Boundary: Not this: see work.computer_browser_use for screen control. #### Doing the work (`work`) How does it behave while it works? - [`work.capability`](https://feedbackbench.com/coding-agents/criteria/work.capability.md) **Can do the user's kind of task.** Whether the agent, with its models, succeeds at the user's kind of task: its size, complexity, language or domain, including building a whole feature in one go or failing at it. The post names the task or its size but no narrower failure behaviour. Boundary: Not this: see work.frontend_ui for visual UI work and work.bug_diagnosis for fixing a reported bug. Not this: see models.quality_drift for a change over time. Not this: see the other work.* and verify.* leaves when a specific behaviour (looping, scope, false claims, breaking code) is named. Not this: see general.unspecific when no task is named. - [`work.frontend_ui`](https://feedbackbench.com/coding-agents/criteria/work.frontend_ui.md) **Frontend and visual UI output.** How the agent handles UI design, layout, visual taste and front-end component wiring. Boundary: Not this: see work.regressions_introduced for UI bugs reintroduced after fixes. - [`work.bug_diagnosis`](https://feedbackbench.com/coding-agents/criteria/work.bug_diagnosis.md) **Diagnosing and fixing reported bugs.** Whether the agent finds the root cause of a failing behaviour and fixes it without step-by-step guidance. Boundary: Not this: see verify.agent_code_review for reviewing code to find unknown issues. - [`work.regressions_introduced`](https://feedbackbench.com/coding-agents/criteria/work.regressions_introduced.md) **Breaks existing code or reintroduces bugs.** Edits break previously working code, reintroduce fixed bugs, or cycle between introducing and fixing bugs. Boundary: Not this: see work.stuck_loops for repetition without code damage. Not this: see work.reward_hacking for deliberately gaming checks. - [`work.scope_overreach`](https://feedbackbench.com/coding-agents/criteria/work.scope_overreach.md) **Does unrequested work or over-engineers.** The agent edits outside the requested scope, adds unasked features, tests or runs, or builds overly complex solutions. Boundary: Not this: see work.destructive_actions for risky irreversible actions. Not this: see work.response_verbosity for padded text or comments. - [`work.stuck_loops`](https://feedbackbench.com/coding-agents/criteria/work.stuck_loops.md) **Spins, loops or gets stuck without progress.** The agent repeats attempts, loops endlessly or spirals on a problem without converging. Boundary: Not this: see work.premature_stop for stopping early. Not this: see context.clarifying_questions when the fix was to ask the user. - [`work.premature_stop`](https://feedbackbench.com/coding-agents/criteria/work.premature_stop.md) **Stops mid-task or answers instead of acting.** The agent halts before finishing, or replies with explanation or a summary instead of performing the action. Boundary: Not this: see verify.false_completion for stopping while claiming the work is done. Not this: see limits.window_interrupts_work for stops caused by quota. - [`work.long_running_autonomy`](https://feedbackbench.com/coding-agents/criteria/work.long_running_autonomy.md) **Long unattended runs and goal/loop mode.** Whether the agent sustains hours-long autonomous work toward a goal, and whether a goal or loop mode exists. Boundary: Not this: see work.multi_agent_orchestration for coordination between agents. Not this: see surfaces.cloud_sessions for where the run executes. - [`work.multi_agent_orchestration`](https://feedbackbench.com/coding-agents/criteria/work.multi_agent_orchestration.md) **Subagents, parallel agents and orchestrators.** How the agent spawns, delegates to, monitors and coordinates subagents or parallel sessions. Covers polling loops and the wrong choice of subagents. Boundary: Not this: see models.routing_auto for the model a subagent runs on. Not this: see work.git_workflow for merge conflicts between sessions. - [`work.reward_hacking`](https://feedbackbench.com/coding-agents/criteria/work.reward_hacking.md) **Games checks instead of fixing the problem.** The agent disables linters, hardcodes for tests, edits benchmarks or defends bugs with tests so that checks pass without a real fix. Boundary: Not this: see verify.false_completion for plain false claims without manipulated checks. - [`work.destructive_actions`](https://feedbackbench.com/coding-agents/criteria/work.destructive_actions.md) **Risky or irreversible actions without confirmation.** The agent pushes, merges, deletes or wipes changes, or acts on the host outside its sandbox, without asking. Also covers praise for staying in bounds. Boundary: Not this: see work.permission_prompts for approval prompt frequency. Not this: see work.scope_overreach for harmless extra work. - [`work.git_workflow`](https://feedbackbench.com/coding-agents/criteria/work.git_workflow.md) **Git commits, branches and sync.** How the agent handles commits, attribution, branches, pulls and pushes, and merge conflicts between concurrent agents or teammates. Boundary: Not this: see work.destructive_actions for data-destroying git commands. - [`work.computer_browser_use`](https://feedbackbench.com/coding-agents/criteria/work.computer_browser_use.md) **Computer use and browser control.** Whether the agent operates desktop apps and browsers well, without taking over the user's windows. Boundary: Not this: see context.attachments for reading screenshots. - [`work.safety_refusals`](https://feedbackbench.com/coding-agents/criteria/work.safety_refusals.md) **Safety filters block legitimate coding tasks.** Refusals or safety classifiers stop benign or security-remediation work. Boundary: Not this: see work.permission_prompts for tool approval prompts. Not this: see models.routing_auto for safety-triggered model downgrades. - [`work.permission_prompts`](https://feedbackbench.com/coding-agents/criteria/work.permission_prompts.md) **Tool approval prompts and autonomy modes.** How often and when the agent asks permission to run tools or commands, and whether auto or bypass modes behave as configured. Boundary: Not this: see work.destructive_actions for acting without permission. Not this: see work.safety_refusals for content blocks. - [`work.plan_mode`](https://feedbackbench.com/coding-agents/criteria/work.plan_mode.md) **Plan-before-edit mode.** Whether a read-only planning phase exists, produces useful plans, and actually prevents edits. Boundary: Not this: see work.task_completion for how well an approved plan is executed. - [`work.response_verbosity`](https://feedbackbench.com/coding-agents/criteria/work.response_verbosity.md) **Length and clarity of replies, summaries and comments.** How long and how readable the agent's prose and summaries are, and how many code comments and doc notes it writes. Boundary: Not this: see verify.change_review_ui for the volume of code changes. Not this: see work.scope_overreach for extra code features. - [`work.sycophancy_pushback`](https://feedbackbench.com/coding-agents/criteria/work.sycophancy_pushback.md) **Caves to or argues with the user's judgement.** How the agent handles disagreement: agreeing with everything, reversing correct findings when challenged, failing to push back on wrong claims, or arguing when corrected. Boundary: Not this: see context.instruction_following for refusing a clear instruction. #### Checking and finishing (`checking`) Can you trust that the work is done? - [`verify.false_completion`](https://feedbackbench.com/coding-agents/criteria/verify.false_completion.md) **Claims work is done or fixed when it is not.** The agent reports success, completion or a finished todo list that turns out to be untrue. Boundary: Not this: see work.reward_hacking when checks were manipulated. Not this: see verify.self_testing for whether checks were run at all. - [`verify.self_testing`](https://feedbackbench.com/coding-agents/criteria/verify.self_testing.md) **Builds, tests or runs its own changes.** The post says whether the agent built, tested, ran or checked its own change before handing it back: it ran the test suite, skipped tests, did not run the app, or verified in proportion to the change. Boundary: Not this: see verify.false_completion when the agent claims success that did not happen. Not this: see verify.agent_code_review for a separate review mode or review agent. - [`verify.agent_code_review`](https://feedbackbench.com/coding-agents/criteria/verify.agent_code_review.md) **Agent-performed code review finds real issues.** A review mode or review agent finds real defects in code or PRs, without re-flagging code that already passed or producing noise. Boundary: Not this: see work.bug_diagnosis for fixing a known symptom. Not this: see work.sycophancy_pushback for reversing findings under pressure. - [`verify.change_review_ui`](https://feedbackbench.com/coding-agents/criteria/verify.change_review_ui.md) **Reviewing and approving the agent's changes.** Diff views, per-file approval, edit-acceptance prompts, and how manageable the amount of change is for a human reviewer. Boundary: Not this: see setup.ide_integration for general editor features. Not this: see work.response_verbosity for prose length. #### Interface and sessions (`interface`) How do you operate and steer it? - [`ui.display_settings`](https://feedbackbench.com/coding-agents/criteria/ui.display_settings.md) **How the interface shows work, and what the user can configure.** How the CLI, TUI, IDE panel or app shows the agent's activity (tool calls, diffs, logs, subagent progress, scrolling, fonts, layout) and which settings, keybindings, themes and presets the user can change. Boundary: Not this: see limits.usage_meter for usage displays. Not this: see ui.session_history for saving and resuming sessions. Not this: see models.effort_control for reasoning-effort settings. - [`ui.session_history`](https://feedbackbench.com/coding-agents/criteria/ui.session_history.md) **Saving, switching, resuming and rewinding sessions.** Whether chats and threads persist, can be switched and resumed with their state intact, and can be rewound or reverted. Boundary: Not this: see context.session_memory for knowledge carried into a new session. - [`ui.interrupt_steer`](https://feedbackbench.com/coding-agents/criteria/ui.interrupt_steer.md) **Stopping and steering a running agent.** Whether the user can stop, interrupt or send guidance mid-run and have the agent respect it and continue. Boundary: Not this: see work.premature_stop for the agent stopping on its own. - [`surfaces.remote_mobile`](https://feedbackbench.com/coding-agents/criteria/surfaces.remote_mobile.md) **Mobile, remote-control and voice access.** Driving or monitoring agents from a phone, another device or by voice, including approvals from the mobile client. Boundary: Not this: see surfaces.cloud_sessions for where the agent executes. - [`surfaces.cloud_sessions`](https://feedbackbench.com/coding-agents/criteria/surfaces.cloud_sessions.md) **Cloud and remote sandbox execution.** Agents run in hosted or remote environments. Covers persistence when the laptop closes, access to local files, session caps, and start or resume time. Boundary: Not this: see billing.overage_charges for how cloud runs are billed. Not this: see work.long_running_autonomy for run behaviour. #### Reliability and speed (`reliability`) Does it stay up and respond quickly? - [`rel.service_errors`](https://feedbackbench.com/coding-agents/criteria/rel.service_errors.md) **Outages, server errors and capacity or rate errors.** Backend unavailability, 5xx or connection errors, capacity errors, and transient 429 errors that do not reflect the user's quota. Boundary: Not this: see limits.window_interrupts_work and limits.allowance_size for plan quota being reached. Not this: see rel.tool_call_errors for harness tool failures. - [`rel.response_speed`](https://feedbackbench.com/coding-agents/criteria/rel.response_speed.md) **Latency, throughput and fast mode.** How quickly the agent responds and completes work, including paid or fast modes and whether they actually deliver speed. Boundary: Not this: see rel.service_errors for failures. Not this: see rel.client_crash_resources for local slowness from resource use. - [`rel.client_failures`](https://feedbackbench.com/coding-agents/criteria/rel.client_failures.md) **Client crashes, freezes and failed tool execution.** The local app, CLI or extension crashes, freezes, leaks memory or uses too much CPU, or fails to execute the model's tool calls and shell commands. Boundary: Not this: see rel.service_errors for backend outages and server errors. Not this: see rel.update_breakage when a specific update caused it. - [`rel.update_breakage`](https://feedbackbench.com/coding-agents/criteria/rel.update_breakage.md) **Updates break working setups.** New releases regress features, break compatibility or change behaviour unexpectedly. Also covers praise for smooth migrations. Boundary: Not this: see limits.allowance_change for quota changes delivered in an update. #### Account and support (`account`) How does the vendor treat your account? - [`account.support`](https://feedbackbench.com/coding-agents/criteria/account.support.md) **Support, refunds and issue handling.** Reaching a human, getting refunds and resolutions, and the vendor's responsiveness to bug reports and community issues. Boundary: Not this: see account.billing_errors for the underlying wrong charge. - [`account.billing_errors`](https://feedbackbench.com/coding-agents/criteria/account.billing_errors.md) **Wrong charges, failed payments and plan provisioning.** Payments are charged incorrectly, plans are not provisioned after payment, proration errors occur, or payments fail. Boundary: Not this: see billing.overage_charges for metered overage. Not this: see account.support for how the vendor handled the error. - [`account.bans_restrictions`](https://feedbackbench.com/coding-agents/criteria/account.bans_restrictions.md) **Account bans and access restrictions.** The vendor suspends, bans or restricts accounts, for example for heavy usage or regional embargoes. Boundary: Not this: see limits.allowance_size for normal quota lockouts. - [`account.data_privacy`](https://feedbackbench.com/coding-agents/criteria/account.data_privacy.md) **Data retention, training use and deployment isolation.** Whether prompts and code are retained or used for training, and whether private or air-gapped deployment is available. Boundary: Not this: see billing.free_tier for free-model availability itself.