The Loss of Collective Intelligence in Reputation Systems John Horton & Joe Golden
Distribution of seller reputation on eBay (Nosko & Tadelis)
Distribution of college grades in the US over time
Goal for this talk •
Explore reputation/feedback scores over time in an large online marketplace
•
Present some evidence on the cause(s) of increasing feedback scores
oDesk (now Upwork) •
Online labor market for work that can be done remotely
•
I worked at oDesk from 2011 - 2013
•
I have data on all the feedback
Public feedback scores on Upwork/oDesk (2015)
It has not always been this way on oDesk
Some hypotheses Both imply an improvement in fundamentals •
Composition & cohort effects
•
Learning and platform improvements
•
Drifting standards for what is “good” and “bad” unwillingness by employers to give “bad” feedback
Do other measures of “real” improvement have a trend? •
1) Reason employer gives for ending a contract: •
•
“Successfully completed” or “Not successfully completed”
2) Sentiment of written feedback
Fraction of contracts privately rated as “successful” over time
Month
Most growth has been in perfect 5-star feedback of “successful” contracts ≤
Numerical public feedback.
Written public feedback.
Written-to-numerical translation 1. Take numerical and written feedback for a 1st quarter of 2008 2. Fit model predicting numerical feedback from written feedback 3. Apply model out-of-sample 4. Look at trend
Predicted avg. feedback versus actual feedback Training period
Actual Predicted
Some hypotheses *Maybe* some evidence of improvements •
Composition & cohort effects
•
Learning and platform improvements
•
Drifting standards for what is “good” and “bad” unwillingness by buyer/employers to give “bad” feedback
The problem of “inflated” feedback •
oDesk was worried about information loss from top-censoring of feedback
•
Main hypothesis: Giving bad feedback was costly and employers “learned” it was easier to give good feedback for bad work; Standards shifted
•
Idea: Try collecting “private” feedback and see if it has more information
New private feedback
Distribution of private feedback
Would rather say nothing than say something bad
Among those having a bad experience, perfect 5 stars is most common rating:
Main take-aways •
Bad experiences = (very often) Public silence •
•
Long-thought to be true (and tested for indirectly in other settings)
Even bad experiences often leads to perfect feedback
Private feedback has more information Outcome: Next Public Feedback of a worker
How private feedback is now used Platform can reveal aggregates of past performance without revealing precise ratings by previous partners
Concluding thoughts •
Substantial inflation in feedback scores over time •
Not entirely due to changes in fundamentals
•
Substantial gap between what raters will say publicly and privately
•
Aggregated private feedback might be a solution to information loss due to inflation
•
Title: The Loss of Collective Intelligence in Reputation Systems
Page 1. Whoops! There was a problem loading more pages. Retrying... John Horton - CI 2016.pdf. John Horton - CI 2016.pdf. Open. Extract. Open with. Sign In.
www.packtpub.com. Page 3 of 698. Android Programming for Beginners by John Horton (Author), PACK Publication.pdf. Android Programming for Beginners by ...
Jun 2, 1998 - Primary Examiner * Allen Wong. (57). ABSTRACT. A method for padding interlaced texture information on a reference VOP to perform a motion ...
CI 310. June, 2016. CIT-003 : WEB BASED TECHNOLOGIES AND. MULTIMEDIA APPLICATIONS. Time : 2 hours. Maximum Marks : 50. (Weightage ; 75%). Note : There are two sections in this paper, Section A and. Section B. All the questions from Section A are comp
Jun 2, 1998 - In digitally televised systems such as video-telephone, teleconference .... The above and other objects and features of the present invention Will ...
And Created For A Language Arts Activity After. Reading This Classic Storybook. For Educational Purposes Only: Not For Profit. Dr. Seuss' Horton Labels.
Los docentes bonaerenses. Las polÃticas educativas. vistas desde el aula. INVESTIGACIONES COORDINADAS POR: ANA ABRAMOWSKI Y BELÃN IGARZÃBAL. ROMINA PAULA CARBONATTO Y MARA MOBILIA. EMILIO TENTI FANFANI. Page 3 of 67. CI#1 UNIPE.pdf. CI#1 UNIPE.pdf
Oct 8, 1998 - (10) Patent Number: US RE42,738 E. Williams ... ality may be incorporated in a relatively small plastics casing. 345/157' 169> 179> ..... 800%. 6,097,372 A. 8/2000 Suzuki. /. 6,108,426 A 8/2000 Stortz all;. 58%;? A1 gig? .... Conditioni
AJit- CD V - Rosa L, Susalo ES. P-M- CD Vl -- L- R. Plscual ES. 2- AlloftEd rirDe for fte Frscntarion per Sciml is 10 minutes. 3, Tb rsnbeis of fte bord ofJrdge$ will be dte Tsschsr4oochs coming fiom cdier. diSicl- CDI - C,oches fiorn CD VI. CD II -
