The Collaboration & Communication Networks Within the Crowd Mary L. Gray Siddharth Suri Microsoft Research

Computer Science & Ethnography On-Demand: Crowds, Platform Economies, and the Future of Work in Precarious Times Key questions: Who are the crowdworkers?

Why do they do crowd work?

Platform Requester

Task

API

How has crowdwork affected their lives? What is the future of crowdwork?

Open the black box of crowdsourcing

Methodology

Crowd

Computer Science & Ethnography On-Demand: Crowds, Platform Economies, and the Future of Work in Precarious Times Key questions: Who are the crowdworkers?

Why do they do crowd work?

Sid

Platform

Requester

Task

Mary API

How has crowdwork affected their lives? What is the future of crowdwork?

Open the black box of crowdsourcing

Methodology

Worker

Why Open the Black Box of Crowdsourcing? To build better crowdsourcing systems and more efficient workflows, we need to understand how does the work get done. More behavioral experiments are using crowdsourcing sites [Mason & Suri 2012] Need to understand our subject pool

The future of work could be more like crowdsourcing Sharing economy, gig economy look more like crowdsourcing than “regular” jobs Growth of Uber, Upwork (oDesk), Top Coder

Outline Introduction

The Crowd is a Collaborative Network M. L. Gray, S. Suri, S. Ali, D. Kulkarni, CSCW 2016

The Communication Network Within the Crowd M. Yin, M. L. Gray, S. Suri, J. W. Vaughan, WWW 2016

Conclusion

Research Questions & Key Findings on Collaboration Research questions: Do crowdworkers collaborate and if so, how do they collaborate and what do they collaborate on? Collaborate to manage the administrative overhead involved with doing crowd work Collaborate to share information about lucrative tasks and good requesters Collaborate to do the work itself Study 4 platforms: Amara, LeadGenius, Mturk, UHRS

Workers Collaborate to Manage Administrative Overhead Kumuda, 34 year old Hindu woman and computer trainer:

“I actually started with outsourcing. I and a friend of mine were searching for job offers….Everywhere it was a scam. [then I found Mturk] I am the first person in my area to find about MTurk. My friends have come to know about MTurk through me [so they know it is safe].”

Survey Data ~25% of those surveyed in the U.S. and India were referred to MTurk by a friend. LeadGenius had even higher rates of referrals from friends

Workers Share Task and Requester Information Fareed, a devout Muslim in his late 20s, and the eldest brother in his family. A native of Hyderabad. “Anyone who sees work posted calls and tells everyone. There is no fixed timing. Whosoever is alert and sees informs everyone and in this way everyone helps everyone else. Around 150 friends (on Facebook).”

Designed a HIT to study how people find HITs: Forums: 41% of traffic Search: 36% of traffic

Workers Collaborate on Doing the Work Poonam and Sanjay (married) hand off UHRS tasks based on their skills Sanjay takes the visual tasks and Poonam takes the written tasks

Lalitha, a Christian mother of two living in Hyderabad Gets help from her sons to categorize “adult content” joking, “they are more qualified to recognize these words than me!...I need their help to keep the internet clean and safe for other families.”

Survey Data: Around 5% of workers across platforms ask friend for help

Organic Collaboration Workers collaborate organically. They self organize. Collaborate to manage the administrative overhead involved with doing crowd work Collaborate to share information about new tasks and good requesters Collaborate to do the work itself Workers communicate!

Outline Introduction

The Crowd is a Collaborative Network M. L. Gray, S. Suri, S. Ali, D. Kulkarni, CSCW 2016

The Communication Network Within the Crowd M. Yin, M. L. Gray, S. Suri, J. W. Vaughan, WWW 2016

Conclusion

Research Question We know that workers collaborate and therefore comunicate. Research Question: What is the topology of the communication network within the crowd? Scale up the discoveries in the previous section

Approach: Build a web app that gives workers an incentive to report their connections while preserving their privacy. Focus on Mturk.

Why Study the Network in the Crowd? Communication network is the most general The collaboration and friendship networks are subnetworks

The communication network impacts how does work get done. How do workers find out about the tasks?

Crowdsourcing is used to train machine learning algorithms If workers communicate then they might not be independent Non-independence can bias the “Wisdom of the Crowd” [Muchnik et al 2013]

HIT: Map the Network of Turkers Designed a network mapping HIT to visualize the communication network where Turkers self-report their connections

Step 5: Exploring Neighbors Clicking on a node connected to you revealed: Workers nickname Information shared only with connections

Workers given a private url so they could come back and add/view/delete edges MTurk Task Paid $1, ~10 minutes

Ran for 2 weeks 10,354 respondents Roughly a census [Stewart et al, 2015]

1,389 workers (13.4%) added at least 1 edge, call them “connected” Network within the crowd

90% of all edges are between pairs of workers who communicate via forums 86% are between pairs communicate exclusively through forums.

Connected Workers Found Our HIT Earlier Sort all workers according to the time that they took our HIT and bin them into groups of 200 The fraction of connected workers who did our HIT decreases ~13% of workers are connected

Connected workers are more likely to hear about our HIT first

Outline Introduction

The Crowd is a Collaborative Network M. L. Gray, S. Suri, S. Ali, D. Kulkarni, CSCW 2016

The Communication Network Within the Crowd M. Yin, M. L. Gray, S. Suri, J. W. Vaughan, WWW 2016

Conclusion

Conclusion Workers talk! About 13% of them talk!

Communication is driven by forum usage “If I had not found TurkerNation, I would not have made as much money for sure. And the fun we have when things are slow: priceless.” [Zyskowksi and Milland 2015]

Confers an informational advantage Connected workers might “starve or crowd out” unconnected ones Workers are not independent, more likely to recruit their contacts

Implications of experimental design, machine learning algorithms, wisdom of crowd studies?

Conclusion Crowdsourcing removes everything from a job except the labor and the pay No promotions, no bosses, no colleagues, no benefits…

Workers are building collaboration and communication back in at their own expense! Workers must value these things, how can we build systems that allow collaboration? In crowdsourcing sometimes you sample an independent individual and sometimes you sample an individual embedded in a network

Mturk

Requester

HIT

w

Mturk

and more like this:



Crowdsourcing is less like this:

w

Requester

HIT

Sid Suri - CI 2016.pdf

Whoops! There was a problem loading more pages. Retrying... Whoops! There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Sid Suri - CI 2016.pdf. Sid Suri - CI 2016.pdf. Open. Extract. Open with. Sign In. Main menu.

507KB Sizes 1 Downloads 182 Views

Recommend Documents

CI PAD
Jun 2, 1998 - Primary Examiner * Allen Wong. (57). ABSTRACT. A method for padding interlaced texture information on a reference VOP to perform a motion ...

CI 310
tag. (a) . (b) . (c) . (d) . (ix) CSS stands for. (a) Covering Style Sheet. (b) Cascading Style Sheet. (c) Center Style Sheet. (d) None of the above.

CI 310
CI 310. June, 2016. CIT-003 : WEB BASED TECHNOLOGIES AND. MULTIMEDIA APPLICATIONS. Time : 2 hours. Maximum Marks : 50. (Weightage ; 75%). Note : There are two sections in this paper, Section A and. Section B. All the questions from Section A are comp

C6U,Ci
Jun 17, 2016 - physical fire precautions and management arrangements necessary to ... CSDO Building, Dasmarifias Community Affair Compound, Brgy.

CI PAD
Jun 2, 1998 - In digitally televised systems such as video-telephone, teleconference .... The above and other objects and features of the present invention Will ...

Canyon Crest SID
R id g e. F a lls. D r iv e. Ivy. Lee. Crest. K en sington. A venue. O. asisB o u le v a rd. Canyon Crest Boulevard. Hardy Way. M e s q u ite. H e ig h ts. R o a d.

CI Project Template Definitions
Jan 8, 2016 - CI Project Templates and Presentations. DATE ... Email: [email protected] I Website: www.depeddasma.edu.ph. Dk"... nAA.

CI#1 UNIPE.pdf
Los docentes bonaerenses. Las políticas educativas. vistas desde el aula. INVESTIGACIONES COORDINADAS POR: ANA ABRAMOWSKI Y BELÉN IGARZÁBAL. ROMINA PAULA CARBONATTO Y MARA MOBILIA. EMILIO TENTI FANFANI. Page 3 of 67. CI#1 UNIPE.pdf. CI#1 UNIPE.pdf

SID 210A 1 Fall 1994 INTRO TO ECONOMICS Carter,Anne P 3 SID ...
INTNL.DEV. Simon,Laurence. 10. SID. 245B. 1. Spring1997. STUDIES IN SUS.INTNL.DEV. Simon,Laurence. 17. SID. 250A. 1. Fall 1994. TOPICS/TOOLS:SUST.

Sid England Transcript - edited.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Sid England ...

Sid meiers colonization reloaded
Out ofmymind pdf.Sid meiers. colonization reloaded.Sid meierscolonization reloaded.Spider man 1963.Pervcity lylastorm. Perfect NaturalKnockers.886861126.Kaleido star ova.50 top bestselling.Heartist feeding fiction.Her turn to cheat brazzers.Veronica

SID Mumbai Bharti [email protected]
The grade pay at the time of retirement should be less than Rs. 7600/-[As. per sixth Pay] and equivalent emoluments as per Seventh Pay commission.

SID Mumbai Bharti [email protected]
Page 2 of 3. “SID, Maharashtra State, Mumbai”. Maharashtra State Police Headquarter, Shahid Bhagat singh Marg, Colaba, Mumbai – 400001. ---**---. Advertisement for contractual appointment of the officers retired from IB/R&AW/SID. ---***---. App

John Horton - CI 2016.pdf
Page 1. Whoops! There was a problem loading more pages. Retrying... John Horton - CI 2016.pdf. John Horton - CI 2016.pdf. Open. Extract. Open with. Sign In.

20140923 - SID Cable - RFQ186 - 0630pm.pdf
IRG/PDP/SID/2014/186. For the Supply of Sheathed Insulated Distribution (SID) Cable to WAPDA. Specification DDS-8:2007. RFQ Issuance Date: September 24 ...

Programa SID Clot 17.pdf
biblioteca. Page 1 of 1. Programa SID Clot 17.pdf. Programa SID Clot 17.pdf. Open. Extract. Open with. Sign In. Main menu. Displaying Programa SID Clot 17.pdf ...

Programa SID Clot 17.pdf
biblioteca. Page 1 of 1. Programa SID Clot 17.pdf. Programa SID Clot 17.pdf. Open. Extract. Open with. Sign In. Main menu. Displaying Programa SID Clot 17.pdf ...

CI-2016-01.pdf
Sign in. Main menu.

Nováčci 2014.pdf
www.atletikavozickaru.cz IČ 26997452 Komerční banka, a.s. – č.ú.: 51-1035190267/0100. [email protected] Reg. MV ČR, č.j. VS/1-1/60 729/05-R.

Q :5 Ci) d3 (:5 5
Oct 8, 1998 - (10) Patent Number: US RE42,738 E. Williams ... ality may be incorporated in a relatively small plastics casing. 345/157' 169> 179> ..... 800%. 6,097,372 A. 8/2000 Suzuki. /. 6,108,426 A 8/2000 Stortz all;. 58%;? A1 gig? .... Conditioni

Wieści Zakrzowieckie październik gdk.pdf
Wieści Zakrzo ... ernik gdk.pdf. Wieści Zakrzow ... iernik gdk.pdf. Open. Extract. Open with. Sign In. Main menu. Displaying Wieści Zakrzowieckie październik ...