The primary goal of this project was to advance the state-of-the-art understanding of how to elicit and aggregate high quality information and beliefs from (online) crowds.

The results can be broken down into three primary directions.

First, this research produced novel online learning algorithms to assign crowdworkers to tasks. Crowdsourcing markets provide a way for requesters to inexpensively obtain distributed labor and information, and have recently become popular among researchers who use them to conduct user studies, run behavioral experiments, and collect data that is easy for humans to generate but difficult for computers. Unlike in traditional labor markets, requesters interact with many workers rapidly and can potentially adjust their behavior as they learn more about salient features of the environment, such as workers’ skill levels, the difficulty of their tasks, and workers’ willingness to accept their tasks at given prices. We addressed the challenge that a requester faces when assigning heterogeneous tasks to workers with unknown, heterogeneous skills. We formalized this “online task assignment problem,” provided a task assignment algorithm that is provably near-optimal given that workers return repeatedly, and evaluated this algorithm on data collected from the popular crowdsourcing platform Amazon Mechanical Turk. We then extended this line of work to cover classification or labeling tasks in which the quality of work cannot be judged immediately.

Second, this research contributed to a more thorough understanding of performance-based incentives in crowdsourcing systems. We designed and ran randomized behavioral experiments on Amazon Mechanical Turk with the goal of understanding when, where, and why performance-based payments improve work quality, identifying properties of the payment, payment structure, and the task itself that make them most effective. Based on our findings, we proposed a new model of worker behavior that extends the standard principal-agent model from economics to include a worker’s subjective beliefs about his likelihood of being paid. We also designed an algorithm using multi-armed bandit techniques for optimally setting the amounts of performance-based payments for tasks assuming workers strategically choose their level of effort.

Finally, this research contributed a thorough study of several key research questions surrounding the design of prediction market mechanisms that elicit and aggregate beliefs from crowds of traders. A prediction market is a market in which traders buy and sell securities with values that are contingent on the outcome of a future event. For example, a security may pay off $1 if a Democrat wins the 2016 US Presidential election and $0 otherwise. The market price of such a security is thought to reflect the traders’ collective belief about the likelihood that a Democrat will win. To facilitate trade, a prediction market can be operated by an automated market maker, an algorithmic agent that offers to buy or sell securities at some current market price determined by a pricing mechanism that makes use of the trade history. The market maker provides liquidity in the market, effectively subsidizing the market and rewarding traders for their private information. This is especially useful in “combinatorial markets” which offer securities defined on a large or infinite outcome space and propagate information (in the form of prices) appropriately across logically-related securities.

We proposed a general framework for the design of efficient pricing mechanisms over very large or infinite outcome spaces. We took an axiomatic approach, specifying a set of formal mathematical properties that any reasonable market should satisfy and fully characterized the set of pricing mechanisms that satisfy these properties....

Please report errors in award information by writing to: awardsearch@nsf.gov.