Best Practices for Running Studies on Mechanical Turk

Written by the FindingFive Team

This tutorial was adapted from an article originally written by Lila Abreu, a former Content Development leader at FindingFive.

MTurk can be useful for recruiting participants quickly, but it also requires careful study design, responsible requester practices, and clear participant communication. Data quality depends on the task, target population, screening process, and quality-control checks, so treat MTurk as one recruitment option rather than a universally better or worse sample source. This tutorial summarizes best practices for running FindingFive studies on MTurk.

For setup instructions, see Integrate FindingFive with Mechanical Turk.

How Mechanical Turk + FindingFive Works

On MTurk, requesters (researchers) post Human Intelligence Tasks (HITs) that workers (participants) can browse and complete. Each HIT includes a description, estimated time, and compensation.

For MTurk studies launched directly through FindingFive, participants complete the study directly on MTurk without needing to navigate to a separate study link. You can also control access to follow-up studies based on completion of earlier studies, which can be useful for longitudinal or multi-part study designs. In addition, FindingFive automatically approves submitted HITs. Many IRB protocols and ethical payment practices discourage withholding payment solely due to poor data quality, and automatic approval helps participants receive payment promptly.

Be a Responsible MTurk Requester

Mechanical Turk can provide access to large participant pools, accelerate data collection, and reduce recruitment costs. However, MTurk workers are also vulnerable to scams, underpayment, and unclear requester expectations. When using MTurk for research, protect participants' time, privacy, and well-being.

Use these practices when designing your study:

• Do not ask for personally identifiable information. Requesting personal information may make participants feel that payment depends on disclosure. It may also violate Amazon's user policies. To contact participants without requesting PII, you can email them using their MTurk Worker IDs directly from FindingFive:

  • On the Studies page, select your study.
  • Open the session dashboard (Sessions) and click the Manage participants icon.
  • Find the participant in the list and click the mail icon to email them.

  Note

  You can only email participants who have already completed a study of yours.

• Represent your study accurately. Use accurate session names (which will become HIT titles), session descriptions, time estimates, and eligibility criteria. Informed consent is displayed before a worker accepts the HIT.

• Pay participants fairly. MTurk workers are often underpaid. Do not use the lowest-paying HITs on MTurk as a benchmark for your own compensation rates.
• Never release data linked to Worker IDs. Worker IDs do not directly contain a participant's identity, but they may be linkable to Amazon shopping profiles in some cases. Publicly releasing data linked to Worker IDs can create privacy risks.

The MTurk requester blog offers additional guidance on responsible requester practices.

Screen Participants

FindingFive's MTurk launch workflow supports MTurk's standard screening options, including:

• Minimum HIT approval rate
• Minimum number of HIT approvals
• Participant location
• Prior attempt or completion status for another study, based on that study's Session ID

Location screening may be imperfect

If your study targets participants in specific US states, note that Amazon's state-level location information may be outdated or unreliable.

You may also want to recruit a more specific population, such as participants in a particular age range, occupation, or demographic group. For highly specific criteria, consider running a screening survey first or use third-party tools.

Use screeners deliberately rather than treating them as a formality. Good screeners should verify eligibility without making the desired answer obvious, and researchers should expect some respondents to fail screeners even in motivated samples.

Run a Screening Survey

A screening survey is a short preliminary study used to identify eligible participants for a later main study. Use this approach when your target population is specific.

To run a screening survey:

1. Create a short survey with eligibility questions and a few distractor demographics. Recruit more participants than you need for the final sample.
2. Mention that completion may make participants eligible for a longer study, but do not reveal the target demographics.
3. Download the screening survey data and identify eligible participant IDs.
4. Enter those participant IDs when launching the main study. FindingFive will limit access to those participants and email them on your behalf.

Participant IDs vs. Worker IDs

Use FindingFive participant IDs in your downloaded data csv to control access to follow-up studies. To protect participant privacy, you will not be able to associate those participant IDs with MTurk Worker IDs.

For example, to recruit school teachers between ages 25 and 35, run a short screening survey asking about age and occupation, plus distractor demographics such as income level, household size, or race. Then, when launching the main study, enter the participant IDs of those who reported being teachers between ages 25 and 35. Only those participants will receive an email invitation to the main study.

Improve Data Quality

MTurk studies run in participants' own environments, on their own devices, and without in-person supervision. The following practices can improve study quality while keeping expectations transparent and fair.

Set High HIT Approval Requirements

HIT approval ratings indicate the percentage of an MTurk worker's submissions that have been approved by requesters. A high rating means the worker's previous submissions have generally been accepted.

Amazon has recommended setting a HIT approval rating of at least 95% and a minimum of 5,000 HIT approvals to improve data quality. For more selective studies, you may consider a minimum approval rating of 98%.

Consider sample naivete

As the minimum number of HIT approvals increases, workers are more likely to have encountered common behavioral task paradigms in previous MTurk HITs. If your study requires a highly naive sample, consider whether these thresholds fit your design (although at the risk of having to exclude more participant data).

Use an Appropriate Timeout Duration

After accepting a HIT, participants have a limited amount of time to complete and submit it. This period is the timeout duration that you can set in FindingFive's session wizard. If a participant does not complete the study before the timeout, the study will terminate and they will not be presented with the opportunity to submit the HIT.

Set a timeout that is long enough for participants to complete the study comfortably, but not so long that participants are encouraged to leave and return much later. Base the timeout on your own pilot testing and the expected completion time, and err on the generous side to avoid accidental nonpayment for participants who are completing the study in good faith.

Write Clear Instructions

MTurk participants cannot ask you for real-time clarification while completing the task, so your study description should be direct and easy to follow.

Consider including:

• Device or environment requirements, such as headphones for audio studies
• Practice rounds before experimental trials
• Repeated reminders during the study when instructions are important
• Accessibility checks, such as avoiding red/green distinctions when color is not central to the study design

Participants will be able to review these instructions before accepting the HIT, so make sure they are clear and accurate to set appropriate expectations.

Include Attention Checks

Attention checks, also called catch trials, can help identify participants who are clicking through a task without reading carefully. For example, a well-designed attention check might include instructions to disregard the apparent question and select a specific response:

To show that you've read this much, please ignore the question and select only "Singing" in the options below. Please select all genres of music that you enjoy listening to.

FindingFive's catch_trials feature can help you insert attention checks periodically throughout a study. In the output spreadsheet, these trials are identified so you can review them during analysis. Carefully designed attention checks may also detect the use of AI tools by participants, since AI may not be able to follow instructions that require understanding of the task context.

Test Your Study Before Launch

Run your study from start to finish before launching it to participants. Ask a few people who are unfamiliar with the task to complete it so you can check that instructions are clear and the study behaves as expected.

You can also launch your study to MTurk's Sandbox through FindingFive. The Sandbox lets you view the study in MTurk's interface from the participant perspective without making the HIT available to real workers and without needing production billing to be ready.

Advanced Topics

Run Longitudinal or Multi-Part Studies

You can run longitudinal or multi-part studies on MTurk. To do so, create multiple studies on FindingFive where each study represents one phase of the larger project. Start by posting Study 1 and recruiting more participants than you need for the final sample, since some attrition is expected.

For later parts, FindingFive provides options for limiting participation based on completion of another study, as long as that study has launched MTurk sessions. This makes it possible to select participants for Part 2 who completed Part 1.

You can also use participant IDs to include a subset of earlier participants when launching a later phase. This can be useful if you need to screen Phase 1 data before deciding who should continue.

Managing Participant Payments

In addition to the preset compensation specified during study launch, you can provide additional payments to reward participants or compensate for technical issues.

Bonus Payments

You can send bonus payments to reward participants who meet certain performance criteria:

1. From the Participant Management page of the session, open the Completed Participants tab.
2. Use the Worker ID column to identify the participant and select the bonus payment icon.
3. Enter the bonus amount and a message.

Note

MTurk charges a 20% fee for all bonus payments. Funds are transferred directly from your AWS account to the participant.

Compensation HITs

If participants abandon a study due to server glitches or connection problems but have completed most of the work, you can create a compensation HIT to ensure they are paid:

• Go to the Session Dashboard and locate the corresponding MTurk session.
• Click the Manage Participants icon and locate the participant in the abandoned participants by worker ID.
• Click the three dots to bring up additional actions and click Create compensation HIT.
• Review the message template (ensure the [HIT_LINK] tag remains) and click Send.

Next Steps

Before launching a production MTurk study, make sure you have:

1. Completed the MTurk integration setup in Integrate FindingFive with Mechanical Turk
2. Piloted your study from start to finish
3. Tested the MTurk launch flow in Sandbox
4. Set fair compensation and realistic timing
5. Checked participant privacy and participant ID handling

Questions or suggestions? Contact researcher.help@findingfive.com.