{"id":38243,"date":"2017-03-17T09:45:38","date_gmt":"2017-03-17T16:45:38","guid":{"rendered":"http:\/\/world.edu\/?p=38243"},"modified":"2017-03-17T09:45:38","modified_gmt":"2017-03-17T16:45:38","slug":"psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls","status":"publish","type":"post","link":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/","title":{"rendered":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls"},"content":{"rendered":"

\"mechanicalTurkCareerPage\"<\/a><\/p>\n

You may not know this, but a great deal of our data about the human mind is based on a relatively small but intensively studied population: first-year undergraduate university students.<\/p>\n

There has long been concern<\/a> about the over-reliance on students as a source of data, particularly around lack of demographic diversity and limited sample sizes. Both concerns have been implicated in the current crisis<\/a> in psychological research, in which many key effects have not been replicated<\/a> by subsequent studies.<\/p>\n

But now there\u2019s a new tool in the psycologist\u2019s arsenal, one that has shown it can produce valid data<\/a>, which can help broaden the population of test subjects: Amazon\u2019s Mechanical Turk<\/a>.<\/p>\n

Mechanical Turk is the most popular online data collection platform. People register with the platform, then choose from thousands of advertisements searching for participants. Compared to the slow slog of testing first-year students in a laboratory, Mechanical Turk offers the opportunity to collect hundreds of responses at a modest cost in a matter of hours.<\/p>\n

Psychologists have embraced Mechanical Turk with gusto, and many recent studies have drawn their data from Mechanical Turk subjects. However, the new system is not without its drawbacks.<\/p>\n

Too good to be true?<\/strong><\/p>\n

Recently, researchers working with Mechanical Turk have raised concerns about whether it comes with hidden costs. The original goal of many researchers using this platform was to conduct scientifically valid studies in large and diverse samples.<\/p>\n

But does online data really provide the solution? Here are some pros and cons.<\/p>\n

Sample Size<\/strong><\/p>\n

Mechanical Turk offers an unparalleled opportunity to collect large samples, particularly in comparison to traditional undergraduate participant pools, many of which cap annual testing at 200 participants.<\/p>\n

In contrast, there are more than 500,000 workers registered on Mechanical Turk. However, researchers must be careful to prevent Mechanical Turk workers from participating in the same study more than once and thus invalidating the results.<\/p>\n

Diversity<\/strong><\/p>\n

Psychologists are concerned with whether findings generalise beyond student samples, or beyond so-called \u201cWEIRD<\/a>\u201d (Western, Educated, Industrialised, Rich and Democratic) samples.<\/p>\n

This is important not only for their own theoretical closure, but also to increase confidence in the general public about the overall validity and importance of the findings. The demographic diversity of Mechanical Turk<\/a> workers is certainly more varied than that of undergraduate students.<\/p>\n

Reliability<\/strong><\/p>\n

One barrier to obtaining reliable data is participant engagement, which is easier to ensure and monitor with students in the lab than it is with Mechanical Turk workers in their own home<\/a>.<\/p>\n

To combat this issue, researchers typically integrate checks into questionnaires to identify participants who are not paying attention, such as asking: \u201cThis is a test item, please answer \u2018not at all\u2019 for this question\u201d. However, there are now reports that Mechanical Turk workers are adept at spotting such questions.<\/p>\n

Fortunately, larger sample sizes do allow researchers to \u201cwash out\u201d the noise of less-than-perfect data. This means that relationships can be detected in noisy data given a large enough sample, but that averages across different samples are likely to differ.<\/p>\n

Naivety<\/strong><\/p>\n

Another concern is the naivety of research participants. Study results are unlikely to be valid if participants know the procedure and expected hypotheses in advance.<\/p>\n

Lack of naivety in this form has the potential to significantly alter results<\/a>, and thus impact on replicability. Here, undergraduate samples present an advantage: students typically complete only a handful of studies in their first year, and most before being exposed to detailed information about psychology.<\/p>\n

In contrast, some MTurk workers treat study completion as a full-time job, completing hundreds of studies per week<\/a>. More concerning still is the availability of online communities in which workers trade information about study hypotheses and procedures and offer tips on completing studies quickly, which inevitably comes at the cost of psychological engagement.<\/p>\n

The future of online data collection<\/strong><\/p>\n

Researchers turned to online data collection platforms like Mechanical Turk because they offered quick, cheap and apparently scientifically valid solutions to problems implicated in the replication crisis.<\/p>\n

Although Mechanical Turk allows for collection of large and diverse samples, it comes with other costs that may compromise scientific rigour, including questionable quality and validity of results.<\/p>\n

This means Mechanical Turk is useful, but only to the extent that researchers are aware of, and compensate for, its pitfalls. This includes:<\/p>\n

1) embedding novel attention checks to keep ahead of savvy workers<\/p>\n

2) ensuring workers do not complete studies more than once<\/p>\n

3) avoiding common procedures that workers have seen hundreds of times; and<\/p>\n

4) diversifying onto other online platforms<\/a> (or creating an Australian platform that is better suited to the requirements of local researchers)<\/p>\n

Overall, online samples should be used as a complement to, not a replacement for, traditional student samples. Both methods have their own strengths and weaknesses, but together produce better science.<\/p>\n

While Mechanical Turk isn\u2019t the silver bullet that psychology researchers had hoped, harnessing its benefits and offsetting its costs will ensure the future of online data collection is still bright.<\/p>\n

Author Bios: <\/strong>Michael Humphreys is Emeritus Professor of Cognitive Psychology, Katharine H. Greenaway is a Research Fellow in Social Psychology and Sarah Bentley is a Researcher Social Psychology at The University of Queensland<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

You may not know this, but a great deal of our data about the human mind is based on a relatively small but intensively studied population: first-year undergraduate university students. There has long been concern about the over-reliance on students as a source of data, particularly around lack of demographic diversity and limited sample sizes. […]\n… learn more→<\/a><\/p>","protected":false},"author":33,"featured_media":38244,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[29],"tags":[65904],"coauthors":[11733],"class_list":["post-38243","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-higher-education","tag-online-data-collection"],"acf":[],"yoast_head":"\nPsychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:locale\" content=\"en_US\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:type\" content=\"article\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:title\" content=\"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:description\" content=\"You may not know this, but a great deal of our data about the human mind is based on a relatively small but intensively studied population: first-year undergraduate university students. There has long been concern about the over-reliance on students as a source of data, particularly around lack of demographic diversity and limited sample sizes. […]\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:url\" content=\"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:site_name\" content=\"World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"article:published_time\" content=\"2017-03-17T16:45:38+00:00\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:image\" content=\"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:width\" content=\"680\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:height\" content=\"201\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"author\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" class=\"yoast-seo-meta-tag\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/\"},\"author\":{\"name\":\"Kevin\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"headline\":\"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls\",\"datePublished\":\"2017-03-17T16:45:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/\"},\"wordCount\":871,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/mechanicalTurkCareerPage.jpeg\",\"keywords\":[\"online data collection\"],\"articleSection\":[\"Higher Education\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/\",\"url\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/\",\"name\":\"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/mechanicalTurkCareerPage.jpeg\",\"datePublished\":\"2017-03-17T16:45:38+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#primaryimage\",\"url\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/mechanicalTurkCareerPage.jpeg\",\"contentUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/mechanicalTurkCareerPage.jpeg\",\"width\":680,\"height\":201},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/world.edu\\\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/world.edu\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/world.edu\\\/#website\",\"url\":\"https:\\\/\\\/world.edu\\\/\",\"name\":\"World leading higher education information and services\",\"description\":\"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/world.edu\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\",\"name\":\"Kevin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"caption\":\"Kevin\"},\"description\":\"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.\",\"sameAs\":[\"https:\\\/\\\/world.edu\"],\"url\":\"https:\\\/\\\/world.edu\\\/author\\\/kevin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/","og_locale":"en_US","og_type":"article","og_title":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services","og_description":"You may not know this, but a great deal of our data about the human mind is based on a relatively small but intensively studied population: first-year undergraduate university students. There has long been concern about the over-reliance on students as a source of data, particularly around lack of demographic diversity and limited sample sizes. […]","og_url":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/","og_site_name":"World leading higher education information and services","article_published_time":"2017-03-17T16:45:38+00:00","og_image":[{"width":680,"height":201,"url":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg","type":"image\/jpeg"}],"author":"Kevin","twitter_misc":{"Written by":"Kevin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#article","isPartOf":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/"},"author":{"name":"Kevin","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"headline":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls","datePublished":"2017-03-17T16:45:38+00:00","mainEntityOfPage":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/"},"wordCount":871,"commentCount":0,"image":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg","keywords":["online data collection"],"articleSection":["Higher Education"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/","url":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/","name":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls - World leading higher education information and services","isPartOf":{"@id":"https:\/\/world.edu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#primaryimage"},"image":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg","datePublished":"2017-03-17T16:45:38+00:00","author":{"@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"breadcrumb":{"@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#primaryimage","url":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg","contentUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/mechanicalTurkCareerPage.jpeg","width":680,"height":201},{"@type":"BreadcrumbList","@id":"https:\/\/world.edu\/psychology-turns-online-crowdsourcing-study-mind-not-without-pitfalls\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/world.edu\/"},{"@type":"ListItem","position":2,"name":"Psychology turns to online crowdsourcing to study the mind, but it\u2019s not without its pitfalls"}]},{"@type":"WebSite","@id":"https:\/\/world.edu\/#website","url":"https:\/\/world.edu\/","name":"World leading higher education information and services","description":"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/world.edu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee","name":"Kevin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4","url":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","caption":"Kevin"},"description":"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.","sameAs":["https:\/\/world.edu"],"url":"https:\/\/world.edu\/author\/kevin\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/38243","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/comments?post=38243"}],"version-history":[{"count":0,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/38243\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media\/38244"}],"wp:attachment":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media?parent=38243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/categories?post=38243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/tags?post=38243"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/coauthors?post=38243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}