{"id":74350,"date":"2024-10-03T04:28:34","date_gmt":"2024-10-03T08:28:34","guid":{"rendered":"https:\/\/world.edu\/?p=74350"},"modified":"2024-10-03T04:28:34","modified_gmt":"2024-10-03T08:28:34","slug":"i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions","status":"publish","type":"post","link":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/","title":{"rendered":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions"},"content":{"rendered":"

\"\"<\/p>\n

It\u2019s been nearly two years since\u00a0generative artificial intelligence<\/a>\u00a0was made widely available to the public. Some models\u00a0showed great promise<\/a>\u00a0by passing academic and professional exams.<\/p>\n

For instance, GPT-4 scored higher than 90% of the\u00a0United States bar exam test takers<\/a>. These successes led to concerns AI systems might also breeze through university-level assessments. However, my\u00a0recent study<\/a>\u00a0paints a different picture, showing it isn\u2019t quite the academic powerhouse some might think it is.<\/p>\n

My study<\/strong><\/h4>\n

To explore generative AI\u2019s academic abilities, I looked at how it performed on an undergraduate criminal law final exam at the University of Wollongong \u2013 one of the core subjects students need to pass in their degrees. There were 225 students doing the exam.<\/p>\n

The exam was for three hours and had two sections. The first asked students to evaluate a case study about criminal offences \u2013 and the likelihood of a successful prosecution. The second included a short essay and a set of short-answer questions.<\/p>\n

The test questions evaluated a mix of skills, including legal knowledge, critical thinking and the ability to construct persuasive arguments.<\/p>\n

Students were not allowed to use AI for their responses and did the assessment in a supervised environment.<\/p>\n

I used different AI models to create ten distinct answers to the exam questions.<\/p>\n

Five papers were generated by just pasting the exam question into the AI tool without any prompts. For the other five, I gave detailed prompts and relevant legal content to see if that would improve the outcome.<\/p>\n

I hand wrote the AI-generated answers in official exam booklets and used fake student names and numbers. These AI-generated answers were mixed with actual student exam answers and anonymously given to five tutors for grading.<\/p>\n

Importantly, when marking, the tutors did not know AI had generated ten of the exam answers.<\/p>\n

How did the AI papers perform?<\/strong><\/h4>\n

When the tutors were interviewed after marking, none of them suspected any answers were AI-generated.<\/p>\n

This shows the potential for AI to mimic student responses and educators\u2019 inability to spot such papers.<\/p>\n

But on the whole, the AI papers were not impressive.<\/p>\n

While the AI did well in the essay-style question, it struggled with complex questions that required in-depth legal analysis.<\/p>\n

This means even though AI can mimic human writing style, it lacks the nuanced understanding needed for complex legal reasoning.<\/p>\n

The students\u2019 exam average was 66%.<\/p>\n

The AI papers that had no prompting, on average, only beat 4.3% of students. Two barely passed (the pass mark is 50%) and three failed.<\/p>\n

In terms of the papers where prompts were used, on average, they beat 39.9% of students. Three of these papers weren\u2019t impressive and received 50%, 51.7% and 60%, but two did quite well. One scored 73.3% and the other scored 78%.<\/p>\n

What does this mean?<\/strong><\/h4>\n

These findings have important implications for both education and professional standards.<\/p>\n

Despite the hype, generative AI isn\u2019t close to replacing humans in intellectually demanding tasks such as this law exam.<\/p>\n

My study suggests AI should be viewed more like a tool, and when used properly, it can enhance human capabilities.<\/p>\n

So schools and universities should concentrate on developing students\u2019 skills to collaborate with AI and analyse its outputs critically, rather than relying on the tools\u2019 ability to simply spit out answers.<\/p>\n

Further, to make collaboration between AI and students possible, we may have to rethink some of the traditional notions we have about education and assessment.<\/p>\n

For example, we might consider when a student prompts, verifies and edits an AI-generated work, that is their original contribution and should still be viewed as a valuable part of learning.<\/p>\n

Author Bio:<\/strong> Armin Alimardani is a Lecturer, School of Law at the University of Wollongong<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

It\u2019s been nearly two years since\u00a0generative artificial intelligence\u00a0was made widely available to the public. Some models\u00a0showed great promise\u00a0by passing academic and professional exams. For instance, GPT-4 scored higher than 90% of the\u00a0United States bar exam test takers. These successes led to concerns AI systems might also breeze through university-level assessments. However, my\u00a0recent study\u00a0paints a different […]\n… learn more→<\/a><\/p>","protected":false},"author":33,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[434804],"tags":[357236],"coauthors":[11733],"class_list":["post-74350","post","type-post","status-publish","format-standard","hentry","category-artificial-intelligence","tag-law-exams"],"acf":[],"yoast_head":"\nI got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:locale\" content=\"en_US\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:type\" content=\"article\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:title\" content=\"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:description\" content=\"It\u2019s been nearly two years since\u00a0generative artificial intelligence\u00a0was made widely available to the public. Some models\u00a0showed great promise\u00a0by passing academic and professional exams. For instance, GPT-4 scored higher than 90% of the\u00a0United States bar exam test takers. These successes led to concerns AI systems might also breeze through university-level assessments. However, my\u00a0recent study\u00a0paints a different […]\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:url\" content=\"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:site_name\" content=\"World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"article:published_time\" content=\"2024-10-03T08:28:34+00:00\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:image\" content=\"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"author\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" class=\"yoast-seo-meta-tag\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/\"},\"author\":{\"name\":\"Kevin\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"headline\":\"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions\",\"datePublished\":\"2024-10-03T08:28:34+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/\"},\"wordCount\":625,\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/ChatGPTlaw.jpg\",\"keywords\":[\"law exams\"],\"articleSection\":[\"Artificial Intelligence\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/\",\"url\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/\",\"name\":\"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/ChatGPTlaw.jpg\",\"datePublished\":\"2024-10-03T08:28:34+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#primaryimage\",\"url\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/ChatGPTlaw.jpg\",\"contentUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/ChatGPTlaw.jpg\",\"width\":680,\"height\":274},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/world.edu\\\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/world.edu\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/world.edu\\\/#website\",\"url\":\"https:\\\/\\\/world.edu\\\/\",\"name\":\"World leading higher education information and services\",\"description\":\"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/world.edu\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\",\"name\":\"Kevin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"caption\":\"Kevin\"},\"description\":\"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.\",\"sameAs\":[\"https:\\\/\\\/world.edu\"],\"url\":\"https:\\\/\\\/world.edu\\\/author\\\/kevin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/","og_locale":"en_US","og_type":"article","og_title":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services","og_description":"It\u2019s been nearly two years since\u00a0generative artificial intelligence\u00a0was made widely available to the public. Some models\u00a0showed great promise\u00a0by passing academic and professional exams. For instance, GPT-4 scored higher than 90% of the\u00a0United States bar exam test takers. These successes led to concerns AI systems might also breeze through university-level assessments. However, my\u00a0recent study\u00a0paints a different […]","og_url":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/","og_site_name":"World leading higher education information and services","article_published_time":"2024-10-03T08:28:34+00:00","og_image":[{"url":"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg","type":"","width":"","height":""}],"author":"Kevin","twitter_misc":{"Written by":"Kevin","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#article","isPartOf":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/"},"author":{"name":"Kevin","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"headline":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions","datePublished":"2024-10-03T08:28:34+00:00","mainEntityOfPage":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/"},"wordCount":625,"image":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg","keywords":["law exams"],"articleSection":["Artificial Intelligence"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/","url":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/","name":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions - World leading higher education information and services","isPartOf":{"@id":"https:\/\/world.edu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#primaryimage"},"image":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg","datePublished":"2024-10-03T08:28:34+00:00","author":{"@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"breadcrumb":{"@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#primaryimage","url":"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg","contentUrl":"https:\/\/world.edu\/wp-content\/uploads\/2024\/10\/ChatGPTlaw.jpg","width":680,"height":274},{"@type":"BreadcrumbList","@id":"https:\/\/world.edu\/i-got-generative-ai-to-attempt-an-undergraduate-law-exam-it-struggled-with-complex-questions\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/world.edu\/"},{"@type":"ListItem","position":2,"name":"I got generative AI to attempt an undergraduate law exam. It struggled with complex questions"}]},{"@type":"WebSite","@id":"https:\/\/world.edu\/#website","url":"https:\/\/world.edu\/","name":"World leading higher education information and services","description":"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/world.edu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee","name":"Kevin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4","url":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","caption":"Kevin"},"description":"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.","sameAs":["https:\/\/world.edu"],"url":"https:\/\/world.edu\/author\/kevin\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/74350","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/comments?post=74350"}],"version-history":[{"count":2,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/74350\/revisions"}],"predecessor-version":[{"id":74363,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/74350\/revisions\/74363"}],"wp:attachment":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media?parent=74350"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/categories?post=74350"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/tags?post=74350"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/coauthors?post=74350"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}