{"id":38189,"date":"2017-03-14T13:52:59","date_gmt":"2017-03-14T20:52:59","guid":{"rendered":"http:\/\/world.edu\/?p=38189"},"modified":"2017-03-14T13:52:59","modified_gmt":"2017-03-14T20:52:59","slug":"artificial-intelligence-deny-credit","status":"publish","type":"post","link":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/","title":{"rendered":"Did artificial intelligence deny you credit?"},"content":{"rendered":"

\"\"<\/a><\/p>\n

People who apply for a loan from a bank or credit card company, and are turned down, are owed an explanation of why that happened. It\u2019s a good idea \u2013 because it can help teach people how to repair their damaged credit \u2013 and it\u2019s a federal law, the Equal Credit Opportunity Act<\/a>. Getting an answer wasn\u2019t much of a problem in years past, when humans made those decisions. But today, as artificial intelligence systems increasingly assist or replace people making credit decisions, getting those explanations has become much more difficult.<\/p>\n

Traditionally, a loan officer who rejected an application could tell a would-be borrower there was a problem with their income level, or employment history, or whatever the issue was<\/a>. But computerized systems that use complex machine learning<\/a> models are difficult to explain, even for experts.<\/p>\n

Consumer credit decisions are just one way this problem arises. Similar concerns<\/a> exist in health care<\/a>, online marketing<\/a> and even criminal justice<\/a>. My own interest in this area began when a research group I was part of discovered gender bias in how online ads were targeted<\/a>, but could not explain why it happened.<\/p>\n

All those industries, and many others, who use machine learning to analyze processes and make decisions have a little over a year to get a lot better at explaining how their systems work. In May 2018, the new European Union General Data Protection Regulation<\/a> takes effect, including a section giving people a right to get an explanation for automated decisions that affect their lives. What shape should these explanations take, and can we actually provide them?<\/p>\n

Identifying key reasons<\/strong><\/p>\n

One way to describe why an automated decision came out the way it did is to identify the factors that were most influential in the decision. How much of a credit denial decision was because the applicant didn\u2019t make enough money, or because he had failed to repay loans in the past?<\/p>\n

My research group at Carnegie Mellon University, including PhD student Shayak Sen and then-postdoc Yair Zick created a way to measure the relative influence<\/a> of each factor. We call it the Quantitative Input Influence.<\/p>\n

In addition to giving better understanding of an individual decision, the measurement can also shed light on a group of decisions: Did an algorithm deny credit primarily because of financial concerns, such as how much an applicant already owes on other debts? Or was the applicant\u2019s ZIP code more important \u2013 suggesting more basic demographics such as race might have come into play?<\/p>\n

Capturing causation<\/strong><\/p>\n

When a system makes decisions based on multiple factors it is important to identify which factors cause the decisions and their relative contribution.<\/p>\n

For example, imagine a credit-decision system that takes just two inputs, an applicant\u2019s debt-to-income ratio and her race, and has been shown to approve loans only for Caucasians. Knowing how much each factor contributed to the decision can help us understand whether it\u2019s a legitimate system or whether it\u2019s discriminating.<\/p>\n

An explanation could just look at the inputs and the outcome and observe correlation \u2013 non-Caucasians didn\u2019t get loans. But this explanation is too simplistic. Suppose the non-Caucasians who were denied loans also had much lower incomes than the Caucasians whose applications were successful. Then this explanation cannot tell us whether the applicants\u2019 race or debt-to-income ratio caused the denials.<\/p>\n

Our method can provide this information. Telling the difference means we can tease out whether the system is unjustly discriminating or looking at legitimate criteria, like applicants\u2019 finances.<\/p>\n

To measure the influence of race in a specific credit decision, we redo the application process, keeping the debt-to-income ratio the same but changing the race of the applicant. If changing the race does affect the outcome, we know race is a deciding factor. If not, we can conclude the algorithm is looking only at the financial information.<\/p>\n

In addition to identifying factors that are causes, we can measure their relative causal influence on a decision. We do that by randomly varying the factor (e.g., race) and measuring how likely it is for the outcome to change. The higher the likelihood, the greater the influence of the factor.<\/p>\n

Aggregating influence<\/strong><\/p>\n

Our method can also incorporate multiple factors that work together. Consider a decision system that grants credit to applicants who meet two of three criteria: credit score above 600, ownership of a car, and whether the applicant has fully repaid a home loan. Say an applicant, Alice, with a credit score of 730 and no car or home loan, is denied credit. She wonders whether her car ownership status or home loan repayment history is the principal reason.<\/p>\n

An analogy can help explain how we analyze this situation. Consider a court where decisions are made by the majority vote of a panel of three judges, where one is a conservative, one a liberal and the third a swing vote, someone who might side with either of her colleagues. In a 2-1 conservative decision, the swing judge had a greater influence on the outcome than the liberal judge.<\/p>\n

The factors in our credit example are like the three judges. The first judge commonly votes in favor of the loan, because many applicants have a high enough credit score. The second judge almost always votes against the loan because very few applicants have ever paid off a home. So the decision comes down to the swing judge, who in Alice\u2019s case rejects the loan because she doesn\u2019t own a car.<\/p>\n

We can do this reasoning precisely by using cooperative game theory<\/a>, a system of analyzing more specifically how different factors contribute to a single outcome. In particular, we combine our measurements of relative causal influence with the Shapley value<\/a>, which is a way to calculate how to attribute influence to multiple factors. Together, these form our Quantitative Input Influence measurement.<\/p>\n

So far we have evaluated our methods on decision systems that we created by training common machine learning algorithms with real world data sets. Evaluating algorithms at work in the real world is a topic for future work.<\/p>\n

An open challenge<\/strong><\/p>\n

Our method of analysis and explanation of how algorithms make decisions is most useful in settings where the factors are readily understood by humans \u2013 such as debt-to-income ratio and other financial criteria.<\/p>\n

However, explaining the decision-making process of more complex algorithms remains a significant challenge. Take, for example, an image recognition system, like ones that detect and track tumors<\/a>. It is not very useful to explain a particular image\u2019s evaluation based on individual pixels. Ideally, we would like an explanation that provides additional insight into the decision \u2013 such as identifying specific tumor characteristics in the image. Indeed, designing explanations for such automated decision-making tasks is keeping many researchers busy<\/a>.<\/p>\n

Author Bio:<\/strong> Anupam Datta is an Associate Professor of Computer Science and Electrical and Computer Engineering at Carnegie Mellon University <\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

People who apply for a loan from a bank or credit card company, and are turned down, are owed an explanation of why that happened. It\u2019s a good idea \u2013 because it can help teach people how to repair their damaged credit \u2013 and it\u2019s a federal law, the Equal Credit Opportunity Act. Getting an […]\n… learn more→<\/a><\/p>","protected":false},"author":33,"featured_media":38190,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[85],"tags":[65898],"coauthors":[11733],"class_list":["post-38189","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-social-change","tag-artificial-intelligence-systems"],"acf":[],"yoast_head":"\nDid artificial intelligence deny you credit? - World leading higher education information and services<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/world.edu\/artificial-intelligence-deny-credit\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:locale\" content=\"en_US\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:type\" content=\"article\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:title\" content=\"Did artificial intelligence deny you credit? - World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:description\" content=\"People who apply for a loan from a bank or credit card company, and are turned down, are owed an explanation of why that happened. It\u2019s a good idea \u2013 because it can help teach people how to repair their damaged credit \u2013 and it\u2019s a federal law, the Equal Credit Opportunity Act. Getting an […]\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:url\" content=\"https:\/\/world.edu\/artificial-intelligence-deny-credit\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:site_name\" content=\"World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"article:published_time\" content=\"2017-03-14T20:52:59+00:00\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:image\" content=\"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:width\" content=\"680\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:height\" content=\"408\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"author\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" class=\"yoast-seo-meta-tag\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/\"},\"author\":{\"name\":\"Kevin\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"headline\":\"Did artificial intelligence deny you credit?\",\"datePublished\":\"2017-03-14T20:52:59+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/\"},\"wordCount\":1150,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/robots.jpeg\",\"keywords\":[\"artificial intelligence systems\"],\"articleSection\":[\"Social Change\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/\",\"url\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/\",\"name\":\"Did artificial intelligence deny you credit? - World leading higher education information and services\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/robots.jpeg\",\"datePublished\":\"2017-03-14T20:52:59+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#primaryimage\",\"url\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/robots.jpeg\",\"contentUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2017\\\/03\\\/robots.jpeg\",\"width\":680,\"height\":408},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/world.edu\\\/artificial-intelligence-deny-credit\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/world.edu\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Did artificial intelligence deny you credit?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/world.edu\\\/#website\",\"url\":\"https:\\\/\\\/world.edu\\\/\",\"name\":\"World leading higher education information and services\",\"description\":\"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/world.edu\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\",\"name\":\"Kevin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"caption\":\"Kevin\"},\"description\":\"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.\",\"sameAs\":[\"https:\\\/\\\/world.edu\"],\"url\":\"https:\\\/\\\/world.edu\\\/author\\\/kevin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Did artificial intelligence deny you credit? - World leading higher education information and services","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/","og_locale":"en_US","og_type":"article","og_title":"Did artificial intelligence deny you credit? - World leading higher education information and services","og_description":"People who apply for a loan from a bank or credit card company, and are turned down, are owed an explanation of why that happened. It\u2019s a good idea \u2013 because it can help teach people how to repair their damaged credit \u2013 and it\u2019s a federal law, the Equal Credit Opportunity Act. Getting an […]","og_url":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/","og_site_name":"World leading higher education information and services","article_published_time":"2017-03-14T20:52:59+00:00","og_image":[{"width":680,"height":408,"url":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg","type":"image\/jpeg"}],"author":"Kevin","twitter_misc":{"Written by":"Kevin","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#article","isPartOf":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/"},"author":{"name":"Kevin","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"headline":"Did artificial intelligence deny you credit?","datePublished":"2017-03-14T20:52:59+00:00","mainEntityOfPage":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/"},"wordCount":1150,"commentCount":0,"image":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg","keywords":["artificial intelligence systems"],"articleSection":["Social Change"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/","url":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/","name":"Did artificial intelligence deny you credit? - World leading higher education information and services","isPartOf":{"@id":"https:\/\/world.edu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#primaryimage"},"image":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg","datePublished":"2017-03-14T20:52:59+00:00","author":{"@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"breadcrumb":{"@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/world.edu\/artificial-intelligence-deny-credit\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#primaryimage","url":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg","contentUrl":"https:\/\/world.edu\/wp-content\/uploads\/2017\/03\/robots.jpeg","width":680,"height":408},{"@type":"BreadcrumbList","@id":"https:\/\/world.edu\/artificial-intelligence-deny-credit\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/world.edu\/"},{"@type":"ListItem","position":2,"name":"Did artificial intelligence deny you credit?"}]},{"@type":"WebSite","@id":"https:\/\/world.edu\/#website","url":"https:\/\/world.edu\/","name":"World leading higher education information and services","description":"Global Higher Education Jobs, University jobs, Lecturer & tenure jobs.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/world.edu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee","name":"Kevin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4","url":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","caption":"Kevin"},"description":"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.","sameAs":["https:\/\/world.edu"],"url":"https:\/\/world.edu\/author\/kevin\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/38189","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/comments?post=38189"}],"version-history":[{"count":0,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/38189\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media\/38190"}],"wp:attachment":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media?parent=38189"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/categories?post=38189"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/tags?post=38189"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/coauthors?post=38189"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}