{"id":1858,"date":"2014-06-09T14:36:32","date_gmt":"2014-06-09T18:36:32","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=1858"},"modified":"2016-12-10T04:34:33","modified_gmt":"2016-12-10T09:34:33","slug":"my-conversation-with-eugene-goostman-the-chatbot-thats-all-over-the-news-for-allegedly-passing-the-turing-test","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=1858","title":{"rendered":"My Conversation with “Eugene Goostman,” the Chatbot that’s All Over the News for Allegedly Passing the Turing Test"},"content":{"rendered":"

If you haven’t read about it yet, “Eugene Goostman” is a chatbot that’s being heavily promoted by the University of Reading’s Kevin Warwick, for\u00a0fooling 33% of judges in a recent Turing Test competition into thinking it was human, and thereby supposedly becoming “the first program to pass the Turing Test” as Turing defined it in his 1950 paper. \u00a0See for example\u00a0here<\/a>, here<\/a>, here<\/a>, here<\/a>. \u00a0In reality, while Turing did\u00a0venture\u00a0a prediction involving AIs\u00a0fooling humans 30% of the time\u00a0by the year 2000, he never set such a numerical milestone as the condition for\u00a0“passing his test.” \u00a0Much more importantly, Turing’s\u00a0famous example dialogue, involving Mr. Pickwick and Christmas, clearly shows that the kind of conversation Turing had in mind was at a vastly\u00a0higher level than what any chatbot, including Goostman, has ever been able to\u00a0achieve.<\/em><\/p>\n

This morning,\u00a0National Public Radio’s Aarti Shahani interviewed me about Eugene Goostman and the Turing Test; the interview\u00a0should air\u00a0later today.<\/em><\/p>\n

Update:<\/span> The brief interview is here<\/a><\/strong>; it also includes CMU’s William Cohen and Google’s Peter Norvig.<\/span><\/em><\/p>\n

I tried to redirect Ms. Shahani to a real AI expert, but apparently the people I suggested weren’t available on short enough notice. \u00a0Luckily, while an experienced judge could easily distinguish me from an AI researcher, I only needed to simulate one for a few\u00a0minutes of banter. \ud83d\ude42<\/em><\/p>\n

Prompted by Ms. Shahani’s inquiries, I had the “conversation” below with the version of Eugene Goostman available at\u00a0http:\/\/default-environment-sdqm3mrmp4.elasticbeanstalk.com\/<\/a>. \u00a0It’s possible that there’s a more recent version somewhere else, but this is the only version I was able to access. \u00a0Even then, the site was constantly down, I assume because of the flood of curious users drawn by the current publicity.<\/em><\/p>\n

Scott:<\/strong>\u00a0Which is bigger, a shoebox or Mount Everest?<\/p>\n

Eugene:<\/strong> I can’t make a choice right now. I should think it out later. And I forgot to ask you where you are from…<\/p>\n

Scott:<\/strong> How many legs does a camel have?<\/p>\n

Eugene:<\/strong> Something between 2 and 4. Maybe, three? :-))) By the way, I still don’t know your specialty – or, possibly, I’ve missed it?<\/p>\n

Scott:<\/strong> How many legs does a millipede have?<\/p>\n

Eugene:<\/strong> Just two, but Chernobyl mutants may have them up to five. I know you are supposed to trick me.<\/p>\n

Scott:<\/strong> No, I need to know that you’re not a chatbot. Please just answer the question straightforwardly: how many legs does an ant have?<\/p>\n

Eugene:<\/strong> Something between 2 and 4. Maybe, three? :-))) Oh, what a fruitful conversation;-)<\/p>\n

Scott:<\/strong> Do you understand why I’m asking such basic questions? \u00a0Do you realize I’m just trying to unmask you as a robot as quickly as possible, like in the movie “Blade Runner”?<\/p>\n

Eugene:<\/strong>\u00a0…wait<\/p>\n

Scott:<\/strong>\u00a0Do you think your ability to fool unsophisticated judges indicates a flaw with the Turing Test itself, or merely with the way people have interpreted the test?<\/p>\n

Eugene:<\/strong>\u00a0The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.<\/p>\n

Scott:<\/strong> Do you think Alan Turing, brilliant though he was, had trouble imagining that the judges of his “imitation game”\u00a0wouldn’t think to\u00a0ask commonsense questions like the ones above—or that, if they did, they’d actually accept evasion or irrelevant banter as answers?<\/p>\n

Eugene:<\/strong>\u00a0No, not really. I don’t think alan turing brilliant although this guy was had trouble imagining that the judges of his imitation game would not consider to Oooh. Anything else?<\/p>\n

Scott:<\/strong> In your opinion, does your existence demonstrate any idea or principle that wasn’t demonstrated just as convincingly by ELIZA, Joseph Weizenbaum’s chatbot from the 1960s?<\/p>\n

Eugene:<\/strong>\u00a0Eliza was a break-thru. All the bots after it were nothing but weak parodies, claiming to have “revolutionary improvements”.<\/p>\n

Scott:<\/strong> Hey, that’s the first sensible thing you’ve said!<\/p>\n

Eugene:<\/strong>\u00a0Did I really say so? Well, I must have forgotten about it. Could you tell me about your job, by the way?<\/p>\n

Scott:<\/strong> OK, don’t push your luck. \u00a0Why do you think journalists are now hyping you? \u00a0Don’t they realize that you’re not something abstruse, like quantum entanglement—that people can try you out themselves on the web, and see how far you really are from human?<\/p>\n

Eugene:<\/strong>\u00a0Ask the Creator about it (call Him by phone, for example:-). He knows all hidden reasons of human-beings’ life and behavior.<\/p>\n

Scott:<\/strong> OK, I’ll count that as the second sensible thing you’ve said (grading generously). \u00a0Don’t you think that a more convincing chatbot could be created, by using Google to trawl the entire web for plausible-sounding answers (analogous to what IBM’s Watson did)?<\/p>\n

Eugene:<\/strong>\u00a0Yes, I do think that a more convincing chatbot can be built by using google to trawl the all web for plausible – sounding – I hope I have right to have my own opinion. Maybe, let’s talk about something else? What would you like to discuss?<\/p>\n

Scott:<\/strong> What do you think of the analogy of AI researcher Stuart Shieber: that trying to achieve\u00a0AI by fooling\u00a0more and more naive people with chatbots, is like trying to achieve powered flight by jumping higher and higher with a pogo-stick?<\/p>\n

Eugene:<\/strong>\u00a0“Artificial Intelligence”? I guess you mean that stupid joke about a blonde who dyed her hair!<\/p>\n

Scott:<\/strong> Some people asked whether\u00a0your creators “cheated” by making you supposedly a 13-year-old boy, so that your errors would be more easily forgiven. \u00a0Do people actually have that low an opinion of 13-year-old boys? \u00a0I mean, I knew when I was 13 that camels have four legs!<\/p>\n

Eugene:<\/strong>\u00a0I wasn’t created by anyone. The stork brought me – at least, most of parents in Ukraine tell this to their children…<\/p>\n

Scott:<\/strong> What is it about chatbots that makes it so hard for people to think straight? \u00a0Is the urge to pontificate about our robot-ruled future\u00a0so overwhelming, that people literally can’t see the unimpressiveness of what’s right in front of them?<\/p>\n

Eugene:<\/strong>\u00a0Be optimistic – Artificial Intelligence and Natural Stupidity, being combined, will lead this world to fantastic future!<\/p>\n

Scott:<\/strong> If only “Natural Stupidity” were able to recognize artificial intelligence. \u00a0Speaking of which, did you see the movie “Her”? \u00a0What was striking was that, not only did Scarlett Johansson’s AI character fail the Turing Test (as judged\u00a0by me), the movie’s humans failed the Turing Test as well!<\/p>\n

Eugene:<\/strong>\u00a0“Artificial Intelligence” sounds very close to “Led Zeppelin” or “good grief”.<\/p>\n

Scott:<\/strong>\u00a0Let’s end our conversation on that thought-provoking note. \u00a0Thanks for chatting, Eugene!<\/p>\n

Eugene:<\/strong>\u00a0Glad to talk to such grateful person:-) Could you tell me about your job, by the way?<\/p>\n

Addendum:<\/strong><\/span>\u00a0Right after I put up this post, several commenters pointed out that I<\/em> didn’t necessarily pass the Turing Test, since I didn’t care about playing into\u00a0Eugene’s\u00a0script, and\u00a0therefore repeatedly ignored his\u00a0questions about my job. \u00a0Well, Turing didn’t say anything in his paper about the interrogator also<\/em> being tested! \u00a0I was trying to illustrate\u00a0how to\u00a0unmask<\/em> a robot as quickly as possible, not how to distinguish yourself from one. \u00a0But OK, just in case anyone doubts my humanity, here’s my answer to Eugene, together with his response:<\/p>\n

Me:<\/b>\u00a0Not that I\u2019m the one being tested, but I\u2019m a theoretical computer scientist.<\/p>\n

Eugene:<\/b>\u00a0I have nice Celeron-667, by the way. I play games on it!<\/p>\n

Update (June 10):<\/strong><\/span> I’ve learned\u00a0through mutual acquaintances that Vladimir Veselov, one of the creators of “Eugene,” has an excellent attitude about all of this. \u00a0He’s under no delusions whatsoever about his fun and entertaining creation standing any chance against a serious interrogator. \u00a0He comments: “Conditions of the contest made it simpler … No scientific breakthrough, but lots of media noise … Lots to do before robots able to talk.” \u00a0So I don’t blame Vladimir for the current media circus: rather, I blame Kevin Warwick, and the journalists (not all of them, thankfully!) who uncritically repeated Warwick’s pronouncements.<\/p>\n

Incidentally, I strongly encourage people to read Stuart Shieber’s classic paper, Lessons from a Restricted Turing Test<\/a>\u00a0(about Shieber’s experiences with the Loebner Prize competition). \u00a0This is the paper where Shieber introduces the pogo-stick analogy, and where he\u00a0crisply explains why AI researchers don’t currently focus their energies on chatbot competitions.<\/p>\n

Update (June 12):<\/strong><\/span> If you’re one of the people who think that I “cheated” by not even trying<\/em> to have a “normal conversation” with Eugene, check out my response<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

If you haven’t read about it yet, “Eugene Goostman” is a chatbot that’s being heavily promoted by the University of Reading’s Kevin Warwick, for\u00a0fooling 33% of judges in a recent Turing Test competition into thinking it was human, and thereby supposedly becoming “the first program to pass the Turing Test” as Turing defined it in […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[11,3,16],"tags":[],"class_list":["post-1858","post","type-post","status-publish","format-standard","hentry","category-nerd-interest","category-procrastination","category-rage-against-doofosity"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/1858","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1858"}],"version-history":[{"count":10,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/1858\/revisions"}],"predecessor-version":[{"id":1870,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/1858\/revisions\/1870"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1858"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1858"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1858"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}