Computational Cognition Ideation Challenge

Computational Cognition Ideation Challenge Organizers Kevin A. Gluck (kevin.gluck@us.af.mil) Vladislav D. Veksler (vladislav.veksler.ctr@us.af.mil) Norbou Buchler (norbou.buchler@us.army.mil) Troy Kelley (tkelley@us.army.mil) Air Force Research Laboratory, USA Army Research Laboratory, USA Paul Bello (paul.bello@navy.mil) James Donlon (jdonlon@nsf.gov) Office of Naval Research, USA National Science Foundation, USA Alonso Vera (alonso.h.vera@nasa.gov) Bob Lee (bob.lee@wbi-icc.com) National Aeronautics and Space Administration, USA Wright Brothers Institute, USA Keywords: Artificial Intelligence; Cognitive Computational Systems; Competition; Challenge Science; Comparison Challenge (Lebiere, Gonzalez, & Warwick, 2010) involved a competitive evaluation of computational systems implemented somewhere in the intersection of AI and cognitive modeling. Both of these were successful and interesting activities. However, they were also both single shot modeling competitions that did not evolve into annually recurring events. An open, recurring competition would provide, it is hoped, a visible measure of scientific and technological progress in understanding and implementing a computational instantiation of the mind. Preliminary input and feedback from the scientific community (Gluck, 2012) indicates resounding interest in such a competition. Agreement and enthusiasm regarding a competition as a mechanism for pursuing this ill-defined goal is all fine and good, but does not in itself provide us with a concrete focus and set of evaluation metrics. There are many places we could focus the attention of an annual competition. We must ask the question, though: Where should we focus our attention, in order to maximize the positive effect such a competition can have on cognitive science and technology? Introduction Across a diverse assortment of topics, competitions have been established to motivate and focus scientists and engineers, as well as students interested in such careers, on ambitious objectives. Examples include the famous RoboCup robotic soccer tournaments, the International Conference on Functional Programming (ICFP) contests, and the High Performance Computing (HPC) challenges. The National Institute of Standards and Technology (NIST) speaker recognition evaluations, although officially considered not to be competitions, also provide intriguing examples of “crowd-sourced” improvements in methodology and capability. Whether we refer to them as competitions, challenges, tournaments, or evaluations, the purpose these recurring annual events serve is to create opportunities for a broad cross-section of interested and capable people to participate in the process of accelerating scientific and technological progress. The organizers of this symposium have rallied around the premise that it is time to establish one or more recurring annual competitions for the cognitive sciences. Given our particular areas of expertise and interests, at this time we are specifically targeting the intersection of cognitive modeling and artificial intelligence. There is a rich history of interaction between these disciplines, but in recent years (or decades) the tendency has been toward increasing fractionation and sub-specialization, with decreasing methodological, scientific, and technological cross- fertilization and integration. Consistent with the spirit of the recent AAAI Fall Symposium on Integrated Cognition and with the far-reaching goals motivating the co-location of the 2014 CogSci and AAAI conferences, our intent is to use competition as a mechanism for bringing these communities together, to advance computational cognitive science and technology. There are precedents for attempting this kind of thing in the past. For instance, the PokerBot Competition (Lebiere & Bothell, 2004) and the Dynamic Stocks and Flows Model An Ideation Challenge In this symposium, we will present for general discussion and debate the outcome of what is called an Ideation Challenge. The Challenge is a general call to the scientific community to provide answers to the question at the end of the introduction: where should we focus the attention of an annually recurring computational cognition competition? In challenges of this sort, a “Seeker,” which in this case is the consortium of organizers of this symposium, posts a public challenge made available to “Solvers” all over the world. By adopting this approach, we are casting a wide net across the global community, soliciting proposals focused on the creation of a recurring annual competition in computational cognition. Our hope is that this will motivate diverse groups to bring their respective strengths together to create compelling new capabilities that are achievable only through multi-disciplinary, integrative cognitive science and technology.