Amongst the various domains of Artificial Intelligence (AI) research being advanced at the moment, one domain has become critical to the broader success of artificial intelligence, while still in its infancy. It’s called multi-agent reinforcement learning.
Multi-agent reinforcement learning is the study of numerous artificial intelligence agents cohabitating in an environment, often collaborating toward some end goal. When focusing on collaboration, it derives inspiration from other social structures in the animal kingdom. It also draws heavily on game theory.
Multi-agent systems are not just a research method, they can be used to model many complex problems of today’s society, such as urban and air traffic control (Agogino & Tumer, 2012), multi-robot coordination (Ahmadi & Stone, 2006; Claes, Hennes, Tuyls, & Meeussen, 2012), distributed sensing (Mihaylov, Tuyls, & Now ́e, 2014), and energy distribution (Pipattanasomporn, Feroze & Rahman, 2009).
Despite all of these useful applications, there are more critical reasons for the research community to focus on multi-agent reinforcement learning:
the role of sociability in increasing a species intelligence
for the safe development of artificial intelligence
Social Life and Brain Size
Sociability - spending time with family, colleagues and strangers - is cognitively demanding. The neurological functions that promote living in societies - empathy, communication, hierarchy comprehension, all demand significant cognitive reserves. It is often taken for granted because humans do it so effortlessly, but the ability to empathise with one’s peer and to comprehend they’re in danger involves staggeringly complex neurological function. To comprehend that your friend is in danger of being hit by a car as they step out onto the road, and to yell to them to warn them, is a truly impressive cognitive collaboration of empathy, causal understanding, communication and other functions. This isn’t just applicable to human society either, when you put aside our own sense of superiority, the neurological complexity of other primates and avian communication systems are admirable and intricate. Many primates and other species warn each other of predatory threats and new food sources, knowing the benefit of community.
Similarly, group hierarchies are an intellectually demanding approach to social living that many species adopt. Humans, other primates, and birds have developed group hierarchies to help navigate the rich ecosystem of social living. (Though one should note that not all social species tend toward hierarchies). A hierarchy can best be understood as a ranking system that helps manage unequal access to limited resources. From food and water, to more abstract human concepts such as money and prestige. It helps us in that we don’t have to fight over the food everytime we sit down to eat. It helps us manage systems such as airline seats, where money is utilised to maintain a seating hierarchy between first class, business, premium economy and economy.
This ushers in a key point - social cooperation and hierarchies are extremely challenging, and this is reflected in the brain. The British Anthropologist Robin Dunbar has shown that across various taxa (e.g. ‘birds’, ‘ungulates’ or ‘primates’) the bigger the average size of the social group in the species, the larger the brain relative to total body size. Secondly, the larger the neocortex, relative to total brain size. Dunbar’s influential ‘social brain hypothesis’ posits that increases in social complexity and the evolutionary expansion of the neocortex has been linked.
This link also occurs within a species. Among some primates, group size can vary tenfold (depending on the richness of the ecosystem). This was modeled in a fascinating neuroimaging study, in which captive macaque monkeys were housed in different sized groups. The bigger the group, the more thickening of the prefrontal cortex and the superior temporal gyrus, a cortical region involved in Theory of Mind, the tighter the activity coupling between the two.
This variety of richness in social ecosystems results in multi-agent spectrum in the animal kingdom: tigers are entirely individualistic, only interacting with other tigers to mate or fight (sometimes both), and ants or bees are entirely motivated by the collective, giving their lives for the group. Our own species sits somewhere in the middle -, we have individual motivations and social motivations. In different circumstances these competing motivations take priority.
What does this have to do with Artificial Intelligence?
If we’re working toward advancing artificial intelligence, it reasonable to posit that the same sort of sociability that is so fundamental to our own intelligence will be important to advancing artificial intelligence.
The work Remi AI have been undertaking in this domain has been primarily focused around either artificial intelligence controls in business operations, i.e. inventory management, fleet optimisation etc. and also language development between different agent systems.
There are many major challenges that have been identified throughout our work, though the largest at this early stage is associating reward with actions.
To illustrate the difficulty of this problem, when designing a single agent to play Space Invaders, it is relatively simple to create a reward and punishment system: the reward is associated in the increase in game points and the punishment is associated with the death of the agent's spacecraft.
Whereas with multi-agent systems it can be difficult to clearly associate a reward with a particular action by a particular agent. For example, if you built a transport simulation, with 10 different agents driving a bus apiece, you could trial rewarding the agents collectively for reducing wait time across all passengers. It is incredibly difficult for the agents to map their own actions to a collective reduced wait time, as the agents can struggle to attribute which action lead to the reduction. This problem increases exponentially with the number of agents. When you factor in the city of London has 8000 buses, you begin to get an understanding of the complexity required.
Multi-agent reinforcement learning and safety.
The majority of modern day Artificial Intelligent systems are, in cognitive terms, still fairly naive and unintelligent.
The supervised methods that are making waves such as deep-learning are analogous, in biological terms, to the development of the visual and auditory cortexes - present in the most many of the simplistic organisms that evolved millions of years ago. State of the art reinforcement learning systems are still largely learning through trial and error, (though we’re making advances in imitation and transfer learning which bring our agent’s learning more similar to the way animals learn from their parents and other tasks respectively).
Although it is incredibly difficult to obtain a consensus around scales of animal intelligence, it is widely considered in A.I research circles that we still haven’t devised an A.I that is as generally intelligent as a rat.
But as our agents become more intelligent as our research advances, multi-agent reinforcement learning will become critical, not only for the development of communication, empathy and other fundamental intellectual capabilities, but because it will teach the agents h