Working with others helped Man to survive, but he had to develop a brain big enough to cope with all the social complexities, they believe.
In a computer model, the team simulated the human brain, allowing a network of neurons to evolve in response to a series of social challenges.
There were two scenarios. The first entailed two partners in crime who had been caught by the police, each having to decide whether or not to inform on the other.
The second had two individuals trapped in a car in a snowdrift and having to weigh whether to cooperate to dig themselves out or just sit back and let the other do it.
In both cases, the individual would gain more from selfishness.
But the researchers were intrigued to find that as the brain evolved, the individual was likelier to choose to cooperate.
"We cooperate in large groups of unrelated individuals quite frequently, and that requires cognitive abilities to keep track of who is doing what to you and change your behaviour accordingly," co-author Luke McNally of Dublin's Trinity College told AFP.
McNally pointed out, though, that cooperation has a calculating side. We do it out of reciprocity.
"If you cooperate and I cheat, then next time we interact you could decide: 'Oh well, he cheated last time, so I won't cooperate with him.' So basically you have to cooperate in order to receive cooperation in the future."
McNally said teamwork and bigger brainpower fed off each other.
"Transitions to cooperative, complex societies can drive the evolution of a bigger brain," he said.
"Once greater levels of intelligence started to evolve, you saw cooperation going much higher."
The study appears in Proceedings of the Royal Society B, a journal published by Britain's de-facto academy of sciences.
Commenting on the paper, Robin Dunbar, an evolutionary anthropologist at Oxford University, said the findings were a valuable addition to understanding brain evolution.
But he said there were physiological limits to cooperation.
Man would need a "house-sized brain" to take cooperation to a perfect level on a planet filled with humans.
"Our current brain size limits the community size that we can manage ... that we feel we belong to," he said.
Our comfortable "personal social network" is limited to about 150, and boosting that to 500 would require a doubling of the size of the brain.
"In order to create greater social integration, greater social cohesion even on the size of France, never mind the size of the EU, never mind the planet, we probably have to find other ways of doing it" than wait for evolution, said Dunbar.