The last year or so I’ve been leading a small team of developers. We’ve been working on a project that involves genomics and molecular biology, bioinformatics, oncology and computional biology. Saying that it’s hugely complex is an understatement. One of the interesting dynamics of this project was that I personally designed and implemented a large portion of the project. A big part of the project was also to solve the actual problem – our client did not have a solution when we came to them, so really we ended up getting access to resources and experts, but no predefined solution.
The question that I’ve been pondering the last week or so is this – if the project had been even more complex; so complex that I wouldn’t have been able to fit it all in my head – could we still have solved it? If I had 50% of the information in my head and someone else had the other 50%, would it still be possible to come up with a working solution?
My intuition is that the answer to this is no. I don’t think a problem of this complexity level could have been solved in a good way by sharing the responsibility for coming up with the solution.
On the plus side, I have never encountered anything even close to this magnitude of complexity before. The projects I’ve been on have been a variety of different enterprise style projects and most of them doesn’t really have much in terms of domain complexity. So maybe this is not a problem in practice for most cases.
But on the other hand, we still have lots of unsolved problems in highly complex and scientific domains. In order to solve them, we need people that can understand both the science and the software aspects at the same time. And based on my experience the last year, I suspect that there are real limits in what kinds of problems we can actually take on in this way. There has to be a better solution. I don’t think we have a solution to this problem yet. Incremental development methodologies really doesn’t help for this.
Another interesting aspect of this project is that we did not have any BAs (business analysts). Most of our projects have BAs and it’s highly unusual to not have them. In retrospect it was the right choice for us and I can now verbalize why – when you have BAs working with the domain, you still have to take into account the communication with the developers and tech leads. If the domain is complex enough and the developers need to have that understanding, having BAs would actually get in the way and the communication surface area would be to large to effectively work. Me and one of my colleagues ended up together doing all the BA work in conjunction with our design and implementation work.
Working on a project like this has been a very different experience. It’s definitely clear to me that our standard ways of working with businesses doesn’t really apply.
2 Comments, Comment or Ping
“My intuition is that the answer to this is no. I don’t think a problem of this complexity level could have been solved in a good way by sharing the responsibility for coming up with the solution.”
I disagree because I believe there are good counter examples. Take the moon landings, the discovery of DNA, the engineering of the A380, the LHC, and so on.
I think collaboration and communication improves our ability to reason and solve complex problems. Having just one person hold the whole problem risks them believing they *can* know everything about a domain.
I do think really complex problems sometimes need someone to go off and think really hard for a while, and there may need to be an overall design authority to keep things coherent, but I think to say the limiting factor is what one person can keep in their head is a rather limiting and individualistic approach.
November 29th, 2012
Good question that you’ve posted, and one that will characterize many of the leading investigations – now that the “easy problems” have been largely solved, and the large ones do seem to require sustained and long-term team efforts.
Ian makes a very good point, offering the examples of the NASA endeavors, etc.
Years ago, I worked in a research environment in which two of the lead researchers were a husband and wife team. They told me that they spent the first several years just learning each other’s technical language and point of reference. This point has stayed with me.
The kinds of challenges that you describe – both as scoping the problem and defining the approach, and building the computational infrastructure, are definitely team-types of problems.
I suggest that the focus be on the long-term, understanding that most team members will invest a lot of time just in learning about the problem and each others’ perspectives/languages/unique skill sets – what the others bring to the project.
Big problems will indeed be larger than one person’s brain, even though one person (or two or three) may be the leads. Some specialized knowledge will always be in the brains of a few core people; perhaps not the team leads, but essential nevertheless.
Team stability – often translating into a long-term vision and sustained funding – are key.
Remember that when NASA solved the moon landing problem, there was a clear and coherent vision, and ample funding. This created the environment in which the challenge was solved.
Also, consider the timeframe: from May, 1961, when President JFK announced the moon landing goal, to the first landing in 1969 – nine years of sustained, focused activity on the part of several teams and industries that had to work together with NASA.
Also, the goal was sufficiently ambitious and heroic enough to attract the best-of-the-best in engineering talent, and cause them to hugely dedicate their lives and their emotional energies to working together to achieve this goal.
That may be a crucial component – that the challenge has to engage the participants at a visceral level, getting their whole-being buy-in, not just as an intellectual pursuit.
Glad you’ve raised this question in your post, and look forward to seeing more from you.
September 14th, 2014
Reply to “Complexity and Brain Size”