Scaling a project

I recently read an article from a local “expert” about scalability of a dev team. The author claimed that every project can only be completed by a team of 7-10 people. The size of the project didn’t matter to the author. The limit is 10 people. No exceptions.

As I read it, I thought maybe he was trying to be ironic or maybe he would do a reversal for theatrical effect.  No. He simply tried to build a case for why he was correct by listing some large projects which had failed.  I guess he was not aware of any successful dev teams who are larger than 10 people.

As I thought further about his statement, I recalled a conversation that I had with a manager, earlier this year. He asked me how many people I could manage. I started with a similar answer; I could directly manage 7-10 people, but if we needed to scale beyond that, we would need to change our team structure and project structure. Just because I can’t personally manage more people, does not mean that our project needed to be limited by me.

Science of Scalability

During my senior year at U Mich, I took a class called “distributed processing”. It is a fascinating topic. I enjoyed it so much that I have studied it my whole career with hopes of applying it as often as possible. As it turns-out, the concepts are applicable to people, and other things beyond just computer processing.

Basics

In distributed processing, a huge hurdle to overcome, is the fact that distributed processing is not practical for all situations. To effectively know whether distributed processing, is the right answer, you must first assess the following:

  1. A task must be divided into sub-tasks that can be distributed. Some tasks easily lend themselves to this concept and some are impossible (or close). Determine the effort to sub-divide your task. Add that to your overall processing time/cost.
  2. After the distributed tasks have been processed, the results need to be merged. Determine the effort to collect and merge the results. Add that to your overall processing time/cost.
  3. If some tasks take longer or shorter, you need to wait for them all to complete, before you can continue. Therefore, you will never be faster than your slowest task.

Once you sum these factors, you can see that it is possible to take longer to [split-up a task, send it somewhere else, get the results and merge it], than it would have taken if you just did it locally.

Therein lies the challenge for distributed processing. Finding processes that can be distributed effectively.  The easy answer is “Don’t bother. It isn’t worth it”, but that is not always true.

Sometimes, there is more than one approach for each of these steps. If you are aware of each different approach, you can evaluate which one will give you optimal performance for your situation. YMMV.  This reveals the bigger challenge.  Your first guess might be wrong, but there might also be a better answer and you haven’t found it yet.

Distributed Processing for Projects

The US Army applies this philosophy every day. Without this, an army could only scale beyond “one big bunch of fellas”. You can imagine how ineffective this would be. Clearly when you need a bigger team, you need to determine a way to scale-out your people.

The key to keeping this all working effectively is organizing them. Everyone knows that this is the value of good “management”.

I suppose some people don’t get it, because they focus on doing their own job well. They have no idea what other people do, unless they do something for you (like HR). Most people rarely (or never) ask a manager to do something for them. So their only interaction with a manager is frequently receiving more work and being asked to do more work, faster. If you have never taken the time to look beyond your work-load, you may never understand why your boss does this. In fact, you will probably be frequently irritated by your boss and might even become spiteful.

Applying it to Your Project

The solution is to realize that management is “a system”. You can learn how these things work and why. You can see how different ones work and what makes them more effective (or less effective) under different circumstances. This is the same thing a SW architect is supposed to do; understand systems, components and options. Determining how to scale a project is like learning how to scale a system.  Learn the parts, know how/when to apply them.  Know how to measure their effectiveness.  Apply, measure, adjust.  The concept is the same.

When you learn this, you will see that management is just another component that you can add into your project, or change as needed. So, to scale your project, a) acquire the necessary skills, or b) get someone who is already good at it.
Don’t be a bottle-neck.  Be ready to overcome limitations. Grow (personally), and scale yourself and your project as needed.

Advertisements

About Tim Golisch

I'm a geek. I do geeky things.
This entry was posted in Lessons Learned, Methodology, Optimization, Team and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s