Mechanical Turk
The history of Wolfgang von Kempelen’s Mechanical Turk . (Amazon Mechanical Turk)


The technology world we live in today simply extends beyond comprehension. It is a fantastic world where you routinely feel like Charlie in the chocolate factory - amazed at what can be done. I had started feeling slightly jaded to the fact that I can earn a decent living selling a non-physical product created by a group of virtual team members to clients I rarely see.

The sense of profound awe was back when I decided to bite the bullet and sign up on Facebook - the second most popular social networking site after MySpace. Within a couple of days of signing up and mildly poking around, I had connected with a few dozen long-lost (and, in cases, completely forgotten) friends and acquaintences. The joy and satisfaction of connecting again with childhood friends and talking about Ethiopian food that we grew up on was probably the highlight of the past month. A week in, I am utilizing widgets written by users that hook into the Facebook application constantly, in the process positioning myself as one mean “vampire-biter” and a generous virtual gift-giver. To boot, the IQ Test widget tells me that compared to my friends, I am on the challenged side. And the movie widget tells me that I have absolutely nothing in common with Phil my business partner.

In this week of discovery, I also stumbled upon Amazon’s Mechanical Turk. In a world where the human workforce is constructed, deconstructed, and reconstructed like Lego pieces, to the constant buzz of words like outsourcing, offshore sourcing, onshore sourcing, and rural sourcing, we can now add another term: “micro-sourcing” (or if you like it better “crowdsourcing”). The idea behind micro-sourcing is simple. In the constant pursuit of cutting labor costs, a lot has been automated. However, there still remain a large swath of simple tasks - like pattern recognition or interpreting the emotion behind a statement - that people do much better than computers. So how to bring cost-efficiency to these processes? Here’s where Mechanical Turk comes in. Mechanical Turk is quite simple and quite brilliant - use a huge pool of internet-connected people with a bit of free time to perform very tiny, simple, routine tasks.

Say a company wants to go through 100,000 images and sort them based on which ones have a human face in it. Hiring people to do this is costly and time-consuming. Micro-sourcing solves this dilemma by farming out the project in tiny chunks to a multitude of people connected to the internet. So now, you have 50,000 people that take five seconds to tag two pictures as “has human face” or “no face here”. For their time, they maybe make a penny. They do it simply because they have nothing better to do at that moment and the task is mildly interesting. Cost to the firm for this picture categorization task? $500. Time to completion? Theoretically, if all people do their two pictures that day - one day!

The applications are too many to count. Some examples include transcription of texts, optical character recognition, data manipulation, content monitoring, gauging consumer sentiment, market and product research, and so forth.

Real life examples? Write a small Facebook widget that allows people to upload a picture of themselves and that picture is sent via Mechanical Turk to be rated on an attractiveness scale of 1-10. Instant feedback on the picture is provided that shows a person their “attractiveness” as ranked by a multitude of people. The user may even be able to drill down and see how different geographic regions ranked their attractiveness. Their score can then be displayed and shared with friends who virally installing the widget themselves. Monetizing such an endeavor? Maybe use the most often used scheme in social network applications - advertising. A less trifle example would be using the network reach of Facebook and/or Mechanical Turk to code free-text survey responses that are a common part of research surveys at universities or think tanks.

The vexing thing about the obvious is that it tends to be invisible and painfully visible when you cross it. Like skipping coffee in the morning or not watering your plant every month. Or forgeting the software development maxim: a process which cannot be repeated consistently is better left unautomated.

On pain of this post being considered a total bore, it must be stated explicitly: until there is consensus in your organization about how a process is to be handled and the chain of activities is repeatable consistently, you are toying with project failure should you try to automate it. Instead of commissioning software developers to build you a tool to automate your process along with its healthy peppering of exceptions, you’re better off bringing in a process engineer.

There is also a different view of the same problem. We understand implictly that failed software projects tend to have a lot to do with not understanding the workflow in question or not gathering the breadth and depth of requirements correctly. But another obvious thing we tend to overlook is: if it’s indeed that hard to gather requirements for a particular workflow, it may be that the workflow isn’t all that defined in the first place.

As a business executive that may be in charge of approving a software project, it is advisable to get your team together and see if they can, unerringly and consistently, walk you through the process you are seeking to automate. As a CEO or CIO, it is too tempting to assume that your birds-eye view of a process that shows no blemishes is indeed so.

You need chaos in your soul to give birth to a dancing star.
Friedrich Nietzsche

Silly star

I really feel like that bit about a dancing star is rather silly. At least, when I was faced with the chaos of leaving home without my all-important laptop and cell phone last Tuesday and cut off from the option of driving back by a solid ribbon of traffic across the nine miles of the I-90 east thruway, I didn’t feel much like a dancing star. It was more like being rubbed down with a wet towel of panic. As you subconsciously try to control the dismay and irritation at sweat ruining your white shirt collar, you try to frantically recall your appointments that were in Outlook, wonder how you’ll communicate throughout the day, how you’ll find the phone numbers for people to call, and, more importantly, how you’ll pass the time between appointments without any work to do.

By the time I got to my office, I had calmed down sufficiently to take stock of the situation. Maybe not everything was sunk. Let’s see, first things first:

1. I utilize MDCall - a product of my company (shameless plug, but true) that specializes in emergency physician contact management. Through MDCall, I can manage where my calls are routed. By logging online, I switched all emergency calls that come into our toll-free number for me to my office phone instead of my cell. So that was done. Services similar to MDCall include, a terrific resource to give to business associates so that by calling one number, they can reach you where you want to be reached - cell, home, office, or the hotel where you are staying at.

2. So that was done. Next, e-mail. With a spare laptop laying around in the office, I quickly logged into my hosted web-based email account through Gmail for Business. Now, before you think I’m a cheap business owner with no regard for professional image, let me state that Gmail for Business is not like personal Gmail or other free email services like Hotmail. It’s a full-powered email service that beats having your email through most hosted services anytime. Here’s why:

  • Gmail can be associated with your domain name. So, your emails do not go out as but rather This is because you are pointing your domain MX records to gmail just like you would point them to your web host.
  • Gmail plays really well with Outlook. You can use Outlook to manage your emails. What’s more, any email that is sent or received now is not only in Outlook but also archived on Gmail.
  • Any email you send by logging into Gmail via the web gets copied over to your Outlook next time you Send/Receive. This means that your emails in Outlook and Gmail are always in sync.
  • Gmail search is fantastic! Searching for “jeff cell san diego” in Outlook took a full 6 minutes. In Gmail - 2 seconds. So now, whenever I want to search for something in my email, I log into Gmail via the web.
  • Gmail servers guarantee 99.9% uptime. So do most web hosts. But I tend to trust Google more on this one.
  • Gmail provides you with 10GB of space for EACH corporate email account.
  • You can access Gmail via a mobile device like you cell phone. I love it when I’m twiddling my thumbs in a restaurant while waiting for someone.

3. So, I have calls forwarded and I have email up. Next is making phone calls. That, of course, is easy - I do have an office phone, believe it or not. But I also use Skype to make calls to any phone in the world using my PC. Since my laptop was not around, I simply installed the Skype client on the temporary laptop, logged into Skype, and was ready to make calls. This would be especially handy when I would stop over later at a Starbucks between meetings.

4. Finally, what of my schedule? Nicely enough, my Gmail for Business account also comes with Google Calendar. Any events I create in Outlook, I automatically send over to Google Calendar. No big worries. I was going to make every meeting and appear on top of my game! 

5. With phone, email, calendaring, and call forwarding, my communication platform was almost back to normal. The only item missing - instant messaging. My team, which includes my partner in Colorado and other consultants, lives and dies by communicating via instant messaging - for those small things that picking up the phone just isn’t worth. I communicate via AOL IM and Yahoo Messenger. What a chore to have to install both software pieces on a temp laptop. Rather, in my browser, I fired up All I needed to do was type in my respective AOL and Yahoo login credentials and click on Sign In. I was ready to chat.

Of no significant relevance this particular chaotic day but still bearing mention: My office uses Send2Fax to receive and send faxes. Faxes for me come directly into my email as a PDF. And I can send faxes directly from my computer without having to print them out and walking over to a silly machine. And I save important documents to our internally-written web-based document management system. Alternatives include Zoho Writer or Google Docs.

So there I was. Completely functional. Happy as could be. But still not too sure about the chaos and dancing star bit… 

If there’s anyone out there that has productivity tools that bear mentioning, comment to this post!

It is very common for enterprises to look at off-the-shelf software that purports to solve the particular business challenge they are faced with. And why shouldn’t they? After all, if you can touch it and see it, you presumably know what you’re buying. Furthermore, packaged software tends to have a lower cost of acquisition (NOT necessarily lower cost of ownership over time, mind you) due to the development cost being spread over a higher number of installations. The risk that the enterprise takes on, thus, should be minimal. That is true enough…in some cases. Cases where:

  • the need is generic (eg. word-processing where MS Word will suffice or email where Outlook is enough), or
  • the requisite solution meets a need that’s standardized within the industry (eg. medical billing software for physical therapists), or
  • the need is non-core and thus the possibility of having to conform your business processes to the software is a worthwhile trade-off

If, however, the solution:

  • has no off-the-shelf alternative, or
  • needs to accomodate unique internal processes (which, incidentally, tends to be true in organizations in highly competitive industries where the key differentiator is usually at the process and management level), or
  • needs to deliver very high levels of efficiency, or
  • will be expected to integrate with a variety of other information systems, or
  • will be expected to grow to meet as-yet unanticipated needs

then, a custom software development project just might make sense.

Custom software development, however, is not for the faint of heart. Anyone that has tried it probably has a war story to tell - stories of botched development suffering from poor understanding, lack of communication, scope creep that leads to over-budget and behind-schedule implementations, and even software that, when deployed, turns out to be totally useless.

Avoiding these pitfalls is hard work, but in no way impossible. Here are two key risks that should be actively managed to give you a fighting chance.

Risk # 1: Trying to automate bad processes

Pearl of Wisdom: Automating bad processes just leads to tears faster.

Risk Avoidance Strategy: Allocate time, money, and organizational resources to get rock-solid requirements. This also includes evaluating your current processes. Is your existing workflow unwaveringly exact every time? Are your current process streamlined already or can they be made more efficient? Have you cut out as many exceptions out of your business rules as possible? Tell-tale signs of bad processes include:

  • your team feels that the process is “quite complex”
  • there is no consensus within your team on what the process is exactly
  • it’s hard to keep track of all the exceptions

Bad processes tend to cause software architects to draw up incomplete and flawed user requirements. These requirements, in turn, cause developers to create elegant solutions for a broken workflow.

Risk # 2: Scope Creep

Pearl of Wisdom: Don’t gold-plate, focus on demonstrable return on investment, instill business-focus rather than technology-focus as a core value.

Risk Avoidance Strategy: Scope creep is a phenomenon defined in project management as uncontrolled changes in a project’s scope. Another way to put it: what exactly is being built keeps changing. Scope creep can occur because:

  • the requirements are ill-defined, processes are ill-understood, and/or stakeholders are ill-selected
  • gold-plating occurs because someone wants a particular “cool” feature added in that has no correlation to greater value
  • developers or IT personnel are rearing to use a new technology they’ve been reading about without any consideration to whether it’s the right tool in the first place
  • management lacks the requisite discipline to define scope and then hold it steady

Gold-plating and focus on favorite technology tends to lead to longer development times, higher cost, and, perversely, despite being provided both, lower usefulness of the software. There are too many features that make a user’s job harder to perform thereby increasing adoption rates and decreasing productivity. IT’s focus on technology rather than focus on business may mean higher development costs on the backend but possibly also higher utilization costs on the frontend should the selected technology lead to performance gaps, limitations to scalability and integration, and higher costs of managing the operating environment.

Those charged with delivering a custom software solution must, thus, instill a “business need is king” mentality where technology knowledge is important, but not as important and sacred as the understanding of and objective assessment of the core business operations. Demand return on investment analysis as well that is based on a strong requirements analysis. Anytime a change to the requirements document is proposed, go back to the ROI document and evaluate what the added time and financial cost means to the company.


It’s like ten thousand spoons when all you need is a knife.

Alanis Morissette


Risk # 3: Mismanagement of Scope

Pearl of Wisdom: Asking for everything to preempt scope creep is a great way to go, assuming you want to lose your shirt in the process.

Perhaps even more fundamental than the risk of scope creep is knowing what the ideal scope should be in the first place. The Pareto Principle, modified for purposes of software development discussion, suggests that 20% of processes account for 80% of contribution to productivity. The holy grail, then, becomes defining this 20% and building that into your initial scope document. Additional features may simply be costly overhead that only lose, confuse, and mislead users and cause them to make mistakes. What’s more, these additional features all need to be accounted for when systems are modified, upgraded, or extended. This leads to higher development and maintenance costs in the future.

Any comments to counter or refine my thoughts would be most welcome!

« Previous Page