Problems and Opportunities with Big Data: a Project Management Perspective

By Dennis D. McDonald[1]

Being able to make wise investments in “big data” capabilities may require more collaborative approaches to project management and decision-making than are currently being practiced. Two recent articles illustrate this:

The first article states an often heard complaint that, despite much marketing hype, real-world examples of big data value are few:

“It does not help when so-called experts suggest that an information revolution is changing–and will continue to change–the nature of the workplace itself. Big data projects can have a transformative effect on business operations and processes. Yet evidence of a knowledge lead revolt in the offices of global blue chips is thin on the ground. Instead large organizations seem to be drowning under a sea of information.”

The second article focuses on the opportunity side of the equation. It addresses how technology management needs to evolve in the face of growing technical data handling capabilities:

“From the top of the stack, more users want access to more data and more combinations. And from the bottom of the stack, more data is available than ever before–some aggregated, much of it not. The only way for data professionals to deal with pressure of heterogeneity from both the top and bottom of the stack is to embrace a new approach to managing data that blends operations and collaboration to organize and deliver data from many sources to many users reliably with the provenance required to support reproducible data flows.”

My own interest — making organizational data more open and accessible to both internal and external user groups — dovetails with both perspectives.

Both views show how people are grappling with the challenge of using modern tools to make sense of the increasing volumes of data in digital form. If there is no clear understanding of what benefits will flow from big data investments, investment justification will be a challenge. Anyone who has ever sold a new tool or technique understands this. This is why practical demonstrations and understandable testimonials are so important.

It is clear that tools and technology are improving in ways that can make sophisticated data usage more accessible and immediate to those with the necessary knowledge. For example, the turnaround time for processing large amounts of data is dropping as new platforms and tools are introduced. Processing speed and the ability to combine analyses of different data types can be documented objectively and technically.

One question is, can we demonstrate the real world benefits of such techniques in meaningful ways without requiring management to possess data science degrees? Just talking about the benefits of speed and volume in technical terms is not enough. Showing how these benefits translate into useful insights and support for useful planning and decisionmaking is required.

Healthy skepticism

The first quotation above reflects a healthy skepticism brought on by much of the hype surrounding big data. Such skepticism is a good thing if it forces proponents to articulate the “whys” and “wherefores” instead of just the “hows” of big data. This need for proof is why supportive “case studies” are in such demand at meetings and conferences. Consultants and vendors understand this and is why so many stories about big data “successes” are industry sponsored.

The second article’s emphasis on infrastructure is typical of tech-oriented solutions that focus more on the technology side of the equation. Yet there is more here than just a veneer of newness. Combining “big data” with “DevOps” makes great sense. I especially like the emphasis on the need for communication and collaboration which are critical to figuring out how best to grapple with taking advantage of powerful new tools.              

Déjà vu

I have a sense of déjà vu when reading articles like these. “Tech hype” has always been with us. Mature tech managers and their bosses long ago mastered the art of healthy skepticism when hearing vendors expound on new technologies. A “show me” attitude is a healthy one to promote especially when price tags are high, a lot of changes are required, and business value may be difficult to pin down.

There’s always a lot of faith and trust required at this stage of technology adoption. That tech vendors are investing heavily in tools, advertising, marketing, and hype need not divert us from asking the tough questions. Inevitably discussions about how best to take advantage of new tools will lead to serious discussions about strategy, governance, quality, and costs.


Pushback from the organization can come in many forms ranging from good old-fashioned resistance to change (how many times have we heard, “You’ve got to change the culture!”) to major implosions of high-profile projects or vendors. What may be different this time around with big data may be the speed with which new tools are introduced and the seeming ease with which tools can be used to analyze and visualize vast amounts of data quickly.

Range of changes

New data tools and technologies may also stimulate the need for changes to current management processes which, more often than not, generates pushback from those being changed. Adopting big data tools and process changes may be associated with a range of organizational changes including:

  • The move to cloud services as a replacement for current infrastructure.
  • The need to learn new tools and techniques.
  • Resistance from business process owners who don’t want to change.
  • Overemphasis on tools and technologies while shortchanging business and strategy.
  • The need to align data services with core business needs, not just with “easy to do” and “low hanging fruit” initiatives.

While it’s important for IT staff to understand new data tools, employees and managers in all functional areas are impacted and must be able to articulate what they want from the data and from the new tools.

This is where the need for collaboration and communication emerge along with the need for basic project management techniques and support. Stakeholders need to be identified and brought on board. Resources need to allocated and managed. Progress needs to be tracked and reported in some fashion. New products and services need to be aligned with goals and objectives that are important to the organization. Inevitable changes need to be addressed as learning takes place. Most important, project goals and objectives need to be clear and understandable to all.

No best practices

I think that, regarding the first article cited above, skepticism is healthy but needs to be specific to the organization’s needs, not general. Searching for “best practices” and examples of whether or not other organizations are able to take advantage of big data applications may have little relevance to one’s own organization. Understanding one’s own needs and requirements must be the starting place, otherwise we run the risk of doing the “same old things” only with new tools.

One criticism of the second article is related to this. That is, while I do believe that we may need to re-think how we manage infrastructure and IT services to take advantage of new tools, services, and platforms, we also need to pay attention to how priorities are set and initiatives managed. Again, that’s basic project management and points out the need for communication and collaboration in how priorities are set and how solutions are implemented.

Whether we call this “DevOps” or “DataOps” is not the issue. The issue is whether or not we can effectively manage projects and programs that take advantage of big data while involving all necessary stakeholders throughout every point in the data value chain, not just IT staff and data analysts.

Data literacy and strategic alignment

If it is inevitable in today’s fast paced world that many fingers are going to be stuck in the “data management pie,” we need to make sure that the heads operating those fingers have a basic understanding of how data are generated, managed, and used; let’s call this “data literacy.” Without basic data literacy participants risk talking past each other with the end result being inefficiency, blind alleys, and disappointment.

The connection between better data services needs to be aligned clearly with the organization’s goals and objectives. Again, this is a basic requirement for effective project management. Management, IT, and data analytics experts need to communicate and collaborate right from the start so that everyone is on the same page.

Related reading:


The author is grateful to Julia Glidden of 21c Consultancy Ltd for her helpful comments on an earlier draft of this article.

[1] Copyright © 2015 by Dennis D. McDonald, Ph.D. Dennis is a management consultant based in Alexandria, Virginia. His experience includes consulting company ownership and management, database publishing and data transformation, managing the integration of large systems, corporate technology strategy, social media adoption, statistical research, open data, and IT cost analysis. Clients have included the U.S. Department of Veterans Affairs, the U.S. Environmental Protection Agency, the National Academy of Engineering, and the National Library of Medicine. He has worked as a project manager, analyst, and researcher for both public and private sector clients throughout the U.S. and in Europe, Egypt, and China. His web site is located at and his email address is On Twitter he is @ddmcd.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s