So What If You Have Big Data? Without Data-Driven Processes And Products, It’s Useless

Editor's note: John De Goes is co-founder and CEO of Precog, an analytics platform for developers. Follow him on Twitter @jdegoes.

These days, you don't have to look hard to find big data. Even a little startup can produce gigabytes a day, and a company the size of Instagram can easily generate 500 terabytes a day. If you're like many companies I talk to, you're sitting atop an ever-growing mountain of data, scratching your head and wondering, "Okay, I've got big data. Now what do I do?"

You don't win gold medals just for having data. The real winners are those like Amazon and Netflix that find ways to leverage their data better than the competition. Without a game plan to turn your data into revenue, you may as well scrap your badass Hadoop cluster and the petabytes of data it contains.

On the other hand, if you find ways to use that data better than your competition, you just might join the ranks of the big (data) boys like Amazon and Netflix.

So, how can you start to turn your data into cash? For most companies, there are two ways they can wield their data assets to create unfair competitive advantages: data-driven processes and data-driven products.

Data-Driven Processes

In the era of big data, business analysts entering equations into Excel and running ad hoc queries in a SQL database doesn't cut it. The new era demands a new breed of intrepid data explorer, one competent at using tools in both the small and big data worlds.

Dubbed a data scientist, this next-generation data geek knows enough about traditional BI tools, query languages, statistics, and machine learning to be considered armed and dangerous.

Good data scientists can help you do everything from figuring out what is and is not working in your products (as they are used at Zynga), to creating predictive models that let you peer into the future so you can make better decisions today (as they are used at @WalmartLabs).

Here are a few concrete examples of ways data scientists can help you:

• If you sell a SaaS application, a data scientist can help you identify the common characteristics of high-revenue users. For example, they make take particular pathways during conversion to a paid account, and they may share particular demographic attributes (gender, income, location, age range, etc.), and use the product in specific ways. All these insights can help you refine advertising, marketing, and product to increase revenue.

• A data scientist can identify to what extent one pricing tier or product is cannibalizing sales from other pricing tiers or products, so you can optimize your pricing strategy and product lines.

• A data scientist can build a predictive model based on historical data that lets you make fairly accurate predictions. For example, you could identify which customers are likely to be female and pregnant (something Target has done), or identify which leads in a sales pipeline are most likely to convert and at what levels.

• A data scientist can help you figure out the right questions to ask about your data. For example, a data scientist might suggest correlating your marketing data to your web log data to your transactional data, to identify the ROI behind marketing campaigns.

Data-Driven Products

The flip side to using data to drive your business processes is using data to enhance the functionality of the products you make (which doesn't apply to all products, for example, toothbrushes and pillows!). Some companies accomplish this by packaging data into a useful, insightful product they can then sell to other companies.

Twitter, while not a data product itself, licenses its data to providers like DataSift, who then go on to create a data product that companies gobble up for the insight it provides them. Some media companies package up their audience viewership data into products they turn around and sell to channel programmers and content creators. And so on.

The majority of companies building data-driven products, however, don't create and sell pure data products. Rather, they use data to make their existing products more efficient, more intelligent, or more insightful in some way that directly or indirectly generates additional revenue.

Here are some real-world examples of the ways data is being used to drive intelligent and insightful features inside existing products:

• An advertising platform that chooses which ad to show to which individual — based on what's known about the ad placement, the ad itself, and the user being shown the ad — in order to maximize the probability of a clickthrough or other revenue-generating user action.

• An e-commerce app that intelligently recommends products to maximize the probability that consumers will purchase both whatever they came in to buy and lots of stuff they didn't come in to buy.

• A publisher that intelligently personalizes every single page for every user, based on whatever is known about the user, to maximize the chance that users will stay on the publisher's site, and thus generate more advertising revenue.

• A video platform that captures all user interactions and provides content creators with detailed analytics that help them optimize important metrics (engagement, plays, conversions). This is an example of indirect monetization. Adding a feature powered by data (analytics) helps make the platform more attractive to users.

You Too Can Be Data-Driven

If my descriptions of data-driven processes and data-driven products has you salivating, but you're still wondering how you can go from mountains of senseless data to piles of cold hard cash, then I've put together some concrete recommendations that should help get you started.

Capture everything centrally. In this day and age of plummeting storage costs and ubiquitous (free!) big data stores, if you're not capturing every single bit of data, you're doing something wrong. I often tell companies that while you can always ignore data you have, you can't analyze something you don't have. Unstructured and semi-structured data stores let you store data in its raw format now and pay the cost of extracting structure only when and if you need to. So there's no excuse for not storing transactions, interactions, behavioral data, sensor data, user-generated content, log files, and whatever else you can get your hands on.

Get yourself a data scientist. If you're a startup, you need at least one data scientist on your team, or someone who can double as a data scientist. If you're a larger company, you need a whole team, and it may be easier to train from the inside rather than hire new data scientists. Data scientists can sometimes be trained from strong business analysts or those with an excellent background in BI and SQL. Data scientists need to be carefully equipped with proper tools and access to company-wide data so they can answer ad hoc questions, perform exploratory data mining, support BI teams, and help with data productization. A great data scientist will help you figure out the questions you need to ask in order to advance your agenda. He or she will also be looking for new ways to take advantage of all the data your company has access to.

Productize your data. Any company with proprietary data should strongly consider using that data to build new products or drive data-driven features inside existing products. Any company who has a desktop, mobile, web, server, or media-based application has proprietary data (which means, in this data and age, most companies have proprietary data!). Companies, especially in advertising and retail, have made millions or even billions of dollars of incremental revenue by using data to drive intelligent features in their applications.

If you're a B2B SaaS vendor, providing your customers with self-service reporting is one simple way you can translate data into a product feature that indirectly drives additional revenue. If you're an e-commerce platform, using all the data you have at your disposal for recommendations and personalizations can drive substantial incremental revenue. If you're a consumer app, using data to make your application smarter can lead to higher usability and better engagement. Having someone on staff to think about what kinds of features and products you can create from your data assets is the first step toward data productization, but ultimately, you will need engineering resources that can turn that data into features and products.

The Data-Driven You

Big data isn't about the data, per se, it's about finding ways to use that data to drive business processes and product functionality within your company. The meteoric rise of data science in the past few years is a testament to the fact that data is the currency of the 21st century. If you don't do anything with your data, you're at a severe competitive disadvantage.

But by taking a few simple steps, such as capturing all the data you can get your hands on, making sure you have at least one data scientist, and working towards data productization, you can ensure that you're effectively "spending" all that currency currently piling up in your data warehouse.