What is big data?

The hot IT buzzword of 2012, big data has become viable as cost-effective approaches have emerged to tame the volume, velocity and variability of massive data. Within this data lie valuable patterns and information, previously hidden because of the amount of work required to extract them. To leading corporations, such as Walmart or Google, this power has been in reach for some time, but at fantastic cost. Today’s commodity hardware, cloud architectures and open source software bring big data processing into the reach of the less well-resourced. Big data processing is eminently feasible for even the small garage startups, who can cheaply rent server time in the cloud.

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.

The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined reports.

The past decade’s successful web startups are prime examples of big data used as an enabler of new products and services. For example, by combining a large number of signals from a user’s actions and those of their friends, Facebook has been able to craft a highly personalized user experience and create a new kind of advertising business. It’s no coincidence that the lion’s share of ideas and tools underpinning big data have emerged from Google, Yahoo, Amazon and Facebook.

The emergence of big data into the enterprise brings with it a necessary counterpart: agility. Successfully exploiting the value in big data requires experimentation and exploration. Whether creating new products or looking for ways to gain competitive advantage, the job calls for curiosity and an entrepreneurial outlook.

What does big data look like?

As a catch-all term, “big data” can be pretty nebulous, in the same way that the term “cloud” covers diverse technologies. Input data to big data systems could be chatter from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data, the list goes on. Are these all really the same thing?

To clarify matters, the three Vs of volume, velocity and variety are commonly used to characterize different aspects of big data. They’re a helpful lens through which to view and understand the nature of the data and the software platforms available to exploit them. Most probably you will contend with each of the Vs to one degree or another.

Volume

The benefit gained from the ability to process large amounts of information is the main attraction of big data analytics. Having more data beats out having better models: simple bits of math can be unreasonably effective given large amounts of data. If you could run that forecast taking into account 300 factors rather than 6, could you predict demand better?

This volume presents the most immediate challenge to conventional IT structures. It calls for scalable storage, and a distributed approach to querying. Many companies already have large amounts of archived data, perhaps in the form of logs, but not the capacity to process it.

Assuming that the volumes of data are larger than those conventional relational database infrastructures can cope with, processing options break down broadly into a choice between massively parallel processing architectures — data warehouses or databases such as Greenplum — and Apache Hadoop-based solutions. This choice is often informed by the degree to which the one of the other “Vs” — variety — comes into play. Typically, data warehousing approaches involve predetermined schemas, suiting a regular and slowly evolving dataset. Apache Hadoop, on the other hand, places no conditions on the structure of the data it can process.

At its core, Hadoop is a platform for distributing computing problems across a number of servers. First developed and released as open source by Yahoo, it implements the MapReduce approach pioneered by Google in compiling its search indexes. Hadoop’s MapReduce involves distributing a dataset among multiple servers and operating on the data: the “map” stage. The partial results are then recombined: the “reduce” stage.

To store data, Hadoop utilizes its own distributed filesystem, HDFS, which makes data available to multiple computing nodes. A typical Hadoop usage pattern involves three stages:

• loading data into HDFS,
• MapReduce operations, and
• retrieving results from HDFS.

This process is by nature a batch operation, suited for analytical or non-interactive computing tasks. Because of this, Hadoop is not itself a database or data warehouse solution, but can act as an analytical adjunct to one.

One of the most well-known Hadoop users is Facebook, whose model follows this pattern. A MySQL database stores the core data. This is then reflected into Hadoop, where computations occur, such as creating recommendations for you based on your friends’ interests. Facebook then transfers the results back into MySQL, for use in pages served to users.

Velocity

The importance of data’s velocity — the increasing rate at which data flows into an organization — has followed a similar pattern to that of volume. Problems previously restricted to segments of industry are now presenting themselves in a much broader setting. Specialized companies such as financial traders have long turned systems that cope with fast moving data to their advantage. Now it’s our turn.

Why is that so? The Internet and mobile era means that the way we deliver and consume products and services is increasingly instrumented, generating a data flow back to the provider. Online retailers are able to compile large histories of customers’ every click and interaction: not just the final sales. Those who are able to quickly utilize that information, by recommending additional purchases, for instance, gain competitive advantage. The smartphone era increases again the rate of data inflow, as consumers carry with them a streaming source of geolocated imagery and audio data.

It’s not just the velocity of the incoming data that’s the issue: it’s possible to stream fast-moving data into bulk storage for later batch processing, for example. The importance lies in the speed of the feedback loop, taking data from input through to decision. A commercial from IBM makes the point that you wouldn’t cross the road if all you had was a five-minute old snapshot of traffic location. There are times when you simply won’t be able to wait for a report to run or a Hadoop job to complete.

Industry terminology for such fast-moving data tends to be either “streaming data,” or “complex event processing.” This latter term was more established in product categories before streaming processing data gained more widespread relevance, and seems likely to diminish in favor of streaming.

There are two main reasons to consider streaming processing. The first is when the input data are too fast to store in their entirety: in order to keep storage requirements practical some level of analysis must occur as the data streams in. At the extreme end of the scale, the Large Hadron Collider at CERN generates so much data that scientists must discard the overwhelming majority of it — hoping hard they’ve not thrown away anything useful. The second reason to consider streaming is where the application mandates immediate response to the data. Thanks to the rise of mobile applications and online gaming this is an increasingly common situation.

Product categories for handling streaming data divide into established proprietary products such as IBM’s InfoSphere Streams, and the less-polished and still emergent open source frameworks originating in the web industry: Twitter’s Storm, and Yahoo S4.

As mentioned above, it’s not just about input data. The velocity of a system’s outputs can matter too. The tighter the feedback loop, the greater the competitive advantage. The results might go directly into a product, such as Facebook’s recommendations, or into dashboards used to drive decision-making.

It’s this need for speed, particularly on the web, that has driven the development of key-value stores and columnar databases, optimized for the fast retrieval of precomputed information. These databases form part of an umbrella category known as NoSQL, used when relational models aren’t the right fit.

Microsoft SQL Server is a comprehensive information platform offering enterprise-ready technologies and tools that help businesses derive maximum value from information at the lowest TCO. SQL Server 2012 launches next year, offering a cloud-ready information platform delivering mission-critical confidence, breakthrough insight, and cloud on your terms; find out more at www.microsoft.com/sql.

Variety

Rarely does data present itself in a form perfectly ordered and ready for processing. A common theme in big data systems is that the source data is diverse, and doesn’t fall into neat relational structures. It could be text from social networks, image data, a raw feed directly from a sensor source. None of these things come ready for integration into an application.

Even on the web, where computer-to-computer communication ought to bring some guarantees, the reality of data is messy. Different browsers send different data, users withhold information, they may be using differing software versions or vendors to communicate with you. And you can bet that if part of the process involves a human, there will be error and inconsistency.

A common use of big data processing is to take unstructured data and extract ordered meaning, for consumption either by humans or as a structured input to an application. One such example is entity resolution, the process of determining exactly what a name refers to. Is this city London, England, or London, Texas? By the time your business logic gets to it, you don’t want to be guessing.

The process of moving from source data to processed application data involves the loss of information. When you tidy up, you end up throwing stuff away. This underlines a principle of big data: when you can, keep everything. There may well be useful signals in the bits you throw away. If you lose the source data, there’s no going back.

Despite the popularity and well understood nature of relational databases, it is not the case that they should always be the destination for data, even when tidied up. Certain data types suit certain classes of database better. For instance, documents encoded as XML are most versatile when stored in a dedicated XML store such as MarkLogic. Social network relations are graphs by nature, and graph databases such asNeo4J make operations on them simpler and more efficient.

Even where there’s not a radical data type mismatch, a disadvantage of the relational database is the static nature of its schemas. In an agile, exploratory environment, the results of computations will evolve with the detection and extraction of more signals. Semi-structured NoSQL databases meet this need for flexibility: they provide enough structure to organize data, but do not require the exact schema of the data before storing it.

In practice

We have explored the nature of big data, and surveyed the landscape of big data from a high level. As usual, when it comes to deployment there are dimensions to consider over and above tool selection.

Cloud or in-house?

The majority of big data solutions are now provided in three forms: software-only, as an appliance or cloud-based. Decisions between which route to take will depend, among other things, on issues of data locality, privacy and regulation, human resources and project requirements. Many organizations opt for a hybrid solution: using on-demand cloud resources to supplement in-house deployments.

Big data is big

It is a fundamental fact that data that is too big to process conventionally is also too big to transport anywhere. IT is undergoing an inversion of priorities: it’s the program that needs to move, not the data. If you want to analyze data from the U.S. Census, it’s a lot easier to run your code on Amazon’s web services platform, which hosts such data locally, and won’t cost you time or money to transfer it.

Even if the data isn’t too big to move, locality can still be an issue, especially with rapidly updating data. Financial trading systems crowd into data centers to get the fastest connection to source data, because that millisecond difference in processing time equates to competitive advantage.

Big data is messy

It’s not all about infrastructure. Big data practitioners consistently report that 80% of the effort involved in dealing with data is cleaning it up in the first place, as Pete Warden observes in his Big Data Glossary: “I probably spend more time turning messy source data into something usable than I do on the rest of the data analysis process combined.”

Because of the high cost of data acquisition and cleaning, it’s worth considering what you actually need to source yourself. Data marketplaces are a means of obtaining common data, and you are often able to contribute improvements back. Quality can of course be variable, but will increasingly be a benchmark on which data marketplaces compete.

Culture

The phenomenon of big data is closely tied to the emergence of data science, a discipline that combines math, programming and scientific instinct. Benefiting from big data means investing in teams with this skillset, and surrounding them with an organizational willingness to understand and use data for advantage.

In his report, “Building Data Science Teams,” D.J. Patil characterizes data scientists as having the following qualities:

• Technical expertise: the best data scientists typically have deep expertise in some scientific discipline.
• Curiosity: a desire to go beneath the surface and discover and distill a problem down into a very clear set of hypotheses that can be tested.
• Storytelling: the ability to use data to tell a story and to be able to communicate it effectively.
• Cleverness: the ability to look at a problem in different, creative ways.

The far-reaching nature of big data analytics projects can have uncomfortable aspects: data must be broken out of silos in order to be mined, and the organization must learn how to communicate and interpet the results of analysis.

Those skills of storytelling and cleverness are the gateway factors that ultimately dictate whether the benefits of analytical labors are absorbed by an organization. The art and practice of visualizing data is becoming ever more important in bridging the human-computer gap to mediate analytical insight in a meaningful way.

Know where you want to go

Finally, remember that big data is no panacea. You can find patterns and clues in your data, but then what? Christer Johnson, IBM’s leader for advanced analytics in North America, gives this advice to businesses starting out with big data: first, decide what problem you want to solve.

If you pick a real business problem, such as how you can change your advertising strategy to increase spend per customer, it will guide your implementation. While big data work benefits from an enterprising spirit, it also benefits strongly from a concrete goal.

Strata - Making Data Work (http://s.tt/1nO4l)

Facebook offers a good example of the uncertainty businesses face. Today, 488 million users reach its pages from mobile devices each month. Yet these visits yield "no meaningful revenue," according the company's IPO prospectus. The ads and games Facebook can sell to PC users don't yet translate to phones. Nearly any company on the Web, small or large, will find itself following users into a domain of similarly doubtful revenue.

Mobile barbarians are already at the gates of vulnerable industries. During last year's Christmas shopping season, Amazon offered an app that let shoppers scan bar codes in stores and then buy for less online; retailers were outraged. (No one, not even Amazon, stood to make much money amidst the hostilities.) On the other hand, tablets and e-readers offer a lifeline to publishers trying to stanch the bleeding of their print businesses.

Technology businesses are also reëvaluating their strategies as the PC-era market order topples. Onetime leaders like Microsoft and Intel are far behind on mobile platforms and striving to catch up. Apple and Google vie for domination. Samsung, a manufacturer that invested heavily in smart phones, just ended Nokia's 14-year reign as the top mobile-device seller.

These technology races also pose philosophical questions pitting open-source against proprietary software and proprietary mobile platforms against the open Web. Apple tightly controls the operating system tied to its profitable gadgets and app store. Google aims to increase revenue from search advertising with open-source Android code available to any device maker.

Today, many companies offer downloadable native apps that are walled off from the searchable Web. But mobile-optimized HTML5 is another emerging option. It's not as functional as mobile software, but developers can avoid the 30 percent revenue cut they must pay to Apple and Google to sell apps. Playboy, banned from Apple's store for nudity, became an early adopter of HTML5 for the iPad; so did the Financial Times, which did not want to pay the fees.

For consumers, mobile computers are still used primarily to consume media or do basic tasks like send e-mail, take photos, or look up map directions. But the mobile-computing experience will increasingly involve more day-to-day interactions with the world around us: what we create and what we buy, how we work, and how we connect with people, places, and things.

This raises questions about how we will interact with our devices. What's the optimal screen size? How good will voice and gesture recognition get? And how will we continue to determine the optimal trade-offs between data speed, data costs, and battery life?

One day, we will not need to ask these questions. Computing will occur on a continuum of devices, including PCs, ultra-books, large-screen smart phones, and phones that act like fully functional computers when docked. Hardware distinctions will become fuzzier as we store more documents and photos on remote cloud servers accessible from anywhere, instead of on local memory drives.

The mobile-computer market whose emergence dominates technology headlines today will also be a "boring" reality for most businesses, says Craig Mathias, a longtime wireless consultant who is president of Farpoint Group. "In five years, the differences will all be more subtle than they are today," he says. "We won't need wireless analysts any more than we need computer analysts today. I'll find something else to do."

Questions for Mobile Computing

Mobile devices outsold PCs last year for the first time, and top smart-phone apps need little more than a year to win the kind of audience it used to take technologies decades to reach. What are the limits of mobile computing?

When Facebook agreed last month to spend $1 billion to buy Instagram, a 12-person company with no revenues, technology watchers had to recalibrate their speedometers.

Facebook, the social network created for personal computers in February 2004, is now planning an IPO that could value it at $100 billion. Instagram, which began giving away its photo-sharing app for the iPhone only in October 2010, represents something new: a shift away from the Web and the PC to a kind of consumer experience built expressly for mobile devices, particularly smart phones.

Mobile computing is advancing much faster than other technologies did in their early years. To reach an audience of 50 million people, radio took 38 years, television 13 years, the Internet four years, Facebook three and a half years. Instagram took 1.3 years. That helps explain why Facebook founder and CEO Mark Zuckerberg reportedly negotiated the deal privately over the course of a weekend. The message was that these are do-or-die times, and only visionary founders with total control—and millions to spend—will be swift enough to keep up.

How fast is the change? Smart phones and tablets outsold personal computers for the first time last year. Forty-two percent of U.S. residents now own a smart phone; in the U.K., it's more than half, comScore estimates. Last week, while reporting record iPad sales, Apple CEO Tim Cook continued to predict that tablets by themselves would soon overtake the PC market. Even Afghanistan is licensing its 3G spectrum; there, many people's first computer will be a mobile one. Forrester Research tells us that globally, more users will access the Internet from mobile devices than from PCs within four years. The market research firm recommends that companies begin hiring for a new position: chief mobility officer.

Navigating this change is a challenge for all kinds of companies, whether or not their main business is tied to mobile technology. For many, pocket computers will offer huge, obvious benefits to customers, clients, or employees. But there are also big uncertainties. Will union employees sue for overtime if they use a smart phone to work during off hours? Will the stream of consumer data from mobile devices be as useful as it seems? For all the clarity the Instagram deal seemed to offer, things are not so clear. Some observers even predict that native apps—those that exist only on your phone, like Instagram—could be a short-lived phenomenon.

This month's issue of Business Impact looks at major questions now facing mobile computing and the factors that could limit or shape its direction. These range from device issues like screen sizes, battery life, and rapidly growing bandwidth consumption to the trade-offs between native apps and the open Web.

Wear this: Health-care entrepreneur Sonny Vu is developing wearable sensors.

The last time your doctor asked how much you exercise, did you tell the truth? Do you even really know the truth—not just how many visits to the gym you've made this month, but how many hours you sit or how many calories you burn in a day?

What if your doctor had already received the information from a tiny device built into your cell phone, wallet, or undershirt? Sonny Vu believes a device like this could fundamentally change health care. "You can't just lie to your doctor—it's all there, recorded," he says. "You cut right to the chase rather than having to tease out all that information."

Vu is an entrepreneur who thinks a lot about how a well-designed mobile device can affect health. As a cofounder of the medical-device company AgaMatrix, he created the first FDA-approved glucose sensor that plugs into an iPhone; it hit Apple stores this month under the brand name iBGStar.

Now Vu is taking his ideas a step further, betting that the next phase for mobile computing is on our bodies. He's heading a new company called Misfit Wearables, which is developing health monitoring devices that he says will fit unobtrusively into the clothing and objects we use every day.

Mobile health devices and software could change medicine profoundly, allowing people to continuously monitor vital signs and better track and modify behavior. That's important because chronic diseases like obesity and diabetes are on the rise. "We're seeing an infusion of mobile technologies into people's lives," says Susannah Fox, who studies technology and health care for the Pew Internet & American Life Project. "And we're seeing a very rainy forecast in terms of people's health."

In health care, however, good ideas often succumb to the realities of human nature. "Health isn't really top of mind for most of us," says Fox. Yet many health-related apps and devices essentially ask people to make health a priority. Pew's research has found that interest in health apps hasn't been increasing among users.

Vu's idea is to remove from the equation what he calls "intentionality"—the deliberate daily choice to use a health technology. Donning a pedometer or entering information into a calorie counter every day is asking too much of most people. "The best products are the ones that you really rely on but you don't have to remember to use," he says.

Vu says that realization came to him after many years of trying to understand why people with diabetes might forget to use their glucose meter, even though their health depends on doing so. (The meters use a drop of blood from a pinprick to measure blood sugar.) "If you have diabetes, what's your main problem? It's that you don't want to have diabetes anymore," says Vu. Carrying around a bulkier glucose meter is annoying and a constant reminder that one is ill. By creating meters that were closely integrated with a phone, something many people never leave the house without, "we enabled people to be closer to where they wanted to be, which was a little less diabetic," he says.

Vu, 39, splits his time between several locations, including Cambridge and his native Vietnam, where he's cultivated a software development team. He calls himself a "product person" who is happiest designing products and obsessing over their details. He founded Misfit Wearables last fall with John Sculley, former CEO of Apple, naming the company after Apple's iconic "Think different" ad ("Here's to the crazy ones. The misfits. The rebels ...").

The company raised $7.6 million this year from prominent investors, including Peter Thiel's Founders Fund and Vinod Khosla, following seed investment from the Cambridge incubator IncTank Ventures. Vu says technology investors are seeking to understand when, and how, computers will become wearable.

As a developer of medical devices, Vu is accustomed to proving his products' worth to the FDA. Now he's bringing that experience into the much less regulated world of consumer health and gadgets. Devices that monitor weight, activity level, heart rate, or other vital signs could, in principle, lower health-care costs by aiding efforts to prevent chronic illnesses like diabetes and heart disease. They could make it possible to provide medical services such as remote monitoring of patients or automatic detection of falls. "Wearable sensor data is going to be the most complete you can get," Vu says. It could make a yearly blood pressure measurement at the doctor's office seem archaic.

But developing these devices is challenging. First there's what Vu calls the "skin and silicon" problem. It's technically difficult to create an interface that accurately collects physiological signals and transmits them to a small mobile device. It's equally difficult to figure out what to do with the data. The people who obsessively analyze their own heart rate are a tiny minority, and even doctors don't have time to wade through raw data about their patients. The key, he says, is to provide software that can hunt for patterns and provide usable insights—that your heart rate veers dangerously high at work, or that your activity level drops on certain days of the week. But even the best device can't make someone follow its advice.

Vu is keeping quiet about the details of the product Misfit is planning to launch, which is still in development. It will function like current fitness monitors—he mentions the Fitbit pedometer and BodyMedia's activity-tracking armband—but will add a novel measurement that no other wearable device supplies. Vu is aiming for a consumer product, but eventually he'd like to conduct a clinical study of its effectiveness and seek FDA approval for a medical application.

The primary goal, however, is invisibility. "You have wearable products right now—they're just not that wearable," he says. "And you have to remember to wear them." He thinks a health monitoring device should would be unobtrusive enough to be incorporated into something you already wear or carry every day: socks, bra, undershirt, cell phone, wallet, keys.

That goal has brought Vu into the world of high-tech fashion. At a recent conference on smart fabrics, he mingled with designers and textile engineers making clothes that light up with fiber optics or heat and cool themselves. Vu believes the textile world could ultimately contribute more creative innovation to wearable computing than device companies do. "Those folks are thinking about clothing and about stuff you're already wearing," he says. "Not 'How can we strap this thing to your body?'"

Why Publishers Don't Like Apps

The future of media on mobile devices isn't with applications but with the Web.

Technology Review

By the time Apple released the iPad in April of 2010, only four months after Steve Jobs first announced his "magical and revolutionary" new machines, traditional publishers were gripped by a collective delusion. They had convinced themselves that tablet computers and smart phones would allow them to unwind their unhappy histories with the Internet.

For publishers whose businesses had evolved during the long day of print newspapers and magazines, the expansion of the Internet was terribly disorienting. The Web taught readers that they might read stories whenever they liked without charge, and it offered companies more efficient ways to advertise; both parties spent less.

For traditional publishers, the scheme was alluring. Because they were once again delivering a discrete product, analogous to a newspaper or magazine, they could charge readers for single-copy sales and subscriptions, reëducating audiences that journalism was something valuable for which they must pay. Software vendors like Adobe promised that editorial created with their print-oriented copy-management systems could be "seamlessly" transferred to the apps. And as for software development ... well, how hard was that? Most publishers had Web-development departments: let the nerds build the apps. Smart phones and tablets seemed to promise a return to simpler days. It was true that digital replicas of print newspapers and magazines (most often read inside Web browsers) had never been very popular, but publishers reasoned that reading replicas on desktop computers and laptops was an unpleasant experience. The forms of the new smart devices were a little like those of magazines or newspapers. Couldn't publishers delight readers by delivering something similar to existing digital replicas but suitably enhanced with interactive features? They told themselves that the new digital replicas would be better than their Web-bound kin because they would run in "native" applications on mobile operating systems like Apple's iOS, and thus would possess the dazzling functions of true software.

Publishers also expected to revive the old print advertising economy. The Audit Bureau of Circulations (ABC), the industry organization that audits circulation and audience information for magazines and newspapers in North America, said the replicas inside apps would count toward "rate base," the measure of publications' total circulation, which includes subscription and newsstand sales. Rate base had been the metric for setting advertising rates in publishing before the emergence of online banner and keyword advertising, where electronic arcana like click-throughs and ad impressions are the accepted currencies. Apps would return media to its proper, historical structure: publishers could sell digital versions of the same ads that appeared in their print publications (perhaps with a markup if the ad had interactive elements), valued with the old measurement of rate base.

People lost their heads. One symptom of the industry's euphoria was a brief-lived literary genre, the announcement of the iPad edition. In late 2010, the New Yorker's editors gushed: "This latest technology ... provides the most material at the most advanced stage of digital speed and capacity. It has everything that is in the print edition and more: extra cartoons, extra photographs, videos, audio of writers and poets reading their work. This week's inaugural tablet issue features an animated version of David Hockney's cover, which he drew on an iPad." Giddiest of all was the chief executive of News Corp., Rupert Murdoch: he lavished $30 million on the launch of The Daily, an experimental iPad-only newspaper with a $39.99 subscription price.

Unpacked in this fashion, the delusion is clear enough, but I succumbed myself—at least a little. I never believed that apps would unwind my industry's disruption, but I felt that some readers would want a beautifully designed digital replica of Technology Review on their mobile devices, and I bet that our developers could create a better mobile experience within applications. I liked the idea of bundling inside an app all the editorial we produced, including the daily news and video we post to TechnologyReview.com. So we created iOS and Google Android apps that were free to download: anyone could read our daily news or watch our videos without charge, but they had to pay to see digital replicas of the magazine.

We launched the platforms in January of 2011. Complimenting myself on my conservatism, I budgeted less than $125,000 in revenue in the first year. That meant fewer than 5,000 subscriptions and a handful of single-issue sales. Easy, I thought. What could go wrong?

Like almost all publishers, I was badly disappointed. What went wrong? Everything.

Apple demanded a 30 percent vigorish on all single-copy sales through its iTunes store. While publishers were accustomed to handing over as much as 50 percent to newsstand distributors, the depth of the cut smarted because it was unexpected; many publishers responded by not selling single copies in Apple's store. Then, for a year after the launch of the iPad, Apple couldn't work out how to sell subscriptions in a way that satisfied ABC, which requires publishers to record "fulfillment" information about subscribers. When Apple finally solved the problem of transferring fulfillment data to publishers, it again claimed its 30 percent share. That hurt more than the vig on single-issue sales: publishers have always hated sharing subscription revenues with third parties, a business they associate with shady resellers who traffic in notoriously disloyal readers. Starting in June of last year, Apple did allow publishers to directly fulfill subscriptions through their own Web pages (a handful of publishers, including Technology Review, had enjoyed the privilege earlier), but the mechanism couldn't match iTunes for ease of use. Google was more reasonable in its terms, but Android never emerged as a significant alternative to the iPad: today, most tablet computers are still Apple machines.

The real problem with apps was that when people read on electronic media, they expect the stories to possess thelinky-ness of the Web—but stories in apps didn't really link.

There were other difficulties. It turned out that it wasn't at all simple to adapt print publications to apps. A large part of the problem was the ratio of the tablets: they possessed both a "portrait" (vertical) and "landscape" (horizontal) view, depending on how the user held the device. Then, too, the screens of smart phones were much smaller than those of tablets. Absurdly, many publishers ended up producing six different versions of an editorial product: a print publication, a conventional digital replica for Web browsers and proprietary software, a digital replica for landscape viewing on tablets, something that was not quite a digital replica for portrait viewing on tablets, a kind of hack for smart phones, and ordinary HTML pages for their websites.

Software development of apps was much harder than publishers had anticipated, because they had hired Web developers who knew technologies like HTML, CSS, and JavaScript. Publishers were astonished to learn that iPad apps were in fact real, if small, applications, written mostly in a language called Objective C, which no one in their Web-dev departments knew. Publishers responded by outsourcing app development, which was expensive, time-consuming, and unbudgeted.

But the real problem with apps was more profound. When people read news and features on electronic media, they expect the stories to possess the linky-ness of the Web—but stories in apps didn't really link. The apps were, in the jargon of information technology, "walled gardens," and although sometimes beautiful, they were small and stifling. For readers, none of the novelty or prettiness of apps overcame the weirdness and frustration of reading digital media closed off from other digital media.

The Daily's fortunes were not atypical: the publication has found only 100,000 subscribers, well short of the half-million Rupert Murdoch said would be necessary to make it a viable business. The gloom was general. With few subscribers and single-copy buyers, there were no audiences to sell to advertisers, and therefore no revenues to offset the incremental costs of app development. Most publishers soured on apps.

The most commonly cited exception to the general bitterness is Condé Nast, which saw its digital sales increase by 268 percent last year after Apple introduced an iPad app called Newsstand. Still, even 268 percent growth may not be saying much in total numbers: digital is a small business for Condé Nast. Wired magazine, among the most digital of Condé Nast's titles, had 33,237 digital-only subscriptions last year, representing just 4.1 percent of a total circulation of 812,434, and 7,004 digital single-copy sales, which is 0.8 percent of paid circulation, according to ABC. (Wired spins the numbers differently, claiming a digital circulation of 108,622; but that sum includes the 68,380 print subscribers who activated free digital access.) Similarly, the New Yorker, another Condé Nast publication, last year had only 26,880 digital-only subscribers among its million subscribers.

Today, most owners of mobile devices read news and features on publishers' websites, which have often been coded to adapt themselves to smaller screens. Or, if they do use apps, the apps are glorified RSS readers, such as Google Reader, Flipboard, and the apps of newspapers like The Guardian, which grab editorial from the publisher's site. A recent Nielsen study reported that while 33 percent of tablet and smart-phone users had downloaded news apps in the previous 30 days, just 19 percent of users had paid for any of them. Apps are good for plenty of things: you can use them to translate street signs in a foreign city, or discover the cheapest bulk source of floor wax, or, if you're carefree and so inclined, hook up with willing partners. But the paid, expensively developed publisher's app, with its extravagantly produced digital replica, is increasingly uncommon.

The recent history of the Financial Times is instructive. Last June, the company pulled its iPad and iPhone app from iTunes and launched a new version of its website written in HTML5, which can optimize a site for any device and provide features and functions that are app-like. For a few months, the FT continued to support the iPad and iPhone app, but on May 1, the paper chose to kill it altogether.

And Technology Review? We sold 353 subscriptions through the iPad. We never discovered how to avoid the necessity of designing both landscape and portrait versions of the magazine for the app. We wasted $124,000 on outsourced software development, a sum that does not begin to capture our allocation of internal resources. We fought among ourselves, and people left the company. There was untold expense of spirit. I hated every moment of our experiment with apps, because it tried to impose something closed, old, and print-like on something open, new, and digital.

Last fall, in version 3.0 of our apps, we moved the editorial content, including the magazine, into simple RSS feeds in "rivers of news." We dumped the digital replica altogether. Now we're redesigning 
TechnologyReview.com, which we have made free to use, and we'll follow theFinancial Times in using HTML5, so that our Web pages will look great on a laptop or desktop, tablet, or smart phone. Then we'll kill our apps, too. Now we just need to discover how to make the Web pay.

Are Smart Phones Spreading Faster than Any Technology in Human History?

Mobile computers are on track to saturate markets in the U.S. and the developing world in record time.

Today's technology scene seems overheated to some. Apple is the most valuable company on earth. Software apps are reaching tens of millions of users within weeks. Major technology names like Research in Motion and Nokia are being undone by rapid changes to their markets. Underlying these developments: the unprecedented speed at which mobile computers are spreading.

Presented below is the U.S. market penetration achieved by nine technologies since 1876, the year Alexander Graham Bell patented the telephone. Penetration rates have been organized to show three phases of a technology's spread: traction, maturity, and saturation.

Those technologies with "last mile" problems—bringing electricity cables or telephone wire to individual homes—appear to spread more slowly. It took almost a century for landline phones to reach saturation, or the point at which new demand falls off. Mobile phones, by contrast, achieved saturation in just 20 years. Smart phones are on track to halve that rate yet again, and tablets could move still faster, setting consecutive records for speed to market saturation in the United States.

It is difficult to conclude categorically from the available data that smart phones are spreading faster than any previous technology. Statistics are not always available globally, and not every technology is easily tracked. Also, because smart phones have not yet reached market saturation, as electricity and television have, the results are still coming in.

The Sudden Rise of the Smart Phone

BellSouth launched the IBM Simon, with its rudimentary touch screen, back in 1993, but the era of the smart phone in America really began in 2002, when existing PDAs took on the ability to make phone calls. That year RIM shipped its first BlackBerry with phone features, Handspring launched its Palm-OS-powered Treo line, Microsoft shipped its Pocket PC Phone Edition, and mobile data technology such as GPRS became increasingly widespread.

Four and a half years later, in late 2006, the quarter before Apple announced its now-iconic iPhone, only 715,000 smart phones were sold, representing just 6 percent of U.S. mobile-phone sales by volume. Up to that point, the smart phone was spreading not much faster than personal computers had in the preceding decades, and more slowly than radio decades before.

That changed when Apple's iPhone sold 1.12 million units in its first full quarter of availability, despite prices starting at $399. Year over year, the market share of smart phones almost doubled, to 11 percent of U.S. mobile-phone sales. Now Nielsen reports that smart phones represent more than two-thirds of all U.S. mobile-phone sales. Nielsen also reports that 50 percent of all U.S. mobile-phone users—which equates to about 40 percent of the U.S. population—now use smart phones.

These figures show that smart phones, after a relatively fast start, have also outpaced nearly any comparable technology in the leap to mainstream use. It took landline telephones about 45 years to get from 5 percent to 50 percent penetration among U.S. households, and mobile phones took around seven years to reach a similar proportion of consumers. Smart phones have gone from 5 percent to 40 percent in about four years, despite a recession. In the comparison shown, the only technology that moved as quickly to the U.S. mainstream was television between 1950 and 1953.

The Mobile Phone Was Truly Global

How rapid is the spread of smart phones globally? For the rest of the world, historical adoption rates of technologies such as TV, radio, and the Internet aren't as generally available. Further, in many regions, like Africa, smart phones are a recent phenomenon. That makes comparisons difficult. However, the unprecedented spread of simpler "feature" models of mobile phones in the developing world appears to put smart phones on a global fast track.

In 1982, there were 4.6 billion people in the world, and not a single mobile-phone subscriber. Today, there are seven billion people in the world—and six billion mobile cellular-phone subscriptions. As with many technologies, the explosion began in the world's most developed countries.

Historically, a technology that reaches saturation in rich countries still spreads through the developing world only in correlation to each country's state of development. In 1963, researchers famously mapped the GDP of nations against their "teledensity," the prevalence of landline telephones. The data showed just this effect, which is known as the Jipp Curve.

The mobile phone, however, is a landmark: over the last decade, the correlation between wealth and teledensity has been completely transformed.

According to the International Telecommunications Union, in 2001 the developed world had six times as many mobile subscriptions per capita as the developing world. By 2011, that gap had collapsed to just 50 percent more phones per capita, and it continues to narrow substantially. Of the world's six billion mobile-phone subscriptions, 73 percent are now in the developing world, even though those countries account for just 20 percent of the world's GDP.

Today, 136 years after Bell received his U.S. patent for an "improvement in telegraphy," only 17 countries have as many as one telephone line for every two people. Less than 30 years after Ameritech phoned Bell's grandson in America's first commercial cellular call, an astounding 158 out of 200 countries the World Bank monitors have passed that threshold with mobile phones—including countries such as Senegal, where the average income is only $5 per day.

The Smart Phone Will Be Global, Too

Although the large majority of mobile phones in the world aren't yet smart phones, the "dumb phones" have established the infrastructure, payment and distribution systems, and networks that are increasingly utilized by smart phones.

The ITU claims that 90 percent of the world's population is already covered by 2G networks, many of which can provide data services like Internet access via slower "2.5G" technologies such as EDGE and GPRS. The more modern 3G networks that have catalyzed the current smart-phone boom by providing richer, quicker mobile experiences have been expanding rapidly and now cover 45 percent of the world's population, more than three billion people.

The cost of a smart phone and a service plan clearly remains an important barrier in poor nations, but it is a shrinking one. ARM Holdings' Cortex A7 mobile CPU, expected in phones next year, is touted as a way to get smart phones to "the next billion people," with a price-to-performance ratio five times that of 2010 models. Meanwhile, the Chinese firm Spreadtrum has already released a chip platform targeting sub-$50 Android smart phones. Despite plummeting device prices, accessible mobile data pricing will be critical as well.

The inevitable trend is already clearly visible. According to IDC, smart phones accounted for 36 percent of global mobile-phone shipments in the first quarter of 2012, up from 25 percent a year earlier. If smart phones continue to gain at even this pace, "feature phones" will be largely a memory in another five years. It remains to be seen whether networks the world over can support such a rapid conversion to smart phones.

The Next Boom and Beyond

Arriving in the wake of smart phones, tablets appear poised for even swifter adoption. After years of false starts, the tablet market sprang to life with the launch of Apple's iPad in April 2010. Only 18 months later, tablet penetration among U.S. households had already hit 11 percent, according to a Google/Ipsos study. No other technology in this comparison has had such a fast start. Since that date, Amazon's (essentially U.S.-only) Kindle Fire was introduced and sold at least five million units. In the last two quarters, Apple has also sold approximately 10 million more iPads in the U.S. market. As a result, the number of consumers in the U.S. who own a tablet computer now exceeds 13 percent just two years into the market's existence.

According to Gartner, there are now at least 1.4 billion PCs in use worldwide. It remains to be seen whether tablets can maintain their record-setting pace. Mobile phones, on the other hand, are already selling more than 1.4 billion units every single year. One thing seems certain: squeezed between tablets and ever-smarter phones, the PC is seeing its reign as the world's "personal" computer draw to a close.