Pokemon Go Mapped the World. Now It's Mapping You

You thought this was just a game? JL

Cecelia D'Anastasio and Dhruv Mehrota report in Kotaku:

Today, when you use Wizards Unite or Pokémon Go or any of Niantic’s other apps, your every move is getting documented and stored—up to 13 times a minute. Even players who know that the apps record their location data are usually astonished once they look at how much they’ve told Niantic about their lives through their footsteps. Ubiquitous computing is ready. The fantasy is that any system mediating someone’s personal experience of the physical world that uses a modern corporation’s digital infrastructure would be objective or neutral. Humans are data and data is money, and this is the business model of many technology firms

Harry Potter: Wizards Unite, the latest game from the company behind Pokémon Go, lets players harness the magic of their childhood to combat monsters and collect shimmering digital artifacts across their local neighborhoods. Niantic’s apps certainly encourage gamers to get outdoors and get active, but behind the scenes, Wizards Unite is quietly casting another spell: collecting a surprising amount of data about where you go.

Wizards Unite is an augmented reality mobile game developed by Niantic in partnership with Warner Brothers. Millions of people play the game—not nearly as many as Pokémon Go’s impressive 150 million active users in 2018, two years after launch, but on release, it rocketed to the top of the app store with hundreds of thousands of downloads. Niantic’s apps regularly dominate app stores, just as the company itself dominates the world of augmented reality gaming. They’re the latest products from a cell of technologists that’s been in operation since 2001, being acquired by and then spun off from Google, while never losing sight of its ultimate goal: to map the entire world digitally and capitalize on that data.

Today, when you use Wizards Unite or Pokémon Go or any of Niantic’s other apps, your every move is getting documented and stored—up to 13 times a minute, according to the results of a Kotaku investigation. Even players who know that the apps record their location data are usually astonished once they look at just how much they’ve told Niantic about their lives through their footsteps.

For years, users of these technologists’ products—from Google Street View to Pokémon Go—have been questioning how far they’re going with users’ information and whether those users are adequately educated on what they’re giving up and with whom it’s shared. In the process, those technologists have made mistakes, both major and minor, with regards to user privacy. As Niantic summits the world of augmented reality, it’s engineering that future of that big-money field, too. Should what Niantic does with its treasure trove of valuable data remain shrouded in the darkness particular to up-and-coming Silicon Valley darlings, that opacity might become so normalized that users lose any expectation of knowing how they’re being profited from.

---

Niantic publicly describes itself as a gaming company with an outsized passion for getting gamers outside. Its games, from Ingress to Pokémon Go to Wizards Unite, encourage players to navigate and interact with the real world around them, whether it be tree-lined suburbs, big cities, local landmarks, the Eiffel Tower, strip malls, or statues in the town square. Niantic’s ever-evolving gaming platform closely resembles Google Maps, in part because Niantic spawned from just that.

“In every neighborhood, there’s probably a story that’s interesting. There’s a mystery that can be built here. It’s this idea of exploration,” said Niantic’s founder and CEO John Hanke during a talk at the 2019 Game Developers Conference in San Francisco. Dressed down in a red Niantic T-shirt, his trademark swoop of brown-grey hair across his temple, he described the tripartite mission of his company: Exploration. Exercise. Social.

“I’m a parent with kids,” he said. “And when you become a parent it’s like you have a second chance, you can reset. I was thinking of all the places I wanted to go and explore with my kids. We started Niantic with those themes in mind.” Pokémon Go players have walked a collective 14.3 billion miles since its release in July 2016. If the new, and similar, Wizards Unite has the same staying power, perhaps its players will cover similar ground in the years to come. (During the course of reporting this story over several months, Kotaku attempted multiple times to secure an interview with Hanke. Niantic never granted us an interview with its CEO in person or by phone, but Hanke did eventually provide us with emailed responses to some of our questions.)

In his GDC talk, Hanke continued on to describe another one of Niantic’s goals, one that may read as innocuous to starry-eyed ‘90s kids who wanted nothing more than a Poké-pal to travel with: “to build stuff that brings the digital and physical world together.” Depending on how you’re looking at it, that can mean an immersive Pokémon experience or, in Hanke’s terms, “ubiquitous computing,” which he said in 2016 meant adding “computing interfaces to everything you encounter in the physical world.”

“This is about being able to walk around with glasses and know more about the world,” said Hanke at his 2019 GDC talk. “You might walk by a restaurant and understand what Yelp has to say about that restaurant,” he continued. “This is a possible future that we might live in.”

Ubiquitous computing isn’t sitting at your desktop playing World of Warcraft for 13 hours, or even using your phone to navigate from your friend’s house to the nearest liquor store. It’s the seamless integration of the digital world into the physical one, anywhere, at any time. Also referred to as “mediated reality,” ubiquitous computing, to some, is the science fiction fantasy of Star Trek, where tasks are mediated by an objective, neutral computation system. It was the vision of Google Glass, the $1,500 device that, through voice commands, could take photos, search on Google, view your calendar or receive travel alerts. Google Glass never caught on, although in his talk, Hanke was adamant that the future of AR is wearable technology.

At 2019’s GDC, Hanke showed a video titled “Hyper-Reality,” by the media artist Keiichi Matsuda. It’s a dystopian look at a future in which the entire world is slathered with virtual overlays, an assault on the senses that everyone must view through an AR headset if they want to participate in modern society. In the video, the protagonist’s entire field of vision is a spread of neon notifications, apps, and advertisements, all viewed from a seat at the back of a city bus. Their hands swipe across a game they’re playing in augmented reality, while in the background an ad for Starbucks Coffee indicates they won a coupon for a free cup. Push notifications in their periphery indicate three new messages and directions for where to exit the bus. Walking through the aisle, where digital “get off now!” signs indicate it’s their stop, and onto the street, the physical world is annotated with virtual information. The more tasks they accomplish, the more points they receive. The whole world is now one big game. It showed a definitively dystopian vision of a world in which the barriers between IRL and URL have been fully collapsed.

Hanke said that the video made him feel “stressed and nervous.” Calling it a work of “critical design,” he noted that it was meant to question this dystopian future for AR, “a world where you’re tracked everywhere you go, where giant companies know everything about you, your identity is constantly at stake, and the world itself is noisy, and busy and plastered with distractions.”

But when a path appeared in front of the video’s protagonist showing them where to walk, Hanke’s response was: “That looks helpful.”

“Some people would say AR is a bad thing because we’ve seen this vision of how bad it can be,” Hanke said. “The point I want to make to you all is, it doesn’t have to be that way.” He showed an image of the Ferry Building, the 120-year-old piece of classical revival architecture in San Francisco where the company is currently headquartered. Just like in the video, it was overlaid with augmented reality windows showing the building’s history, a public transit schedule, and tabs for nearby restaurants. Hanke described a world where people can better navigate public transit and understand their surroundings because of digital mapping initiatives like Niantic. He talked about the possibility of hologram tour guides in San Francisco, and how they’d rely on a digital map to navigate their surroundings, and about designing shared experiences of Pokémon games in a Pokémon-augmented world. Cute, nice things for nice people who love cute things.

Ubiquitous computing is still a fantasy, but not because the technology isn’t ready. It is. The fantasy is that any system mediating someone’s personal experience of the physical world that uses a modern corporation’s digital infrastructure would be objective or neutral. Humans are data and data is money, and this is the business model of many of the technology firms up to the task of ubiquitous computing.

Despite Hanke’s apparent cognizance of how, in the wrong hands, location-based AR apps can intrude and prey on reality, it remains to be seen whether the globe-sized snowball he’d started rolling at the dawn of his career in mapping and location tech is one that any one man could stop himself.

Niantic’s headquarters in the Ferry Building are located just a few blocks down from what remains of its namesake, a former whaling vessel from the mid-1800s. Known for ferrying entrepreneurs to California during the gold rush, the Niantic was eventually brought aground in San Francisco and turned into a hotel. The hotel is gone, but the hull of the Niantic remains buried under the streets of modern San Francisco, a plaque marking its location. The plaque is now a Pokémon Go gym.

John Hanke’s takeaway from the story of the ship buried beneath the skyscrapers, as he explained it in a 2016 talk to the Berkeley School of Business, was that “there was this hidden information out in the world,” and he wanted to develop “technologies can help you learn about these things as you move about in the physical world.”

Born in 1967 and raised in the tiny thousand-person town of Cross Plains, Texas, Hanke found a bag full of National Geographic magazines as a child and fell in love with the maps—the Nile Delta, the far East—which he saw as a way of exploring far-away places from the isolation of his small town. Later, Hanke would buy himself a computer with the money he’d earned mowing lawns. In lieu of taking over his father’s small ranch operation, Hanke focused his attention on school, where he would win a statewide computer programming competition. (Hanke’s biography is laid out in the 2018 behind-the-scenes book Never Lost Again, a history of Google Maps written by Keyhole/Google/Niantic executive Bill Kilday. When Kotaku asked whether that biographical information was accurate, Hanke said that he had not read the entire book.)

After a brief post-college stint with the U.S. Foreign Service, Hanke joined the game developer Archetype Interactive and shipped Meridian 59, the first massively multiplayer online role-playing game with 3D graphics. Later, after getting his MBA from the University of California Berkeley, Hanke was tapped to lead a company developing state-of-the-art mapping technology using satellite imaging and aerial photography. Keyhole, named for the reconnaissance satellites that the United States used starting in 1959 to spy on other countries, did mapping better than anything before it. Keyhole reinvented mapping for the digital era of the early 2000s. Hanke’s mission, he said at GDC 2019, was to “build a map to put the whole world on your desktop”—an audacious vision in 2001, when the cutting edge of computer mapping was printing out MapQuest directions and hoping for the best.

The CIA’s venture capital investment firm, In-Q-Tel, took a particular interest in Keyhole. According to a 2003 press release, within two weeks of In-Q-Tel’s investment, Keyhole’s tech was implemented to support Pentagon initiatives in Iraq.

In-Q-Tel’s leader at the time was Gilman Louie, a video game industry veteran who’d designed or produced a handful of flight simulators, including the Falcon series. Louie, who currently sits on the board of Niantic and whose venture capital firm is a major investor in the company, has had a decades-long relationship with Hanke, apparently fueled by their shared enthusiasm for mapping and augmented reality. Keyhole also proved seductive to Google, whose cofounder Sergey Brin led the initiative to acquire Keyhole for a reported $35 million in 2004.

Google began to buy up other mapping tech companies, especially ones in the business of taking aerial or 3D images of cities around the world. According to Never Lost Again, it took a while for Keyhole employees to realize that its new parent company had grand plans to map the whole entire world in 3D, and make it searchable.

“I don’t recall them articulating a pre-existing desire to digitally map the world in the way you describe,” said Hanke via email when Kotaku asked whether the account was accurate. “I remember Google executives (during our pre-acquisition meetings) being very excited about the vision we described to them of creating an ‘earth browser’ to explore information using the earth itself as the discovery mechanism.” Hanke would help head up Google Geo, the name for Google’s map, Street View, and Earth projects.

The February 2005 launch of Google Maps, soon followed by the launch of Google Earth, were met with falling-over-itself fanfare from the California techies. It was the first free, interactive, global visual map of Earth. In its first 24 hours, Google Earth was downloaded 450,000 times. Users were overwhelmed by the opportunity to be at the helm of a digital starship that could land basically anywhere. Foreign governments trembled, citing concerns that private military facilities would be discovered. (“They ought to have asked us,” said India’s surveyor general to the New York Times.) For his part, in interviews, Hanke leaned into the good Google Earth was doing, like facilitating wildfire mitigation or hurricane relief efforts.

Google Earth and its cousins were also immediately the darling of big brands. Target painted its logo on some of its stores’ rooftops so they could be seen from the overhead view. Volkswagen worked with Hanke to convert Google’s maps into in-car navigation modules. By April of 2004, before Keyhole was even acquired by Google, Google was looking into marketing Google’s Search by Location function to advertisers. Now, businesses could target ads geographically. This was over ten years ago, and the birthing process for this advanced form of surveillance capitalism is still in its experimentation stages.

---

It cost Google a lot of money to acquire all this data. It paid an army of telemarketers to pester tens of millions of businesses worldwide for up-to-date information on their operations and locations. It paid tens of millions of dollars to organizations like Digital Globe for their satellite imagery of cities. When Google launched Street View in 2007, bringing eye-level perspectives of major cities to users with smartphones or computers, its data collection methods were a little different, and a little more controversial.

Those methods were apparently so important that their inception involved several of the most important people at the company. As the story goes in Never Lost Again, it was Google co-founder Larry Page who first took his car and a camcorder out one Saturday to California’s Half Moon Bay, with Sergey Brin and the head of Google’s search team Marissa Mayer in the back seat. This C-suite road trip would eventually result in a fleet of Google vans that by 2009 would map 13 million miles of the physical world across 22 countries. Homes, mailboxes, front lawns, bikini-clad sunbathers—Google grabbed it all.

In 2010, Germany’s data protection commission noticed something shady about Google’s fleet of image-vacuuming vans. Their sensors were eating, or “sniffing,” traffic from wireless networks that were unencrypted, and in the process, scooping up emails, medical information, financial records, audio and video files, and passwords. Hanke, still in charge of Google Geo, said he didn’t know anything about it. Google later apologized, saying it had “mistakenly included code in our software that collected samples of payload data from WiFi networks.”

As the 2019 book The Age of Surveillance Capitalism explains it, Google did know something about it: “The records showed that the engineer had emailed links to his software documentation to project leaders, who then shared them with the entire Street View team. [The FCC] also found evidence that on at least two occasions, the engineer told his colleagues that Street View was collecting personal data.”

The so-called “Wi-Spy scandal” made waves across watchdog blogs and tech media. “Asking Google to educate consumers about privacy is like asking the fox to teach the chickens how to ensure the security of their coop,” wrote Consumer Watchdog. Data was becoming the pillar of tech companies’ business strategies, and legitimately or not, consumers began fearing that their information was, in fact, the actual commodity these companies dealt in. (Never Lost Again completely omits the event from its history of Google Maps.)

“After the mapping part was done, we kind of turned to the next obvious question,” said Hanke in a 2019 speech. “You have this amazing service, this unprecedented amount of detail about the planet. What other kinds of things could you build with it?” By 2010, the year of the “Wi-Spy” scandal, Hanke was growing frustrated in his position as a vice president of Geo. He pivoted to launching Niantic Labs as an internal startup within the Google mothership.

“If you have mobile computing and services people are using while moving around in the world,” said Hanke in a 2016 talk to the Berkeley School of Business, “the question we asked ourselves was, ‘Could these services influence how people behave in the physical world? Could the products they’re using cause them to walk a different path, drive a different path, divert from the trajectory that they’re normally going to go on? If you could do that through information services that you’re offering to people, there’s tremendous opportunity for businesses that might want to change the behavior of people to get them to go places where they wouldn’t otherwise go.” He added: “We were also interested in being an advertising company.”

Niantic Labs’ first app was called “Field Trip.” Developed primarily for Google Glass, it would pop up interesting factoids about landmarks and locations around you as you wandered. Hanke called it an example of “ubiquitous computing.”“When you’re out walking with your family, you’re not going to pause every 20 feet and do a Google search,” he said in a 2012 interview with the Mercury News. “So the notion is that you can have this process that runs in the background that knows something about where you are, and about your interests. It can proactively offer up information that can help you have a richer experience but in a way that’s seamless and doesn’t interrupt the flow of your activity.” You don’t even have to unlock your phone.

“Niantic was created to explore applications for various kinds of wearable computing devices that we believed would some day replace the mobile phone,” Hanke told Kotaku. But Niantic quickly shifted focus to phone games starting with 2012's Ingress. Niantic’s games would all be location-based, augmented-reality experiences, games that sought to collapse the barrier between the digital and the physical. Niantic would crowdsource cultural and physical landmarks from players and spin them into gameplay mechanics.

In Ingress, “agents” (players) use their “scanners” (cell phones) to overtake “portals” (specific IRL landmarks), in a game of hacker capture-the-flag. The groundbreaking game still sustains a medium-sized, dedicated following. Niantic spun out from Google in 2015, with Google setting it up with some seed money. A generous investment from The Pokemon Company and Nintendo helped. A year later, the launch of Pokémon Go would be three times bigger than those of Google Maps and Google Earth.

Since its 2016 release, Pokémon Go has netted over $2.3 billion. In it, players collect items from PokeStops—also real-life locations and landmarks—so they can catch and collect Pokémon, which spawn around them. Almost immediately, Pokémon Go sparked its own privacy controversy, also blamed on a bug, which involved users giving Niantic a huge number of permissions: contacts, location, storage, camera and, for iPhone users, full Google account access, which was not integral to gameplay. Minnesota senator Al Franken penned a strongly-worded letter to Niantic about it, expressing concern “about the extent to which Niantic may be unnecessarily collecting, using, and sharing a wide range of users’ personal information without their appropriate consent.” Niantic said that the “account creation process on iOS erroneously requests full access permission,” adding that Pokémon Go only got user ID and email address info.

Three years later, Niantic would release Wizards Unite, an augmented reality game where players collect “Confoundables” around their neighborhood, fight enemies with magic, and interact with location-locked resources like inns and greenhouses. What makes Wizards Unite an “augmented reality” game is the fact that, when a player encounters something magical they must deal with in-game, the phone’s camera turns on. In the foreground of the player’s office desk, commuter train, or neighborhood street, some Harry Potter-ish object or monster appears. Between these encounters, the game idly follows the player as they walk down the street, which is dotted with magical, interactable things.

Running all of these games is what Niantic calls its Real World Platform, an “operating system” that allows other developers to build their own apps and games that use all of the work that Niantic has already done in mapping and tagging the world. Niantic hopes that eventually other developers will make more games using its Real World Platform, leading to a proliferation of its tech.

“I think you can create over time more and more types of applications utilizing our platform in the future,” said Niantic’s vice president of product management, Kei Kawai. “Right now, our focus is more on the game.” In the future, Niantic says on their website they’re planning on releasing a “planet-scale real-world AR platform,” which would more believably integrate game experiences into a live camera feed of the real world.

From the player’s perspective, the Wizarding World is manifesting itself into their real one. But what’s happening behind the scenes, and what does Niantic see? Players give Wizards Unite permission to track their movement using a combination of GPS, Wi-Fi, and mobile cell tower triangulation. To understand the extent of this location data, Kotaku asked for data from European players who had all filed personal information requests to Niantic under the GDPR, the European digital privacy legislation designed to give EU citizens more control over their personal data. Niantic sent these players all the data it had on them, which the players then shared with Kotaku.

The files we received contained detailed information about the lives of these players: the number of calories they likely burned during a given session, the distance they traveled, the promotions they engaged with. Crucially, each request also contained a large file of timestamped location data, as latitudes and longitudes.

In total, Kotaku analyzed more than 25,000 location records voluntarily shared with us by 10 players of Niantic games. On average, we found that Niantic kept about three location records per minute of gameplay of Wizards Unite, nearly twice as many as it did with Pokémon Go. For one player, Niantic had at least one location record taken during nearly every hour of the day, suggesting that the game was collecting data and sharing it with Niantic even when the player was not playing.

When Kotaku first asked Niantic why Wizards Unite was collecting location data even while the game was not actively being played, its first response was that we must be mistaken, since the game, it said, did not collect data while backgrounded. After we provided Niantic with more information about that player, it got back to us a few days later to let us know that its engineering team “did identify a bug in the Android version of the client code that led it to continue to ping our servers intermittently when the app was still open but had been backgrounded.” The bug, Niantic said, has now been fixed.

Because the location data collected by Wizards Unite and sent to Niantic is so granular, sometimes up to 13 location records a minute, it is possible to discern individual patterns of user behavior as well as intimate details about a player’s life. As an experiment, we approached players with insights we gleaned by combing through five days’ worth of their data, to find out how accurate our inferences were.

In five days of gameplay, Niantic kept 2304 location records for one player. By looking at the timestamps and frequency with which this user would return to particular areas, we were able to correctly identify their employer and address of residence. Furthermore, by plotting these data points on a map we could correctly discern the routes the user took from work to home, their daily schedule, and even their eating habits. When we asked them about their propensity to eat Burger King for lunch, they were surprised that we knew that, saying afterwards that they were “addicted to fast food.”

While identifying an individual player’s routines might tell you a lot about their life, oftentimes looking at deviations from that routine can be more revealing. Another player, Laszlo, usually wakes up between 7 and 8 a.m. in his house in a tree-lined neighborhood on the Buda side of Budapest. At around 9:30, he travels to work, always in a vehicle, always passing a Starbucks where he “could catch at least three creatures every time.” (Niantic says that while Pokémon will appear more frequently near Pokestops and Gyms, they do not appear more frequently near sponsored locations.)

However, on one particular Monday, Laszlo deviated from his predicted path and visited a pharmacy near a shopping mall. While the rest of his day is unremarkable, in the evening, instead of taking his usual midnight stroll to the soccer fields, Laszlo’s data went dark until the morning.

Kotaku asked Laszlo if he had been sick that day. He laughed. “The pattern is that I usually walk my dogs quite late to that sports complex… and I have two kids… so it happens quite often that I have to buy nasal drops from that pharmacy when they get a cold,” he said. In itself, this deviation was not a juicy revelation. However, it’s hard to imagine that Niantic, with all of its location intelligence, doesn’t have records of players traveling to revealing locations that they might not want others to know about.

After we reviewed the data, the Burger-King loving player was surprised to find that the game would “collect so much,” calling the app’s behavior “scary” and “unexpected.” They were in the habit of leaving the game running in the background, they said, but now that they “saw all the data,” they “close the game fully more times” during the day.

This is not the only player who changed their behavior once they obtained their data from Niantic. One player who lives in Spain told Kotaku that she was surprised by the files Niantic returned to her, and has now become more intentional about where she plays the game.

“I try not to play in my house and other places where I go all the time, because I don’t want them to know my routine and all that,” she told Kotaku, saying she felt deceived by Niantic. “I don’t understand why they have to keep all that information … It’s like they are trying to control us.”

“Niantic’s privacy policy details the data we collect and process in order to provide location-based games to our users,” the company told Kotaku. “A user who is uncomfortable with their report may also request that Niantic delete their data.”

Niantic is far from the only company collecting this sort of data. Last year, the New York Times published an expose on how over 75 companies receive pinpoint-accurate, anonymous location data from phone apps on over 200 million devices. Sometimes, these companies tracked users’ locations over 14,000 times a day. The result was always the same: Even though users had signed away their location data to these companies by agreeing to their user agreements, a lot of the time, they generally had no idea that companies were taking such exhaustive notes on what kind of person they are, where they’d been, where they were likely to go next, and whether they’d buy something there.

That Niantic is yet another company that can infer this type of mundane personal information may not be, in itself, surprising. Credit card companies, email providers, cellular services, and a variety of data brokers all have access to your personal information in increasingly opaque ways. Remember when Target figured out that a high school girl was pregnant before her family did?

It’s important to note that the personal data that players requested from Niantic and voluntarily shared with Kotaku is, according to Niantic, not something that a third party could buy from them, or otherwise be allowed to see. “Niantic does not share individual player data with third party sponsored location partners,” a representative said, adding that it uses “additional mechanisms to process the data so that it cannot be connected to an individual.”

Niantic’s Kawai told Kotaku that the anonymized data that Niantic shares with third parties is only in the form of “aggregated stats,” such as “how many people have had access or went to those in-game locations and how many actions people take in those in-game locations, how many PokeStop spins to get items happened on that day and… what unique number of people went to that location.”

“We don’t go any further than that,” he said.

The idea that data can successfully be anonymized has long been a contentious one. In July, researchers at Imperial College London were able to accurately reidentify 99.98 percent of Americans in an “anonymized” dataset. And in 2018, a New York Times investigation found that, when provided raw anonymized location data, companies could identify individuals with or without their consent. In fact, according to experts, it can take just four timestamped location records to specifically identify an individual from a collection of latitudes and longitudes that they have visited.

“‘Anonymous data’ is a contradiction in terms,” said Justin Brookman, a privacy expert at Consumer Reports who formerly worked for the Federal Trade Commission. “I think it’s bad practice if information shared for one narrow purpose—especially sensitive data like geolocation—is shared opaquely with third parties,” he said. “There should be law about this to constrain third-party sharing of data beyond what’s reasonably necessary for an app to work. There’s no need for Burger King or Starbucks to know I was catching Charmander in there.”

“Any time an app collects location data, you have to hope the app developer has given some thought to the risks involved, but this is hard to tell from a privacy policy alone,” said Joseph Jerome, a privacy consultant who formerly served as policy counsel for the Privacy & Data Project at the Center for Democracy & Technology. “A lot of companies will say that IP addresses and other technical information is not personal. Niantic is not making [these claims], which is a good thing.”

---

Niantic’s focus might be on games for now, but its investors may be looking to a post-Pokémon future. In January, Niantic raised $245 million in one round of funding. Big-name players in the tech industry have contributed, like Samsung Ventures, Battery Ventures, and IVP, who says Niantic is “delivering an operating system for applications that unite the digital world with the physical world.” Niantic’s current valuation is over $4 billion, technically qualifying it as a “unicorn” startup, higher on the valuation list than CreditKarma or 23andMe.

“We have been clear in our statements about our desire to create broadly appealing entertainment products that follow our goals of encouraging exploration, exercise, and real-world social interaction as well as our desire to create the Niantic Real World Platform and supporting technologies such as the AR Cloud,” said John Hanke in an email. “I believe investors value us based on those goals.”

Niantic makes a staggering amount of money off in-game microtransactions, a reported $1.8 billion in Pokémon Go’s first two years. It also makes money from sponsorships. By late 2017, there were over 35,000 sponsored PokeStops, which players visited over 500 million times. Hanke described foot traffic as the “holy grail of retail businesses” in a 2017 talk to the Mobile World Congress. 13,000 of the sponsored stops were Starbucks locations. Starbucks offered a sickeningly sweet custom Frappuccino if you played Pokémon Go while in one of its stores. Sprint put lures in front of their stores to attract more Pokémon, and if people walked into a Sprint and inquired about their services, the company’s Trainer Reward Program would offer incentives to retain them.

“We have always been transparent about this product and feel it is a much better experience for our players than the kind of video and text ads frequently deployed in other mobile games,” Hanke told Kotaku. He then shared a link to an Ad Age article announcing Pokémon Go’s sponsored locations and detailing its “cost per visit” business model.

Big-money tech companies rarely make money in just one or two ways, and often inconspicuously employ money-making strategies that may be less palatable to privacy-minded consumers. Mobile app companies are notorious for this. One 2017 Oxford study, for example, analyzed 1 million smartphone apps and determined that the median Google Play Store app can share users’ behavioral data with 10 third parties, while one in five can share it with over 20. “Freemium” mobile apps can earn big revenue from sharing data with advertisers—and it’s all completely opaque to users, as a Buzzfeed News report explained in 2018.

Advertising market research company Emarketer projected that advertisers will spend $29 billion on location-targeted advertising, also referred to as “geoconquesting,” this year. Marketers target and tailor ads for app users in a specific location in real-time, segment a potential audience for an ad by location, learn about consumers based on where they were before they bought something, and connect online ads to offline purchases using location data—another manifestation of “ubiquitous computing.” One of the biggest location-targeted ad companies, GroundTruth, taps data from 120 million unique monthly users to drive people to businesses like Taco Bell, where it recently took credit for 170,000 visits after a location-targeted ad campaign.

Geoconquesting is now fully integrated into digital mapping, and has been for years. Google Maps offers its location-targeted advertising service, which capitalizes on Google’s knowledge of where you typically go in the world and places you seem most interested in based on searches. Foursquare, once a social app for telling friends what restaurants or museums you’ve been to, is now leveraging its 13 billion check-ins into its new business model: “a location data and technology platform” that pairs “location tech with other data points, like transaction history.” 150,000 business partners, including Apple and Uber, receive audience profiles and granular location data, which they can tailor into, for example, a curated, personalized itinerary for a hotel guest that also drives foot traffic.

Niantic said it is not in the business of selling user location data. But it will send its users to you. Wizards Unite recently partnered with Simon Malls, which owns over 200 shopping centers, to add “multiple sponsored Inns and Fortresses” at each location, “giving players more XP and more spell energy than at any other non-sponsored location in the U.S.” As online shopping overtakes the mall experience, new technology is necessary to attract the sort of foot traffic that could keep malls off life support.

What might help is technology offering insights into how people behave at malls, too, so retailers can tweak their strategies for luring in foot traffic and pushing product. If the goal is to unite the physical with the digital, insights gleaned from how long users loiter outside a Coach store and how long they might look at a Coach Instagram ad could be massively useful to these waning mall brands. Uniting these worlds for a field trip around Tokyo is one thing; uniting them to consolidate digital and physical ad profiles is another.

“This is a hot topic in mall operation—tracking the motion of people within a mall, what stores they’re going to, how long they’re going,” said Ron Merriman, a theme park business strategist based in Shanghai (who, he noted after we contacted him for this story, happened to go to business school with Hanke). Merriman says that tracking users in malls, aquariums, and theme parks to optimize merchandising, user experiences, and ad targeting is becoming the norm where he lives in Asia. Retailers polled by Emarketer in late 2018 planned on investing more in proximity and location-based marketing than other emerging, hot-topic technologies like AI.

“Until quite recently, less than the last 10 years, for our business, your connection to your consumer started when they showed up at the front base and ended when they clicked the turnstile on the way out after the fireworks show.” he said. Today, his industry tracks users’ phones, faces, even clothing for the purposes of promoting experiences and products.

This is not a business that Niantic says it is involved in, but a patent that the company filed in January suggests a possible future for the augmented-shopping experience. It describes something called the “commercial game feature module,” a technology that lets advertising partners tweak a game’s features in realtime to better incentivize nearby users to visit. An app using the tech could get requests from advertisers to do things like “locating virtual elements at specified locations in the virtual world,” or “providing virtual items and/or enhanced powers to specific players.” It also describes the technology’s ability to “continuously monitor the positions of players” and based on this information “identify players with a predefined radius of commercial activity.”

“Patentable ideas may arise as companies do various types of research and exploration, but they often reflect ideas or technical implementations that do not correspond to product plans, and it can be inaccurate to extrapolate meaning from them,” a Niantic representative said via email in response to Kotaku’s questions about its patents.

Advertisement

“There is often a big gap between what’s in a [patent] application and what the company is actually doing,” said David Stein, an attorney specializing in technology patents. Stein was a source in a recent Slate piece about how technology companies will often patent ideas they have little intention of actually using.

That doesn’t mean, he said, that the patents are meaningless. “If you take a step back, you can figure out generally what they are researching, and generally what sorts of things they are interested in doing,” he said. “Companies generally don’t throw away money on things they have no intention to use.”

With those caveats in mind, let’s consider some of Niantic’s other recent research. A patent issued in August of this year, which Niantic acquired when it bought a company called Tick Tock AI in 2018, describes a technology meant to incentivize the collection of data through gameplay. An example: “...a computing device is programmed to ask questions to users to cause or incentivize data to be collected for a map of an entire environment (e.g., home), and provide feedback in the form of points for gameplay on the interface of the computing device. Gamification provides a motivation to participate, through in-game rewards and other feedback, such as encouragement (e.g., indications of ‘great job!’).”

When asked about this patent, Kei Kawai said that he was not familiar with specific patents filed but that Niantic’s engineering team researches a lot of ways to use what he called “computer vision” to help “users’ phones or devices understand their situations or where they are better, and reiterated that Niantic’s plans are aimed at “what’s fun and makes sense and fits our goal of getting people outside.”

While Kawai stresses that Niantic’s goal is always to get people outside, it’s worth noting that these patents specifically refer to mapping floor plans, home offices, kitchens and even the items that you own.

Another patent held by Niantic, also acquired from Tick Tock AI, describes software that can use computer vision to “identify shoes, determine specific model, determine that shoes belong to John Smith, and disambiguating between other shoes owned and worn by John Smith.” According to this patent, a host of intimate contextual information can then be associated with the item by using “...audio (e.g., noises of cooking), images of the food prior to cooking, activity recognition data of a person cooking, a time at which cooking was initiated and a time at which cooking was completed.”

“We do not infer or attempt to infer the floor plan of a player’s house or other interior space in any of Niantic’s products,” said a Niantic representative when asked about this patent.

Whether Niantic does it or not, this is one likely future of mapping for augmented reality. In a series of Medium posts, Ryan Hickman, the co-founder of TickTock AI, argued that complex systems like self-driving cars or truly seamless augmented reality experiences require a level of situational awareness and spatial reasoning that AI is currently missing. “These virtual characters that are supposed to be the future of our digital lives still have zero understanding of the world,” he wrote. “They rely on imperfect mobile sensors that struggle with ‘optically uncooperative surfaces.’” To Hickman, this is a mapping problem, and one that tech companies are eager to solve with your camera data.

How does this work? If a company wants to create an AR product where you can yell commands at your virtual pet that are picked up by some ubiquitous computing hardware—“Maggie, go lie down on my bed!”—their software needs to know what a “bed” is. It needs to know where in your house the bed is located, and which bed is yours. The software needs to learn, based on images from hundreds of thousands of other people’s unmade beds, how to associate the label “bed” with the collection of pixels that comprise your bed, as seen through your camera.

In this computational future, all of this detail comes collectively from us, and relies on turning over a great deal of information to companies with business interests that are often at odds with our own.

As one former Niantic employee put it: “It is almost irrelevant to talk about the company’s original intent, because I think that to create such an organization is to invite rot over time. It’s more of a question of, how can we structure companies that have fiscal responsibility to public welfare?”

In a future that continues to blur the barriers separating the physical world and the digital one—one mediated heavily by companies with commercial incentives—advertisers may glean new and secret wormholes into users’ lives and pocketbooks. Yes, it’s possible that companies with unfathomable financial incentives to capitalize on their data goldmines will remain responsible with users’ privacy, proving that, all along, the point of all this digital mapping momentum was pocket monsters and magic wands and getting gamers outside. But what happens over the years as these companies’ executives, shareholders, goals, and priorities change and grow, and the data set they hold on to just gets more and more detailed?

Today, Niantic can believably cast itself as a company aimed at getting gamers outside. But John Hanke sometimes describes it as an advertising company or a digital mapping company, too. The open question of where this market-leading technology goes from here—whether it’s smiling pocket pets more believably hopping around our bedrooms, or the Hyper-Reality of ubiquitous computing, saturated with data-probes and targeted advertising—is what may make space for the rot to set in.

What is surprising about Niantic is not what it’s doing, but what it’s capable of, and the fact that most of its users won’t ever understand just how much they’re handing over whenever they use a location-based app.

“We live in a technological surveillance capitalism state,” said the former employee. “How could it be any other way?”