Route Leak Causes Major Google Outage

Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.

Google also announced the problem through their status page:

We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.

In order to understand this issue, MainOne Inc (AS37282) is peering at IXPN (Internet Exchange Point of Nigeria) in Lagos where Google (AS151169) and China Telecom (AS4809) are also members.

Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.

Looking at the RIPE stats it is evident that the first announcement via MainOne Inc (AS37282) was recorded at 21:12 UTC and the issue lasted for more than an hour:

As per the tweet from BGPMon, the issues lasted for 74 minutes:

Looking at the circumstances around this incident, it’s likely this was an inadvertent leak from MainOne caused by a configuration mistake. A Google representative is quoted in ArsTechnical as saying “officials suspect the leak was accidental and not a malicious hijack”, and also added that the affected traffic was encrypted which limited the harm that could result from malicious hijackings.

Later in the same day, the MainOne Twitter account posted on the BGPMon analysis thread, accepting the mistake and assuring the world that corrective measures are now in place:

So this was a configuration mistake that was quickly rectified and didn’t cause any reported financial damage (even though service outages do cause financial and reputational damage to the service provider and its users), but it does demonstrate the problems that can be caused by accidental mistakes, and especially how an actor with bad intent could do a great deal of damage  as with the Amazon Route 53 hijack. It therefore illustrates why greater efforts need to be made towards improving the security and resilience of the Internet.

This BGP leak could have been easily avoided if proper prefix filtering had been undertaken by MainOne (AS37282) or China Telecom (AS4809). It is very difficult for the networks in the middle to block such leaks, because the prefixes are still legitimately originating from the correct AS number (in this scenario AS15169 – Google).

As mentioned in many previous blogs, Mutually Agreed Norms for Routing Security (MANRS) can be part of the solution here. It calls for four simple but concrete actions that ALL network operators should implement to reduce the most common routing threats, including filtering which prevents the propagation of incorrect routing information (the other three are anti-spoofing, address validation, and global coordination).

Network operators have a responsibility to ensure a globally robust and secure routing infrastructure, and your network’s safety depends on a routing infrastructure that weeds out accidental misconfigurations and bad actors. The more network operators who work together, the fewer incidents there will be, and the less damage they can do. It’s time to implement the MANRS actions now!

The post Route Leak Causes Major Google Outage appeared first on Internet Society.

Google is closing its Schaft robotics unit after failing to find a buyer

Sad news for anyone into giant robots: Google is closing down Schaft, its secretive unit that develops bipedal robots aimed at helping out in disaster efforts and generally looking badass.

The news was first reported by Nikkei, but Google confirmed to TechCrunch that the business will be shuttered. It said it is helping staff find new roles, most of which will likely be outside of Google and its Alphabet parent.

Firstly up, many people — myself included — might have forgotten that Google owns Schaft .

The company was scheduled to be sold to SoftBank alongside Boston Dynamics — another of Google’s robotics ventures — through a deal that was announced last year. Boston Dynamics made the transition but Schaft didn’t. Softbank never shouted that omission from the rooftops, but a source with knowledge of the deal told us that certain conditions agreed for the deal were not fulfilled, hence Schaft remained with Google.

Our source explained that Google’s robotics focused shifted away from Schaft and instead to non-humanoid robots and industry-led solutions such as robotic arms. The departure of Andy Rubin, the controversial robotics evangelist who reportedly got a $90 million payout to leave amid sexual misconduct allegations, seemed to speed up its demise inside the organization.

Google shopped the Schaft business fairly widely — since 2016 and after the SoftBank deal collapsed — but to no avail, we understand. That left closing it down as the last remaining option.

Schaft was founded in 2012 by a group led by University of Tokyo professor Yuto Nakanishi.

Alphabet acquired Shaft and Boston Dynamics in 2013, the former was part of a group of seven acquisitions, in undisclosed deals.

There’s been plenty of attention on Boston Dynamics and its crazy, even scary, robots which can trek across all terrains and get up instantly when knocked over, but Schaft maintained a fairly quiet presence. Indeed, its first major prototypes weren’t revealed until some two years after its acquisition.

UK watchdog has eyes on Google-DeepMind’s health app hand-off

The shock news yesterday that Google is taking over a health app rolled out to UK hospitals over the past few years by its AI division, DeepMind, has caught the eye of the country’s data protection watchdog — which said today that it’s monitoring developments.

An ICO spokesperson told us: “An ICO investigation and an independent audit into the use of Google Deepmind’s Streams service by the Royal Free both highlighted the importance of clear and effective governance when NHS bodies use third parties to provide digital services, particularly to ensure the original purpose for processing personal data is respected.

“We expect all the measures set out in our undertaking, and in the audit, should remain in place even if the identity of the third party changes. We are continuing to monitor the situation.”

We’ve reached out to DeepMind and Google for a response.

The project is already well known to the ICO because, following a lengthy investigation, it ruled last year that the NHS Trust which partnered with DeepMind had broken UK law by passing 1.6 million+ patients’ medical records to the Google owned company during the app’s development.

The Trust agreed to make changes to how it works with DeepMind, with the ICO saying it needed to establish “a proper legal basis” for the data-sharing, as well as share more information about how it handles patients’ privacy.

It also had to submit to an external audit — which was carried out by Linklaters. Though — as we reported in June — this only looked at the current working of the Streams app.

The auditors did not address the core issue of patient data being passed without a legal basis when the app was under construction. And the ICO didn’t sound too happy about that either.

While regulatory actions kicked off in spring 2016, the sanctions came after Streams had already been rolled out to hospital wards — starting with the Royal Free NHS Trust’s own hospitals.

DeepMind also inked additional five-year Streams deals with a handful of other Trusts before the ICO’s intervention, including Imperial College Healthcare NHS Trust and Taunton & Somerset.

Those Trusts are now facing being switched to having Google as their third party app provider.

Until yesterday DeepMind had maintained it operates autonomously from Google, with founder Mustafa Suleyman writing in 2016 that: “We’ve been clear from the outset that at no stage will patient data ever be linked or associated with Google accounts, products or services.”

Two years on and, in their latest Medium blog, the DeepMind co-founders write about how excited they are that the data is going to Google.

Patients might have rather more mixed feelings, given that most people have never been consulted about any of this.

The lack of a legal basis for DeepMind obtaining patient data to develop Streams in the first place remains unresolved. And Google becoming the new data processor for Streams only raises fresh questions about information governance — and trust.

Meanwhile the ICO has not yet given a final view on Streams’ continued data processing — but it’s still watching.

Google gobbling DeepMind’s health app might be the trust shock we need

DeepMind’s health app being gobbled by parent Google is both unsurprising and deeply shocking.

First thoughts should not be allowed to gloss over what is really a gut punch.

It’s unsurprising because the AI galaxy brains at DeepMind always looked like unlikely candidates for the quotidian, margins-focused business of selling and scaling software as a service. The app in question, a clinical task management and alerts app called Streams, does not involve any AI.

The algorithm it uses was developed by the UK’s own National Health Service, a branch of which DeepMind partnered with to co-develop Streams.

In a blog post announcing the hand-off yesterday, “scaling” was the precise word the DeepMind founders chose to explain passing their baby to Google . And if you want to scale apps Google does have the well oiled machinery to do it.

At the same time Google has just hired Dr. David Feinberg, from US health service organization Geisinger, to a new leadership role which CNBC reports as being intended to tie together multiple, fragmented health initiatives and coordinate its moves into the $3TR healthcare sector.

The company’s stated mission of ‘organizing the world’s information and making it universally accessible and useful’ is now seemingly being applied to its own rather messy corporate structure — to try to capitalize on growing opportunities for selling software to clinicians.

That health tech opportunities are growing is clear.

In the UK, where Streams and DeepMind Health operates, the minister for health, Matt Hancock, a recent transplant to the portfolio from the digital brief, brought his love of apps with him — and almost immediately made technology one of his stated priorities for the NHS.

Last month he fleshed his thinking out further, publishing a future of healthcare policy document containing a vision for transforming how the NHS operates — to plug in what he called “healthtech” apps and services, to support tech-enabled “preventative, predictive and personalised care”.

Which really is a clarion call to software makers to clap fresh eyes on the sector.

In the UK the legwork that DeepMind has done on the ‘apps for clinicians’ front — finding a willing NHS Trust to partner with; getting access to patient data, with the Royal Free passing over the medical records of some 1.6 million people as Streams was being developed in the autumn of 2015; inking a bunch more Streams deals with other NHS Trusts — is now being folded right back into Google.

And this is where things get shocking.

Trust demolition

Shocking because DeepMind handing the app to Google — and therefore all the patient data that sits behind it — goes against explicit reassurances made by DeepMind’s founders that there was a firewall sitting between its health experiments and its ad tech parent, Google.

“In this work, we know that we’re held to the highest level of scrutiny,” wrote DeepMind co-founder Mustafa Suleyman in a blog post in July 2016 as controversy swirled over the scope and terms of the patient data-sharing arrangement it had inked with the Royal Free. “DeepMind operates autonomously from Google, and we’ve been clear from the outset that at no stage will patient data ever be linked or associated with Google accounts, products or services.”

As law and technology academic Julia Powles, who co-wrote a research paper on DeepMind’s health foray with the New Scientist journalist, Hal Hodson, who obtained and published the original (now defunct) patient data-sharing agreement, noted via Twitter: “This isn’t transparency, it’s trust demolition.”

Turns out DeepMind’s patient data firewall was nothing more than a verbal assurance — and two years later those words have been steamrollered by corporate reconfiguration, as Google and Alphabet elbow DeepMind’s team aside and prepare to latch onto a burgeoning new market opportunity.

Any fresh assurances that people’s sensitive medical records will never be used for ad targeting will now have to come direct from Google. And they’ll just be words too. So put that in your patient trust pipe and smoke it.

The Streams app data is also — to be clear — personal data that the individuals concerned never consented to being passed to DeepMind. Let alone to Google.

Patients weren’t asked for their consent nor even consulted by the Royal Free when it quietly inked a partnership with DeepMind three years ago. It was only months later that the initiative was even made public, although the full scope and terms only emerged thanks to investigative journalism.

Transparency was lacking from the start.

This is why, after a lengthy investigation, the UK’s data protection watchdog ruled last year that the Trust had breached UK law — saying people would not have reasonably expected their information to be used in such a way.

Nor should they. If you ended up in hospital with a broken leg you’d expect the hospital to have your data. But wouldn’t you be rather shocked to learn — shortly afterwards or indeed years and years later — that your medical records are now sitting on a Google server because Alphabet’s corporate leaders want to scale a fat healthtech profit?

In the same 2016 blog post, entitled “DeepMind Health: our commitment to the NHS”, Suleyman made a point of noting how it had asked “a group of respected public figures to act as Independent Reviewers, to examine our work and publish their findings”, further emphasizing: “We want to earn public trust for this work, and we don’t take that for granted.”

Fine words indeed. And the panel of independent reviewers that DeepMind assembled to act as an informal watchdog in patients’ and consumers’ interests did indeed contain well respected public figures, chaired by former Liberal Democrat MP Julian Huppert.

The panel was provided with a budget by DeepMind to carry out investigations of the reviewers’ choosing. It went on to produce two annual reports — flagging a number of issues of concern, including, most recently, warning that Google might be able to exert monopoly power as a result of the fact Streams is being contractually bundled with streaming and data access infrastructure.

The reviewers also worried whether DeepMind Health would be able to insulate itself from Alphabet’s influence and commercial priorities — urging DeepMind Health to “look at ways of entrenching its separation from Alphabet and DeepMind more robustly, so that it can have enduring force to the commitments it makes”.

It turns out that was a very prescient concern since Alphabet/Google has now essentially dissolved the bits of DeepMind that were sticking in its way.

Including — it seems — the entire external reviewer structure…

A DeepMind spokesperson told us that the panel’s governance structure was created for DeepMind Health “as a UK entity”, adding: “Now Streams is going to be part of a global effort this is unlikely to be the right structure in the future.”

It turns out — yet again — that tech industry DIY ‘guardrails’ and self-styled accountability are about as reliable as verbal assurances. Which is to say, not at all.

This is also both deeply unsurprisingly and horribly shocking. The shock is really that big tech keeps getting away with this.

None of the self-generated ‘trust and accountability’ structures that tech giants are now routinely popping up with entrepreneurial speed — to act as public curios and talking shops to draw questions away from what’s they’re actually doing as people’s data gets sucked up for commercial gain — can in fact be trusted.

They are a shiny distraction from due process. Or to put it more succinctly: It’s PR.

There is no accountability if rules are self-styled and therefore cannot be enforced because they can just get overwritten and goalposts moved at corporate will.

Nor can there be trust in any commercial arrangement unless it has adequately bounded — and legal — terms.

This stuff isn’t rocket science nor even medical science. So it’s quite the pantomime dance that DeepMind and Google have been merrily leading everyone on.

It’s almost as if they were trying to cause a massive distraction — by sicking up faux discussions of trust, fairness and privacy — to waste good people’s time while they got on with the lucrative business of mining everyone’s data.

DeepMind hands off role as health app provider to parent Google

DeepMind’s recent foray into providing software as a service to U.K. hospitals has reached the end of its run.

The Google -owned AI division has just announced it will be stepping back from providing a clinical alerts and task management healthcare app to focus on research — handing off the team doing the day to day delivery of the Streams to its parent, Google. 

Announcing the move in a blog post entitled “Scaling Streams with Google,” DeepMind’s co-founders write: “Our vision is for Streams to now become an AI-powered assistant for nurses and doctors everywhere — combining the best algorithms with intuitive design, all backed up by rigorous evidence. The team working within Google, alongside brilliant colleagues from across the organisation, will help make this vision a reality.”

DeepMind’s 2015 plunge into the health apps space always looked like a curious departure for an AI specialist because — despite the above quote — the Streams app does not use any AI.

Rather, it uses a National Health Service algorithm. The design of the app was also outsourced to a U.K.-based app studio.

Yet DeepMind began its foray into health with grand ambitions about applying AI to patient data, quietly inking an expansive data-sharing arrangement plus memorandum of understanding with an NHS Trust to get access to millions of patients’ full (and fully identifiable) medical records, as we reported at the time.

It also made a 2015 ethics application with the NHS’ Health Research Authority to apply AI to the patient data. Though it later said it quickly realized that clinicians’ “most urgent problems” were rather more fundamental than a pressing need to rush into experiments with AI. (And DeepMind has always maintained that the patient data it obtained under its arrangement with the Royal Free NHS Trust, with whom the Streams app was co-developed, was never used for AI.)

The Streams project ran into major controversy in May 2016 when fuller details emerged about the scope and terms of the data-sharing underpinning the app — and questions started being asked about data governance due process, legal bases for data-sharing and Google’s role and potential interest in people’s medical records.

After a year, the initial data-sharing arrangement between DeepMind and the Royal Free was scrapped and replaced with a tighter contract. (You can download a redacted version of the contract published by DeepMind here.)

Then last year the U.K.’s data protection watchdog ruled the first arrangement had breached U.K. law — with the information commissioner saying patients “would not have reasonably expected” their sensitive medical records to be used for developing an app.

Although by then the Streams app had already been deployed into Royal Free hospitals. And DeepMind had inked a few more deals with NHS Trusts to use the app.

It also emerged that DeepMind was providing Streams to Trusts essentially free of charge for the first five years. And a panel of external reviewers engaged by DeepMind with the aim of boosting trust warned in their annual review earlier this year of the risk of it “exert[ing] excessive monopoly power” as a result of a data access and streaming infrastructure bundled with the Streams app.

The whole episode opened a Pandora’s box of data governance, privacy and trust issues — which DeepMind now appears to be dumping directly onto Google, which will now be fully in the frame as the health app provider (and patient data handler) behind Streams.

“The Streams team will remain in London, under the leadership of former NHS surgeon and researcher Dr Dominic King,” write the DeepMind co-founders now. “We’re fully committed to all our NHS partners, and to delivering on our current projects and more. We’ll be working closely with them as we plan for the team’s transition, and information governance and safety remain our top priorities. Patient data remains under our partners’ strict control, and all decisions about its use will continue to lie with them.”

They add that DeepMind’s role from here on in will be focused on research, rather than software as a service, saying: “As a research organisation, DeepMind will continue to work on fundamental health research with partners in academia, the NHS and beyond. When we have promising results that could have impact at scale, we’ll work closely with the Streams and translational research teams at Google on how to implement research ideas into clinical settings.”

So provision of any health AIs that DeepMind develops in the future will be left to Google to deploy and scale. (And on the scale front Google might be feeling buoyed by the U.K.’s new minister for health being very pro-app.)

As will the task of winning patient trust — which may well prove the biggest challenge here.

The trust issue was also flagged by DeepMind’s independent reviewers last year, when they wrote in their annual report: “As far as we can ascertain, DMH [DeepMind Health] does not share its data with Google, yet the public perception that this might be the case, now or in the future, will be difficult to overcome and has the potential to delay or undermine work that could be of great potential benefit to patients.”

It’s not clear whether Google will engage a panel of independent reviewers to oversee its provision of Streams going forward, as DeepMind had. We’ve asked the company to confirm its intention vis-à-vis oversight.

Update: A spokesperson has now told us: “The Independent Reviewers Panel was a governance structure for DeepMind Health as a UK entity. Now Streams is going to part of a global effort this is unlikely to be the right structure in the future.”

Poynt raises $100M for its smart payment terminal

Elavon, a U.S. Bank-owned payment processing company, and National Australia Bank have participated in the $100 million Series C for Poynt, a developer of smart payment terminals and an open operating system that powers any payment terminal worldwide.

Palo Alto-based Poynt was launched in 2014 by Osama Bedier, the former vice president of Wallet and Payments at Google. Prior to joining Google in 2011, Bedier had been the head of platform, mobile and new ventures at PayPal.

In four years, Poynt has brought in a total of $133 million from backers such as Google Ventures, Matrix Partners, Oak HC/FT, Webb Investment Network and Nyca Partners. In the last 16 months, it has shipped some 150,000 terminals. The company says total payment volume will exceed $25 billion in the next year.

“Our vision is to transform retail by becoming that innovation platform for payment terminals everywhere,” Bedier wrote in a statement. “We give developers a technical canvas to build the experiences merchants and their customers have come to expect and ultimately, make visiting your local store the personal experience it was always meant to be.”

With the investment, Poynt plans to bring its technology to Asia, Europe and South America.

Google goes down after major BGP mishap routes traffic through China

Google goes down after major BGP mishap routes traffic through China

Google lost control of several million of its IP addresses for more than an hour on Monday in an event that intermittently made its search and other services unavailable to many users and also caused problems for Spotify and other Google cloud customers. While Google said it had no reason to believe the mishap was a malicious hijacking attempt, the leak appeared suspicious to many, in part because it misdirected traffic to China Telecom, the Chinese government-owned provider that was recently caught improperly routing traffic belonging to a raft of Western carriers though mainland China.

The leak started at 21:13 UTC when MainOne Cable Company, a small ISP in Lagos, Nigeria, suddenly updated tables in the Internet’s global routing system to improperly declare that its autonomous system 37282 was the proper path to reach 212 IP prefixes belonging to Google. Within minutes, China Telecom improperly accepted the route and announced it worldwide. The move by China Telecom, aka aka AS4809, in turn caused Russia-based Transtelecom, aka AS20485, and other large service providers to also follow the route.

The redirections, BGPmon said on Twitter came in five distinct waves over a 74-minute period. The redirected IP ranges transmitted some of Google’s most sensitive communications, including the company’s corporate WAN infrastructure and the Google VPN. This graphic from regional Internet registry RIPE NCC shows how the domino effect played out over a two-hour span. The image below shows an abbreviated version of those events.

Read 10 remaining paragraphs | Comments

YouTube VR finally lands on the Oculus Go

Today, Google’s YouTube VR app arrives on the $199 Oculus Go, bringing the largest library of VR content on the web to Facebook’s entry-level VR device.

YouTube brings plenty of content in conventional and more immersive video types. It’s undoubtedly the biggest single hub of 360 content and native formats like VR180, though offering access to the library at large is probably far more important to the Oculus platform.

One of the interesting things about Oculus’s strategy with the Go headset is that gaming turned out to be the minority use case following media consumption. If you find it hard to believe that so many people are out there binging on 360 videos it’s because they probably aren’t. Users have kind of co-opted the device’s capabilities to make it a conventional movie and TV viewing device, there are apps from Netflix and Hulu while Facebook has also built Oculus TV, a feature that’s still in its infancy but basically offers an Apple TV-like environment for watching a lot of 2D content in a social environment.

At the company’s Oculus Connect conference this past year CTO John Carmack remarked how about 70 percent of time spent by users on the Go has been watching videos with about 30 percent of user time has gone to gaming. Oculus has positioned itself as a gaming company in a lot of ways via its investments so it will be interesting to see how it grows its mobile platform to make the video aspect of its VR business more attractive.

With YouTube, the company has pretty easy access to effortlessly bringing a bunch of content onboard, this would have been a great partner for Oculus TV, but a dedicated app brings a lot to users. It wasn’t super clear whether Google was going to play hardball with the YouTube app and keep standalone access confined to its Daydream platform, as the company’s homegrown VR ambitions seem to have grown more subdued, it looks like they’ve had some time to focus on external platforms.

You can download the YouTube VR app here.