Thoughts on Google's 'BeyondCorp and the long tail of Zero Trust' article

VPN
1

Today, I was reading Google’s ‘BeyondCorp and the long tail of Zero Trust’ article from last year about handling the most challenging use cases - https://www.usenix.org/publications/loginonline/beyondcorp-and-long-tail-zero-trust.
TL:DR, Google had a long tail of applications which did not work well with a reverse proxy and HTTP/HTTPS. Therefore, they had to develop a micro-segmented VPN solution to serve as a catch-all option for tools requiring arbitrary IP connectivity across networks. They also had to allow VPNs, in exceptions, for certain specialized use cases. Google chose an approach which they felt was the most appropriate solution for major workflows, with mitigations put in place to ensure they did not use network-based trust.

Google’s experience demonstrates to us why we cannot just use proxies to achieve a zero trust architecture. Yes, they provide a seamless user experience and no management burden to IT admins when compared to tunnel-based solutions, but they cannot cover all use cases. I believe this is why we must start the journey of zero trust with the end in mind, how we can ultimately enable all use cases, including the long tail. Even better, choose a technology which allows you to handle any use case, with the ability also to support ‘clientless’ access similar to a proxy. This did not exist when Google began their BeyondCorp journey in 2009 with Operation Aurora. Luckily for you, it now does.

We built (and open sourced) OpenZiti (https://github.com/openziti) as a general-purpose zero trust overlay network. It includes a clientless endpoint called BrowZer - https://blog.openziti.io/introducing-openziti-browzer.

2

I see the problem with BeyondCorp’s deployment, and the answer isn’t because of proxies but because of the architecture. Take a look:

Traversing a proxy can degrade performance compared to direct access to a local server, especially if that proxy is not physically close to the user.

That’s not unique to hosted proxies. That’s inherent to any hosted solution. A VPN or Proxy with their server located further to the user than the server itself will have the same degradation in performance.

This is like saying you shouldn’t eat cake because it has sugar and therefore you will get fat, but cookies are fine.The problem is using hosted solutions. Until someone invents a way to send data at faster than the speed of light, the nature of physics dictates that “the further the distance your data needs to travel, the worse the performance.”

The solution? Self-hosting your proxy or access control solution. There should not be an intermediary server between the user and the service they’re trying to access to maximize performance and latency.

This is why the first thing anyone should do when looking at a solution is ask themselves: does this solution require my data to traverse their servers? Because if yes, they’re going to degrade your performance whether they’re a VPN or proxy.

As an added benefit, self-hosting removes the MitM attack angle. Zero trust architecture naturally asks: why are you allowing your data to be passed through infrastructure you do not own?

Proxies cannot solve every use case, but they’re the best for any layer 7 HTTPS-based traffic so long as you self-host.

Finally, we recognize the value of application-layer (preferably HTTP-based) applications from a security standpoint, as they facilitate proxying and traffic inspection. Overlay networks providing VPN-like connectivity often lack the transparency and granular access controls of HTTP proxies, which diminishes the benefits of adopting BeyondCorp and may ultimately result in placing trust in the overlay network itself.

  • You want that traffic inspection through your proxy, which cannot be done through VPNs and overlays
  • You want best performance, which means self-hosting
  • You do not want the host-in-the-middle to be the one inspecting your traffic for zero trust reasons
  • Therefore, you want a self-hosted reverse proxy to achieve all of the above.
3

I’m here with pop corn, good stuff on both sides.

4

Great arguments on both sides. Self hosted reverse proxy VS a vendor controlled PEPs .

Locking down the saas app by pinning to PEP ips is a game changer.

Wouldn’t the rest of your guys discussion come down to which direction can preform continuous posture checks per request. Not just on MFA.

How would you enforce/scale least privilege adaptive access with proxy?Dlp, CASB, tls decryption.

5

Traversing a proxy can degrade performance

That is one of the issues. The author, prior to that comment, points out our apps with the following characteristics, too: (1) “While HTTPS-based, could not be easily configured to present a machine certificate to a proxy”, (2) “Required IP-layer connectivity to a variety of backends using non-HTTPS protocols”, (3) “Explicitly required IP-based client allowlists to function.” Changing the location of the proxy does not change those characteristics.

That’s inherent to any hosted solution

Not if you build P2P connections or across a smart routing fabric. The P2P connections (e.g., Wireguard) will have the same latency as if you just accessed the internet. A smart routing fabric can reduce E2E latency (e.g., how OpenZiti works or something like Cloudflare Argo Smart Routing). It’s not about breaking the laws of physics; it’s about utilising the advantages of the internet and not having a single point of failure/backhaul.

The solution? Self-hosting your proxy or access control solution

The proxy needs to be hosted somewhere. Maybe self-hosting performs better than a hosted SaaS as I can self-host it close to my applications. But if you are deploying at an enterprise scale, you have likely distributed users across the country/globe and probably distributed apps. So now you need distributed proxies, and bam, you’re building your own Cloudflare/Zscaler. Can you compete with their PoPs and Tier1 peering agreements, maybe, maybe not. The point is that self-hosting is not a silver bullet. Besides, you can self-host a VPN or zero trust overlay, so your answer also applies to those other technologies.

does this solution require my data to traverse their servers? Because if yes, they’re going to degrade your performance whether they’re a VPN or proxy.

Building on the above points, this is only true if they have few servers. If you can deploy many and use smart routing, then you may actually be able to increase performance. It’s basically the principles of MPLS, but across the internet.

As an added benefit, self-hosting removes the MitM attack angle. Zero trust architecture naturally asks: why are you allowing your data to be passed through infrastructure you do not own?

Any product/technology/vendor which needs or can sell you information on your data passing through their stack is not zero trust IMHO. The better solutions use E2EE so that packets are encrypted from source to destination regardless if they go through hops. If you want it to be more secure, use your own keys for that E2EE so that it’s literally impossible for the vendor hosting the overlay to decrypt/see your packets as they traverse their infra. Dont like that, go further and host the data plane yourself.

Therefore, you want a self-hosted reverse proxy to achieve all of the above.

I believe we debunked bullets 2 (best performance) and 3 (MITM). If you want traffic inspection, then you can combine a proxy with an overlay network. But do you need inspection? A ZTN overlay network allows you to close all inbound ports, completely stopping external network attacks. If you embed ZTN in your apps, then you have no listening ports on the underlay network, your application is literally unattackable via conventional IP-based tooling, and all conventional network threats are immediately useless. Both of those are at least orders of magnitude reduction of attack surface. If your use case still requires inspection, then enable that.

6

:slight_smile: Keep following, as I am sure we are not done yet. I just replied myself.

7

Self hosted reverse proxy VS a vendor controlled PEPs .

Both reverse proxy or PEPs (i.e., a zero trust overlay network which incl PEPs, PDPs, etc) can both be self-hosted. The one I work on, OpenZiti, is open source so can be anywhere (though my company can also provide a managed, hosted version).

Ziti supports continous posture checks. We have 5 implemented today (Posture Checks | OpenZiti) and many more are scoped out/easily possible. If you cannot pass psoture checks (as well as having strong identity, the endpoints cannot connect to the PEP (as we implement authenticate-before-connect).

Practitioners I speak to either do DLP/ISP/IDS on the endpoint before moving packets across Ziti or they route the traffic to an inspection point/bump in the wire between conenctions. Many options are available.

8

I’ll do a writeup that might take some time, but I think the fundamental gap between our approaches is due to different philosophy.

The way I understand yours approach is “how do I lock down this service or network so only I can access it?”

Whereas the zero trust approach I fall under is: “How do I ensure each service or network can verify any incoming request and reject anything but authorized and authenticated requests?”

I’m not dunking on your approach either. I think your approach has merit and it should also be followed. It’s important to view it from that angle as well and apply mechanisms that do all that you’ve said.

But at the same time, I can’t help but return to the “network” part. My idea of zero trust is application-centric: the application must always assume its network is also hostile. It must assume that anything that can reach it is hostile until proven safe — it doesn’t matter if the network is locked down correctly. If a breach happens and an authorized account gains access to the network, how does this application protect itself?

In short, how do you give your services the ability to enforce their own access control after assuming that everything that is not itself is hostile? That’s what I’m trying to solve.

I think the best deployments will apply both your approach and mine. But if I had to pick or choose one or the other (just as a thought experiment), I’d rather have an unlocked network with all my apps being able to enforce access control over a locked down network with none of my apps being able to enforce access control.

9

I would love to see that write-up. Let’s clear up our approach/philosophy.

Yes, you could set up OpenZiti (and overlay networks using zero trust networking principles) to lock down a service or network so only I can access it. OpenZiti (and any overlay network doing ZTN properly) flips the model of how we think about networking to authenticate and authorise before any connectivity is allowed. Lets explore.

verify any incoming request and reject anything but authorized and authenticated request

Traditional networks work by having inbound ports and listening for connections. This is normally combined with networking-based filtering (e.g., ACLs, IP/geo white/blacklisting) as well as packet filtering via WAF, API GW, and IPS/IDS. This inherently means some attacks cannot be stopped, e.g., zero days or Denial of Service.

OpenZiti strongly believes that the better approach is to use strong cryptography to identify, authenticate, and authorize users before they are granted network access. This allows you to close all inbound ports. Instead of playing defense against the Internet, businesses minimize the attack surface to authorized sessions only.

I explain this more in this blog I wrote which compares zero trust networking using Harry Potter analogies - https://netfoundry.io/demystifying-the-magic-of-zero-trust-with-my-daughter-and-opensource/. TL;DR, If you listen on the network, silly muggles can find you (muggles = hackers). Table stacks to doing ZTN is making yourself invisible. Even better, embed ZTN into your app so that it has an invisibility cloak and port key. What is app embedded I hear you wonder… let me explain:

My idea of zero trust is application-centric: the application must always assume its network is also hostile

Fully agree. This is why we created application-embedded zero trust networking. Usually, an application has to listen to the IP-based underlay network because that’s how it’s always been done, but this is not the case with OpenZiti. When embedding your app with a Ziti SDK, your app has no listening ports on the underlay network. It’s literally unattackable via conventional IP-based tooling. Seriously, stop and consider that for just a moment. By adopting an OpenZiti SDK into the app, all conventional network threats are immediately useless. Even if attackers breach a machine, they cannot get into the application and overlay network. If you want to read more on this, this is a great blog using Golang as an example - https://blog.openziti.io/go-is-amazing-for-zero-trust

​

I had to pick or choose one or the other

I would start with the locked-down overlay, which allows apps to enforce access control. Stopping all external network attacks is orders of magnitude reduction of risk. It can also support any use case (which Google says a proxy cannot). Your preference may be correct when compared to non-Ziti VPNs/‘zero trust networks’, but in my opinion, Ziti does zero trust correctly.

I think the best deployments will apply both your approach and mine.

Could be. This is an example of embedded zero trust into a proxy using Nginx - https://blog.openziti.io/nginx-zerotrust-api-security.