To Cloudflare customers: Are you using Argo/Cloudflare Tunnel to connect to your origin server? Have you set up authenticated origin pulls with a customer-generated certificate (not the Cloudflare cert, that doesn't help)? If you answered no to both questions, then all of your Cloudflare security measures can be bypassed.

This includes your WAF, page rules, firewall, DDoS protection... it's all irrelevant if you don't set up your zone carefully, and even the Cloudflare documentation itself doesn't (currently) make this point super clear. And, for security, these best practices should be excruciatingly clear.

I'm not gonna reveal any new information or exploit, I'll just be using documented Cloudflare features to defeat Cloudflare security measures commonly considered to be enough to protect origin servers. I'm writing this post because it bothers me that people are unaware of this general issue that I'll talk about - it's not part of the security model people have in mind when talking about Cloudflare, and it should be.

Assumption: Origin Server IP Not Secret

I'm gonna make a bold assumption: Your origin server IP address is already exposed.

Hear me out. Relying on keeping your origin server IP a secret is security through obscurity. There are many ways that your origin IP could be leaked, for example, outgoing HTTP requests (do you generate link previews server-side, or access APIs?), headers of outgoing emails, non-HTTP services (is it also an FTP server? SSH?), etc.

Did you know that it only takes 5 minutes to scan the entire IPv4 space looking for web servers? IP addresses were not originally designed to be a security feature at all, let alone a password or a method of authentication, so it is fundamentally either flawed or hacky to rely on them for any security purpose.^[1]

(Flawed) Mitigation: Whitelisting Cloudflare IPs

Let's start with the simple stuff. Cloudflare documentation^[2]:

As a best practice we recommend to explicitly block all traffic not originating from Cloudflare IPs or your trusted partners, vendors, or application IP addresses.

This is the most common advice when it comes to protecting the origin server, for good reason. It prevents naive scans from revealing that your origin web server is a web server in the first place.

You definitely should do this, but keep in mind the attack below.

Account Fronting Attack

I call this attack an account fronting attack^[3]. Let's say an attacker knows that you're on Cloudflare^[4], and then manages to find your origin IP address. Furthermore, let's say she has the ability to set up her own free Cloudflare account.

To access your origin server, she can make sure WAF is turned off^[6] on her own zone on her Cloudflare account, and then she points an orange-clouded record on her zone to your origin IP. Then, through her domain name, she can access your origin as if it were her own website, and perform SQL injections to her heart's content, without your account's WAF in the way.

Now, I know what you're thinking. This can't happen to your setup, because your reverse proxy checks the Host header to make sure it's one of yours before proceeding. (Are you sure it does?^[7] What does it actually return when an invalid or nonexistent Host hits it? Do you log this?) Well...

(Flawed) Mitigation: Host header checking

A naive approach to making sure the request came from your own domain is to check the Host header, and, if it's not recognized, don't allow any access to the website. Reverse proxies like Apache and nginx support this behavior out of the box, and the account fronting attack, in its original form, is mitigated.

However, nowhere in the Cloudflare documentation is it stated that you should check the Host header. This is because, even though the Host header is used by Cloudflare for routing, it is not guaranteed by Cloudflare to be safe, and, in particular, isn't guaranteed to match the zone that's actually issuing the request.

In particular, Cloudflare allows enterprise customers to change the Host header arbitrarily using a page rule aptly named "Host Header Override" in the dropdown box.

Why is this feature enterprise-only? My headcanon is, this feature is restricted not because it is only useful for enterprise-level customers (the reason most enterprise features are restricted), but because Cloudflare is aware of this type of abuse and wants to limit it to companies it has vetted.

But, if my guess is true, then shame on them. Cloudflare should explicitly document how to mitigate account fronting issues, rather than rely on the obscurity of this feature and any non-documented hand-enforced rules regarding its use. Otherwise, you have people, especially free customers unaware of this feature, relying on implicit security guarantees that the Host header seems to, but doesn't actually, have.

Spectrum as a Poor Man's Host Header Override

If Cloudflare is scanning for abuse of the Host Header Override page rule, another Cloudflare feature an enterprising enterprise attacker could use is Cloudflare Spectrum. It is a TCP proxy through Cloudflare, only available to enterprise customers, and it can be targeted at any port.

I'm only a free customer, so I couldn't try this out, but the IP addresses that Spectrum operates through aren't documented separately anywhere I can find, which means they're probably part of the general "Cloudflare IPs" set. If this is true, it means there's nothing stopping you from pointing Spectrum at a victim's origin server's port 80, just to bypass the IP whitelist, and then sending arbitrary HTTP requests at it, including a fabricated Host header that matches what the origin expects.

Now, Cloudflare is smart. Maybe they also apply their nigh-magical network stream scanning technology to Cloudflare Spectrum too, scanning every TCP stream for things that look like HTTP requests with this kind of abuse. But, even if they do, what if the origin server accepts HTTPS connections?

Spectrum as a Poor Man's Open TCP Proxy

Cloudflare Spectrum is, essentially, an open TCP proxy that allows any Cloudflare enterprise customer to fully control a TCP connection that appears like it's coming from Cloudflare IPs. So, with that in mind, what if, as the Cloudflare documentation recommends, the SSL mode is not set to Flexible, so Cloudflare contacts the origin server with TLS?

If the origin server accepts HTTPS connections, abuse scanning becomes impossible because Cloudflare can't terminate the TLS connection, without violating the promise it makes when it labels itself as a TCP proxy. There are definitely legitimate reasons to have a TLS connection through Cloudflare Spectrum, and the Host header is safely inside the TLS wrapper, unable to be read passively^[8]. It's understandable for Cloudflare to terminate, and potentially MITM, highly-suspicious connections, but a TLS wrapper does not a highly suspicious connection make, and if Cloudflare MITM'd all TLS connections going through Spectrum, enough things would break (think SSL VPN, or certificate pinning) that a typical Spectrum customer would notice and something would've been said in public.

Cloudflare could disallow Spectrum connections to origin servers which, on another account, are origins for proxied web DNS entries. But if this was the case, then once the attacker has your Spectrum origin server IP, she could set that as one of her web origin servers in order to denial-of-service the legitimate Spectrum TCP service! Constant vigilance!

(Mostly Flawed) Mitigation: Authenticated Origin Pulls

Authenticated origin pull is a feature where Cloudflare uses a TLS client certificate to connect to your origin server. Here is the entire preamble of its Cloudflare documentation (emphasis in original):

Authenticated origin pulls ensure requests to your origin server come from the Cloudflare network.

This authentication becomes particularly important with the Cloudflare Web Application Firewall (WAF). Together with the WAF, you can make sure that all traffic is evaluated before receiving a response from your origin server.

This recommends using authenticated origin pulls, great! But it's a bit too enthusiastic. Authenticated origin pulls by themselves do not certify that the request has gone through a WAF. The language on this documentation is confusing, and I believe the author hasn't considered the account fronting case, because coming from the Cloudflare network is not the same as coming from a zone owned by you and the difference is pretty significant when the question is whether the request has gone through a WAF.

More specifically, the documentation gives you three options for setting up authenticated origin pulls:

• Zone-Level Authenticated Origin Pull using Cloudflare certificates
• Zone-Level Authenticated Origin Pull using customer certificates
• Per-Hostname Authenticated Origin Pull using customer certificates

These three options are presented in parallel, in a way that makes them seem roughly equivalent, but in reality, the first one is useless, and you really need to generate your own certificates. Remember, attackers can be Cloudflare customers too, and our attacker way back in the Account Fronting Attack section can easily enable authenticated origin pulls and hit your origin with a certificate signed by Cloudflare in the same way that you can.

On the other hand, if you upload your own certificate to your Cloudflare account, the attacker doesn't have access to that certificate. So, she won't be able to access your origin through their account at all, and only then can you rest assured that "all traffic is evaluated" by the WAF, as the documentation claims.

A quick, implausible, convoluted tangent

If we assume that Cloudflare scans Spectrum for Host header abuse^[9] and the target uses authenticated origin pulls with Cloudflare-level (insecure) certificates, then we can still falsify the Host header. If it doesn't immediately come to you why, remember, Cloudflare Spectrum makes Cloudflare an open TCP proxy.

Spoiler warning: The attacker sets up an origin server of her own, and, in her own zone, sets up an authenticated origin pull enabled orange-clouded A record for her own server, with a Host Header Override page rule that sets it to the attacker domain. Seems fine, right? The trick is, the attacker's origin server doesn't actually terminate TLS and instead takes the TCP stream directly back to Cloudflare, this time as a client connection into a different endpoint they own.

This endpoint may even be on a separate Cloudflare enterprise account, for further untraceability. The attacker has Cloudflare Spectrum set up on this endpoint, to forward the bytes, TLS still unterminated, directly to the victim's origin server's port 443. What does the victim's origin server see? A valid TCP request, with a valid Cloudflare-unified-certificate signed client TLS handshake, and a correct Host header inside. Completely valid request.

Alternative: Argo/Cloudflare Tunnel

Cloudflare Tunnel (previously called Argo Tunnel) is an alternative way for Cloudflare to communicate with the origin server. With it, the origin server doesn't even need any ports open^[10], to anyone, not just Cloudflare. It only needs to make outgoing TCP connections to the Internet.

When you set up Cloudflare Tunnel, after installing cloudflared on the origin server, you create a tunnel with a tunnel ID similar to 6a4d12b4-b9bb-479b-9194-5715489375ca.cfargotunnel.com, tied to your Cloudflare account. This address doesn't resolve on the Internet, but if you CNAME to it from the account that owns it, incoming client connections get tunnelled to a loopback connection from cloudflared to a localhost-bound port on the origin server. Even if this tunnel ID is exposed, if another Cloudflare account^[11] tries to CNAME it, it'll fail with "Error 1014: CNAME Cross-User Banned".

This clearly completely mitigates all account fronting attacks, because other accounts will not have access to the tunnel. Moreover, with a tunnel set up, the origin server could be behind three layers of NAT and have no ports open (and be a raspberry pi on the shelf in a cruise ship). The attacker has no choice but to go through Cloudflare, with all the security settings you set.

Let's Get Theoretical

Conceptually, if we zoom out a lot, the overall problem is that Cloudflare doesn't actually know who the origin server belongs to. Think about what Cloudflare knows. Both the attacker and you have Cloudflare accounts, and both of you are able to enter the origin server IP address into the Cloudflare web interface. How can you tell which one of you is real, and which is fake?

This brings up an interesting tangent: It only takes 5 minutes to scan the entire IPv4 space for origin servers which do not restrict themselves to Cloudflare IPs. But, instead of flooding IPv4, what if you did the scanning through Cloudflare (much slower, of course)? We can script setting an orange-clouded DNS record or a Spectrum endpoint to every single IPv4 address, and then discover all of the origin servers, using Cloudflare only to forward our scanning so it comes from a Cloudflare IP. Then, even origin servers which check the Host header will be detected, because they can't reject the TCP connection, since it doesn't know the connection is illegitimate until it sees the Host header (or SNI). This would be a neat research idea. Disclaimer: This will almost certainly set off Cloudflare monitoring alarm bells. I'm guessing, but it seems like something they would've already thought of. Don't try this at home.

On the other hand, your origin server also doesn't know which Cloudflare account is legitimate. When a request comes in from Cloudflare, it doesn't actually know whether it came through your account, and therefore your WAF, or not. It could check the Host header, but that could be spoofed; it could check for a Cloudflare certificate and a Cloudflare IP, but those only certify it came from Cloudflare, not from a particular account... The origin server needs to have a way of determining whether the account that directed the request to go to it is yours. With that in mind, the unobvious concept that a unified Cloudflare AOP cert is much less secure than a customer-provided cert becomes obvious; the former does nothing to identify the request's account to the origin server, so it actually doesn't solve the problem.

Therefore, theoretically, mitigations for account fronting attacks must logically fall into one of these two classes. Either make it so that Cloudflare can identify the correct owner of the origin server (ex. a tunnel), or so that your origin server can identify the correct Cloudflare account (ex. a customer-provided AOP certificate).

Let's Get Practical

Which brings us back to the first question in this blog post. Are you using Argo/Cloudflare Tunnel to connect to your origin server? Have you set up authenticated origin pulls with a customer-generated certificate?

Either way, you have to either let Cloudflare know which origin server is the correct one for a particular zone, or by letting your origin server know which Cloudflare account is correct. Otherwise, there's nothing that distinguishes your setup from that of an account fronting adversary.

You could let Cloudflare know which origin server is the correct one, by attaching a Cloudflare Tunnel to the origin. I know of no other way to make Cloudflare itself perform this check, because aside from Cloudflare Tunnel, the only way Cloudflare contacts origin servers is through the public internet.

You could let your origin server know which Cloudflare account is correct. The only way the origin server can tell which Cloudflare account connected to it (remember, Spectrum allows anyone to connect arbitrary TCP data to you through Cloudflare) is by providing Cloudflare with a secret associated with the account. The standard^[12] way to do this is authenticated origin pulls, with a certificate that you generated so that your web server knows it's your account whose zone signed the request that's coming in.

Conclusion

I'm almost certain I'm not the first one to think of these ideas. But, in an informal, anecdotal survey where I asked a few close friends about their Cloudflare setups, none of them used Cloudflare Tunnel or authenticated origin pulls.

Heck, I only started using Cloudflare Tunnel for my personal, small-scale projects after deciding to write this up thing as a blog post, and Cloudflare Tunnel was only made free four months ago so before that update, the only way to mitigate this for free customers was authenticated origin pulls with a customer-generated certificate. In my years of amateur web service hosting, I've seen exactly nobody tout the precept that "the only way to secure your origin is by enabling AOP with a customer certificate".

At least this practice, if not this theory, should be in all of those elementary point-and-click tutorials on how to set up Wordpress that include putting Cloudflare in front. They should say how important authenticated origin pulls with customer-generated certificates are. The Cloudflare documentation on authenticated origin pulls should say it, or at least, have a warning that indicates that it is much more secure to upload your own certificate rather than relying on Cloudflare's unified certificate, which doesn't guarantee anything^[13] more than restricting incoming IP addresses.

Overall, I believe the importance of these kinds of account fronting attacks is misunderstood, by most people in the field of web hosting. Therefore, with this blog post, I'm doing my part^[14] to help spread the word.

---

1. Also, have a look at CrimeFlare. ↩︎

2. The "Security Recommendations at Origins" section does say that trusting IP addresses is not enough, but that was added sometime in the last month. Since I first talked about this idea with a friend, who discussed it with an engineer at Cloudflare, I'd like to think it was my private complaint that got that added. ↩︎

3. I call an account fronting attack internally, just to myself, because I haven't heard a good standard name for it. I'm not pretentious enough to assume that I'm the first to think of this idea. ↩︎

4. There's numerous ways the attacker could confirm you're on Cloudflare. She could dig ns your.website.com, or look at the SSL cert, or... don't get complacent if you have fancy enterprise custom NS and certs... there's also your.website.com/cdn-cgi/trace she could try to visit, or, failing everything else, she could set up, on her own account, an orange-clouded CNAME root record (so it's CNAME flattened), point it at your orange-clouded domain, and see if she gets "Error 1014: CNAME Cross-User Banned" when she visits it^[5]. In any case, Cloudflare is the most popular service of its kind. Having the fact that your website is behind Cloudflare be a secret isn't just security through obscurity, it's not even obscure. ↩︎

5. And if you, just to obfuscate the fact that you use Cloudflare, you use your enterprise powers to turn on cross-account CNAMEing for all other accounts, well that's going to beget lots of other problems that I'm not going to add a third layer of footnotes to explain. ↩︎

6. I use WAF as an example here to stand-in for all the security features Cloudflare may provide, because many people use Cloudflare for its WAF. However, I have strong opinions about WAFs, which are too large for this footnote to contain. In short words: They're a bandaid at best, and a crutch at worst. They exist for the same reason that mysql_real_escape_string used to exist, and should be deprecated in the same way that was. ↩︎

7. Did you know that nginx automatically and silently uses the first server block if none of them are default_server and none of the server_name directives match? Apache does something similar and silently uses the first vhost if none of the VirtualHost directives match. If you didn't know this, you should go check right now to make sure your configuration doesn't route unknown Hosts to anything sensitive. ↩︎

8. Sure, Cloudflare could look at the SNI and raise a red flag if it is not a host that the customer controls, but ESNI exists, and at this many levels of "what if", any more pondering without testing is not worth the brainpower. If any of my zero readers are enterprise customers who would like to help test this stuff out (keep in mind, the good result is that Cloudflare terminates your account for abuse), let me know. ↩︎

9. And they don't scan the page rule, which is why this scenario is implausible. But the solution is fun! So I hope this isn't completely useless of an idea to have devoted a section to here. ↩︎

10. Thus, Cloudflare Tunnel is able to completely replace Dynamic DNS services! I use Cloudflare Tunnel for accessing my personal home network, with web-interfaced security cameras and whatnot, because I would much rather rely on Cloudflare over some random free service like remote.it or Teamviewer, or set up a VPN server on a residential internet connection. ↩︎

11. Account, not zone! When setting up Cloudflare Tunnel, during the cloudflared tunnel login flow, it misleadingly asks you to pick a zone, but actually, the tunnel only associates itself with the account that owns the zone you picked, not the zone itself. Any other zone owned by the same account can CNAME to it, and any other zone not in the same account cannot. ↩︎

12. I guess you could also add a transform rule that adds a secret header that only your origin can then use, and then only accept requests with that header, but why? Also, you'd want to make it a challenge-response thing so that you don't have to have a static secret and worry about manual rotation. I suppose you could leverage Cloudflare workers here, to implement that. But, hey, this is essentially reinventing TLS client authentication, and re-doing crypto stuff yourself is an anti-pattern, so the next thought is that you could have your Cloudflare worker pull in a TLS library and... oh wait hol up. ↩︎

13. Doesn't guarantee anything in the scope of this attack. Certificates are, at the very least, better than IPs because they cannot be spoofed without breaking the encryption. IP addresses, on the other hand, can potentially be spoofed, especially if you don't trust your router, or your ISP, or your government, etc. ↩︎

14. Since this is only the third post on my blog and I have zero readers, the impact of this noble statement kinda falls flat. But eh, everyone's gotta start somewhere. If you're reading this, I love you. ↩︎