OARC 32

I attended the OARC 32 in San Francisco workshop today and in this post I’ll provide some notes on talks I found particularly interesting. For those interested the full agenda can be found here and some information on the DNS-OARC organization themselves here.

Brian Somers: Recursive Resolution from the ground up
This was a super-interesting talk from Brian Somers, Cisco, in which he laid out how they implemented DNSSEC in OpenDNS. He discussed the – various – challenges they faced on that journey, together with some design decisions. Initially they had planned to support fragmented responses which, in hindsight, he called a ‘huge debacle’, for reasons shown on this slide (regular readers might have an idea why I mention this ;-):

This one gives an idea of the inherent complexity of their undertaking (well, more correctly: of the complexity of DNSSEC…):

Furthermore he mentioned that they only saw a 10% increase of network traffic after setting the DO (‘DNSSEC OK’ => ‘willing to get signed responses’) flag, and that they didn’t notice a significant CPU penalty (but they had performed some optimizations in the course of the project, plus hardware upgrades).
Paul Vixie gave a related lightning talk (PPT slides here) on the “Avoid IP fragmentation in DNS” Internet-Draft in the afternoon, out of which one might find this slide particularly interesting:

Slides of Brian Somers talk here (PPT).
Video here (starts 17:30).
Slides of their DEF CON 27 talk on DNS poisoning attacks based on fragmentation here.
Some notes on DNS flag day 2020 plus links to the relevant research papers in this post.

Edward Lewis: The Different Ways of Minimizing ANY
RFC 8482 discusses potential approaches how DNS servers can react to queries of the ‘ANY’ type. Measurements performed by Edward show that different servers respond in many different flavors:

which in turn raises several questions/concerns:

Again, regular readers know I have a certain stance on protocol complexity – as an IPv6 person I heavily suffer from it on a daily basis ;-), see also this post and my RIPE 74 presentation on IPv6 security aspects. I hence very much liked this part of Edward’s talk

Overall a quite refreshing presentation, and I always have a sympathy for talks with real-life data.
Slides here.
Video here (starts at 52:50)

Ralf Weber & Mark Dokter (both Akamai): DNS Encryption Operational Experience and Insights
Another good talk with real-life measurements, this time on the behavior of different DoH/DoT clients. Two quick points on this one: first, having done some work in the space of X.509 certificate infrastructures in an ancient stage of life, I’m a bit worried when looking at this 😉

Secondly I found their notes on tuning the stack/kernel of the lab servers (quicker re-use of ports and increase of filehandles, as every connection needs one) quite interesting:

Slides here.
Video here (starts 34:30).

Dan Mahoney (from ISC): F-Root Updates
Yet another real-life case-study talk – lovely! ;-). Dan discussed challenges, design decisions and choices of hardware & software during their recent upgrade of the F-Root. While one might think that ‘the F-Root’ is mainly about DNS-related components, in reality many pieces are involved like network hardware and routing daemons:

Very solid talk, with quite some interesting technical insights!
Slides here.
Video here (starts 28:15).

While DNS is not my primary domain of work or expertise I really enjoyed the day incl. some good conversations during the breaks.

Wishing everybody a great Sunday, and those of you still in SFO have a good NANOG 78 meeting.

Organizational Security Implications of IPv6

In this post (with a very German, bulky title 😉) I’ll take a look at the implications of IPv6 on the security controls & processes in common enterprise environments, based on a few customer projects I performed in 2016–2019.

As I laid out in my presentation in the IPv6 WG at the RIPE79 meeting in Rotterdam there’s three main, say, ‘dimensions’ of (the advent) of IPv6 in an IT environment:

• IP addresses identify entities (e.g. interfaces of systems) within networks, for a certain duration/amount of time (with IPv6 the latter plays a much more important role than in IPv4 networks, think ‘lifetime’). Here (the existence of) an IP address usually precedes the actual communication act – “I have an IP address, therefore I am [ready to communicate]”.
• Certain systems can use the IP addresses of other systems to take specific actions, like routers performing forwarding decisions based on destination IP addresses of packets or firewalls dropping packets based on their addresses. The IP address is looked at during the act itself – “Show me the address(es) of a packet and let’s see what I’ll do with it”.
• IP addresses can be processed (e.g. read from or written to files, put into databases or be analysed based on algorithms) by applications or system processes. Some of these types of operations are highly security-related, like in the context of incident response or fraud detection. This can happen in close temporal proximity (‘real-time’) or in retrospective. For this temporal dimension the above mentioned property of validity/lifetime has to be taken into account.

From an IPv6 deployment perspective namely the third above dimension is of great importance (and usually is the one leading to multi-year IPv6 programs) as the places/functions where IP addresses are processed usually constitute dependencies to be solved before the first two former dimensions can even be touched.
Configuring IPv6 rules on a firewall (2nd dimension) doesn’t make sense if the analysis framework/infrastructure isn’t capable of IPv6 (3rd dimension), and merely enabling IPv6 on systems (1st dimension) without having the capability to filter IPv6-related traffic (2nd dimension) can actually be quite dangerous.
That’s why, as some of you dear readers have painfully experienced, usually one has to start an IPv6 project with (taking care of) the 3rd dimension…

Now, to better understand the implications of IPv6 on security controls & processes, one approach could be to take a look at the NIST Cybersecurity Framework (which I also briefly discussed in my talk in the Troopers 2019 Defense Track) and to identify all (sub-) categories potentially affected by IPv6, in one of the above dimensions.

When doing so some generalizations can be observed.

In the context of the Identify function mainly three areas are affected:

• Wherever IP addresses are used to contribute to a function (e.g. as an identifying property of an asset/a system), the underlying databases or -models might have to be adapted, in order to account for the different format of IPv6 addresses (128 bits instead of 32 bits, hexadecimal characters, different delimiter of parts of the address) and for the fact that usually multiple IPv6 addresses co-exist on individual interfaces (see also some of the points in this old post).
• Evidently vulnerability management will be heavily affected in most organizations. Sequential scanning of subnets doesn’t work anymore, systems might expose different vulnerabilities over IPv6 than over IPv4 (see the sources mentioned in this thread) and link-local connectivity might play a role, too.
• Finally there’s some sub-categories in the Identify function covering threats & risk management. Those have to consider IPv6, both wrt to IPv6-specific threats & risks (some of those were covered in my RIPE78 tutorial which is linked in the beginning of this post) and as for suppliers/the supply chain where IPv6 might also kick in.

Not too surprisingly IPv6 will also heavily affect controls & processes in the field of the Protect function:

• Many technical controls used to protect assets will have to be reviewed as for their IPv6 capabilities, and their configuration (and maybe even the underlying architecture) might have to adapted for IPv6.
• Audit & logging is one of the sub-categories in Protect (see PR.PT-1) and this is a classic example of the above 3rd dimension of IP addresses (‘being processed by a system process or an application’) which might require (potentially complex) adaptions for IPv6.

In a similar way several (sub-) categories of the Detect & Respond functions will be affected:

• Detection, analysis and correlation methods might have to be modified/enhanced for IPv6.
• IPv6-specific training of personnel performing tasks in the context of these functions will be needed.
• Processes in the incident response and digital forensics context have to take different types of addresses, their intricacies (interfaces with multiple addresses of varying lifetimes) and the co-existence of different address families into account.

As you can see, quite a few security-relevant elements within an organization will have to be reviewed and adapted in the course of an IPv6 deployment effort. Transitioning to IPv6 is much more (complex) than just enabling IPv6 addresses on systems.
Challenge accepted, I’d say, given we don’t have any alternative to IPv6…

Happy to receive any type of feedback (Twitter is usually a good channel ;-).
You all have a great week,
Enno

A slide deck with some of these thoughts can be found here:

ipv6_security_implications-2Download

Some Notes on IPv6 Bogon Filtering

At first a happy new year to you all!
For some of us it will bring IPv6-related happiness (and maybe a few – although temporary 😉 – sorrows). The IPv6 world is full of sources of pain & heated discussions, but in today’s post I will discuss a topic where life got presumably easier with IPv6 than it was with IPv4, that is the filtering of inbound traffic at network borders, based on source addresses considered to be invalid with regard to the place (of filtering) and the direction of traffic. This is often called “bogon filtering” or “filtering of martians” (more details and terminology to follow in a second).

People do this for various reasons like

• General/basic network hygiene (“obviously such packets are – at least when coming in over the Internet – somewhat invalid, so we don’t want to see them in our network”).
• Potential protection from (D)DoS attacks using spoofed source addresses from such prefixes (e.g. based on random source addresses). There’s varying numbers & opinions as for the extent of those addresses within such attacks, so one might follow this line of reasoning or not (=> this debate is not in scope of this post).
• Some regulatory frameworks might require such filtering, e.g. PCI DSS section 1.3.3 (as of v3.2.1) mandates for anti-spoofing filters.
• It’s considered best practice in certain contexts (no judgement from my side implied here).

Frankly I’m not a super big fan of spending much energy on this as I don´t really see a reasonable cost (operational effort, in particular when not done properly) to benefit (risk mitigation) ratio here – I will happily swap a full bogon filter list for some solid filtering of IPv6 extension headers if I had to choose between the two 😜 – , but it’s a recurring topic in IPv6 security circles. In fact after my RIPE79 talk on (types of) IP addresses and their security implications a reader reached out to with a question along the lines: “so should we just filter all prefixes as of slide 23 of your deck?” (which essentially contained a screenshot of the “IANA IPv6 Special-Purpose Address Registry”). Let me hence take this as an opportunity to discuss some intricacies of address-based network ingress filtering.

Usually the related rules of such ACLs can be broken down to the following categories:

• Drop “special use”/”special-purpose” prefixes which are not supposed to be seen in across-the-Internet production traffic, like the (majority of) addresses prefixes as of these lists for IPv4 and IPv6.
• Drop packets originating from address ranges not (yet) allocated by IANA or the RIRs.
• Anti-spoofing (drop packets at the network borders / ingress that pretend to originate from internal networks).
• Others, e.g. filtering packets based on (presumed) country of origin (see for example this post by Koen van Impe).

Before we analyze an actual example I’d like to come up with a bit abstract classification (regular readers know I sometimes expose a tendency to over-intellectual abstractions 😉) of address ranges typically found in such lists, together with a discussion of their operational implications.
I suggest going with the following categories:

• Special-purpose addresses that can usually be dropped without much consideration (=> default approach: include in list). The ‘documentation’ prefix 2001:db8::/32 (see RFC 3849) can serve as an example here – I can’t see any reason why one would accept packets with such a source address over the Internet.
• Special-purpose addresses that shouldn’t be filtered in most cases (=> default approach: do not include). For example I generally recommend not touching link-local addresses on transit networks between routers as this might impact (namely the session establishment of) routing protocols.
• Special-purpose addresses that need some consideration, based on an individual environment’s needs and objectives (no default, but “think about it”).
• Unallocated space. Here the main thing to keep in mind is that this might change over time (e.g. the RIPE NCC just started de-bogonising 2a10::/12). So one might only include prefixes of this type if operational procedures and maturity allow to update in case changes occur in this space (after following those in the sense of ‘noting them’…). There’s people only advocating this once prefixes of this group are received from a dynamic feed (like the well-known Team Cymru fullbogons list, disclaimer: this is just an example and I don’t have any commercial relationship with them) which in turn has its own set of intricacies and issues (trust the source of updates? what if this was subverted? contractual controls with the source? trust the transmission mechanism? etc.). See also here for the currently allocated IPv6 address space as of IANA.
• Anti-spoofing: put your own prefixes here (and make sure you don’t overlook any ;-).

Let’s have a look at the “Current IPv6 martians” list maintained (last on 2015-07-28…) by Job Snijders and to be found here (it’s for route filtering as opposed to ACLs, but for our discussion this doesn’t really matter):

It is mostly a combination of filtering special-purpose addresses (e.g. lines 1–4, 8–15 + 17, 25–27) and filtering unallocated blocks (namely lines 5–7 and 18–24, again also see this IANA link) , and it hence has some elements of several of the categories I introduced above.

Some parts deserve special attention & discussion though. First it should be noted that the first seven lines can be summarized to/replaced by ::/3. Admittedly though this doesn’t look as impressing on the stage of the security theater and/or ‘to the auditors’ 😂…

Furthermore the list contains both the TEREDO (2001::/32) and the 6to4 (2002::/16) prefixes. There is some debate if filtering those is appropriate as one might actually see legitimate traffic (albeit small amounts) from those. For example my friend Jason Fesler, who runs the very useful test-ipv6 site, told me he usually sees about 0.5% traffic coming in from the 6to4 prefix. On the other hand some people argue that 6to4 is so broken anyway that it would be better to filter it completely than expose users to the horrible user experience (then potentially attributed to IPv6) they will go thru via 6to4, e.g. see Gert Doering‘s contribution to the related thread (in May 2019) on “IPv6 ingress filtering” on the ipv6-ops (“IPv6 operators forum”) mailing list (a similar thread on ‘list of IPs to block outbound’ of October 2019 on the NANOG mailing list to be found here). So this would be a case of the above “think about it” category…

I now realize that, when it comes to IPv6 bogon filtering, things aren’t as easy as I presumed when I started writing this post ;-). In a nutshell potential filters in such lists can be broken down to some ‘always filter’ entries, some ‘stay away from this if you don’t exactly know what you’re doing’ pieces and some ‘it depends’ candidates. Whatever you do it will make sense to accurately comment individual entries and why you handle them in the way you do. Your fellow security admins (and the auditors…) will like that. In a follow-up post I might come up with a nicely commented list, for copy+paste purposes… stay tuned. And thank you for your attention while reading!
May 2020 bring lots of IPv6 joy for us.

Updated: Basic IPv6 Troubleshooting Commands / IPv6 Rosetta Stone 2019

Hi everybody,

This post is meant to provide an overview of basic IPv6 troubleshooting approaches and commands, for three main operating systems: Windows, Linux & macOS. For these OSs (and some specific tasks only) it includes an updated version of the “IPv6 Rosetta Stone”, a website I often refer IPv6 beginners to in order to identify OS-specific flavors of relevant commands, but which is a bit outdated as of today.

Some Notes on IPv4 Address Space

Most of you will be aware there are currently several initiatives to make IPv4 address space formerly & formally considered ‘special addresses’ – and hence not to be used for common Internet traffic – usable (by some definition of ‘usable’).
The motivation behind such efforts is clear: namely depletion of RFC 1918 space within many organizations, often aggravated by insufficient assignment and distribution schemes in the past. This motivation is understandable. However there is – as will turn out for good reasons – quite some debate about this, and in this post I want to give some technical overview plus provide a bit of perspective, too.

As I laid out in my “A Brief History of the IPv4 Address Space” post the concept of ‘special addresses’ has existed for a long time. The following picture shows the current “IANA IPv4 Special-Purpose Address Registry”:

Looking at this it becomes clear that there are significant differences between them as for their function or scope. There might also be quite some differences as for the actual handling of them on network devices or hosts.

Their ‘can not be routed’ attribute might have a different scope:

• over the Internet.
• in transit even within seemingly isolated/closed networks.

This property of not being forwarded across specific links (as explicitly prescribed for some of them in the individual RFCs) can be enforced by different methods:

• filtering of routing information of such networks (see e.g. RFC 1918).
• filtering of traffic with a source address from such networks. This is the classic ‘bogon filtering’ performed in many enterprise networks on their Internet uplink devices.
• filtering of traffic with a destination address of these blocks.

These respective methods can be implemented on different devices:

• on routers
• on middleboxes, for example (but not only) firewalls
• on individual hosts (by means of a local packet filtering component or within the IP stack/on the kernel level).

Furthermore this behavior can be configurable, or not. It might be enabled by default, or not. Operators or sysadmins might be aware of the behavior and/or related config options, or not.
Some of the above intricacies are nicely illustrated in Emile Aben’s RIPE Labs article on “The Curious Case of 128.0/16”.

On the other hand looking at the numbers and the size of some of these blocks it becomes clear that a good chunk of the overall IPv4 address space belongs to the ‘Special-purpose addresses’ which, in times of IPv4 address space depletion ever getting more acute, whets the appetite of people thinking about ways to make them usable. The most prominent initiative is the “IPv4 unicast extensions project”.

The main idea here is to modify the Linux kernel (and others, but the idea was primarily presented in Linux circles) in a way that allows to use various parts of the special address space as regular unicast addresses. The line of reasoning is summarized in a slide set containing this one:

As a long-time Internet enthusiast I have to admit there’s a lot of good thoughts in that presentation. I particularly like the innovation angle and the care for startups.

Still, let’s evaluate the idea from a practitioners’ perspective. As laid out above there’s a variety of handling (usually: suppression) methods and places besides the Linux kernel for these addresses. This in turn makes it probably very hard in a sufficiently large or complex network to create full transparency if/where those addresses are filtered, especially as in the past there wasn’t much need to keep track of their actual handling (as they were dropped anyway, at some point in the network).
Honestly, even without my IPv6 hat (ok, here we are, I mentioned it 😉 ) I would be highly skeptical if the unicast-extensions approach is going to work outside small & isolated environments. E.g. how would one make sure that users from remote corporate sites connecting through different types of SOHO firewalls (different models from different ages) can actually flawlessly connect to data center located services operating on ‘extended IPv4 unicast addresses’?

Finally there’s the question if these addresses are going to be used as another block of private addresses (as the recently added support of 0.0.0.0/8 in the Linux 5.3 kernel seems to imply) or if those are going to a new pool of ‘available IPv4 addresses’ to be distributed to the public via IANA/the RIRs. In that case, observing IPv4 address space related market dynamics in the last years, those addresses might well & soon end up in the hands of ‘the incumbents’, e.g. large cloud providers. Which wouldn’t exactly help innovators & startups then… but wait, ‘innovators’ are not exactly the population that should learn IPv4 anyway, right? 😉

tl;dr: I do not expect these addresses to be fit for (whatever) purpose within the next 24–48 months in the vast majority of environments.