draft-ietf-dnsop-dns-capture-format-00.txt   draft-ietf-dnsop-dns-capture-format-01.txt 
dnsop J. Dickinson dnsop J. Dickinson
Internet-Draft J. Hague Internet-Draft J. Hague
Intended status: Standards Track S. Dickinson Intended status: Standards Track S. Dickinson
Expires: June 9, 2017 Sinodun IT Expires: August 25, 2017 Sinodun IT
T. Manderson T. Manderson
J. Bond J. Bond
ICANN ICANN
December 6, 2016 February 21, 2017
C-DNS: A DNS Packet Capture Format C-DNS: A DNS Packet Capture Format
draft-ietf-dnsop-dns-capture-format-00 draft-ietf-dnsop-dns-capture-format-01
Abstract Abstract
This document describes a data representation for collections of DNS This document describes a data representation for collections of DNS
messages. The format is designed for efficient storage and messages. The format is designed for efficient storage and
transmission of large packet captures of DNS traffic; it attempts to transmission of large packet captures of DNS traffic; it attempts to
minimize the size of such packet capture files but retain the full minimize the size of such packet capture files but retain the full
DNS message contents along with the most useful transport meta data. DNS message contents along with the most useful transport metadata.
It is intended to assist with the development of DNS traffic It is intended to assist with the development of DNS traffic
monitoring applications. monitoring applications.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 9, 2017. This Internet-Draft will expire on August 25, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Requirements Terminology . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 4 3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 5
4. Design Considerations . . . . . . . . . . . . . . . . . . . . 6 4. Design Considerations . . . . . . . . . . . . . . . . . . . . 6
5. C-DNS conceptual overview . . . . . . . . . . . . . . . . . . 8 5. Conceptual Overview . . . . . . . . . . . . . . . . . . . . . 8
6. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8 6. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8
7. C-DNS CBOR format . . . . . . . . . . . . . . . . . . . . . . 8 7. The C-DNS format . . . . . . . . . . . . . . . . . . . . . . 9
7.1. CDDL definition . . . . . . . . . . . . . . . . . . . . . 8 7.1. CDDL definition . . . . . . . . . . . . . . . . . . . . . 9
7.2. Format overview . . . . . . . . . . . . . . . . . . . . . 9 7.2. Format overview . . . . . . . . . . . . . . . . . . . . . 9
7.3. File header contents . . . . . . . . . . . . . . . . . . 9 7.3. File header contents . . . . . . . . . . . . . . . . . . 10
7.4. File preamble contents . . . . . . . . . . . . . . . . . 10 7.4. File preamble contents . . . . . . . . . . . . . . . . . 10
7.5. Configuration contents . . . . . . . . . . . . . . . . . 10 7.5. Configuration contents . . . . . . . . . . . . . . . . . 11
7.6. Block contents . . . . . . . . . . . . . . . . . . . . . 11 7.6. Block contents . . . . . . . . . . . . . . . . . . . . . 12
7.7. Block preamble map . . . . . . . . . . . . . . . . . . . 12 7.7. Block preamble map . . . . . . . . . . . . . . . . . . . 13
7.8. Block statistics . . . . . . . . . . . . . . . . . . . . 12 7.8. Block statistics . . . . . . . . . . . . . . . . . . . . 14
7.9. Block table map . . . . . . . . . . . . . . . . . . . . . 13 7.9. Block table map . . . . . . . . . . . . . . . . . . . . . 14
7.10. IP address table . . . . . . . . . . . . . . . . . . . . 13 7.10. IP address table . . . . . . . . . . . . . . . . . . . . 15
7.11. Class/Type table . . . . . . . . . . . . . . . . . . . . 13 7.11. Class/Type table . . . . . . . . . . . . . . . . . . . . 15
7.12. Name/RDATA table . . . . . . . . . . . . . . . . . . . . 14 7.12. Name/RDATA table . . . . . . . . . . . . . . . . . . . . 15
7.13. Query Signature table . . . . . . . . . . . . . . . . . . 14 7.13. Query Signature table . . . . . . . . . . . . . . . . . . 16
7.14. Question table . . . . . . . . . . . . . . . . . . . . . 16 7.14. Question table . . . . . . . . . . . . . . . . . . . . . 18
7.15. Resource Record (RR) table . . . . . . . . . . . . . . . 17 7.15. Resource Record (RR) table . . . . . . . . . . . . . . . 18
7.16. Question list table . . . . . . . . . . . . . . . . . . . 17 7.16. Question list table . . . . . . . . . . . . . . . . . . . 19
7.17. Resource Record list table . . . . . . . . . . . . . . . 17 7.17. Resource Record list table . . . . . . . . . . . . . . . 19
7.18. Query/Response data . . . . . . . . . . . . . . . . . . . 18 7.18. Query/Response data . . . . . . . . . . . . . . . . . . . 19
7.19. Address Event counts . . . . . . . . . . . . . . . . . . 20 7.19. Address Event counts . . . . . . . . . . . . . . . . . . 22
8. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 21 7.20. Malformed packet records . . . . . . . . . . . . . . . . 23
8.1. Name Compression . . . . . . . . . . . . . . . . . . . . 22 8. Malformed Packets . . . . . . . . . . . . . . . . . . . . . . 23
9. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 23 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 25
9.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 23 9.1. Name Compression . . . . . . . . . . . . . . . . . . . . 26
9.2. Message identifiers . . . . . . . . . . . . . . . . . . . 24 10. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 26
9.2.1. Primary ID (required) . . . . . . . . . . . . . . . . 24 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 27
9.2.2. Secondary ID (optional) . . . . . . . . . . . . . . . 24 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 27
9.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 24 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 27
9.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 24 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 28
9.5. Algorithm Limitations . . . . . . . . . . . . . . . . . . 25 10.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 28
9.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . . 25 10.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 28
9.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 25 10.5. Algorithm Limitations . . . . . . . . . . . . . . . . . 28
9.8. Post Processing . . . . . . . . . . . . . . . . . . . . . 25 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 28
10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 28
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 10.8. Post Processing . . . . . . . . . . . . . . . . . . . . 29
11. Security Considerations . . . . . . . . . . . . . . . . . . . 25 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29
12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 12. Security Considerations . . . . . . . . . . . . . . . . . . . 29
13. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 26 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29
14. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 14. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 29
14.1. Normative References . . . . . . . . . . . . . . . . . . 26 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
14.2. Informative References . . . . . . . . . . . . . . . . . 27 15.1. Normative References . . . . . . . . . . . . . . . . . . 30
14.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 28 15.2. Informative References . . . . . . . . . . . . . . . . . 31
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 28 15.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Appendix B. DNS Name compression example . . . . . . . . . . . . 33 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 33
B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 34 Appendix B. DNS Name compression example . . . . . . . . . . . . 39
B.2. Knot Authoritative compression algorithm . . . . . . . . 35 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 40
B.3. Observed differences . . . . . . . . . . . . . . . . . . 35 B.2. Knot Authoritative compression algorithm . . . . . . . . 41
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 35 B.3. Observed differences . . . . . . . . . . . . . . . . . . 41
Appendix D. Sample data on the C-DNS format . . . . . . . . . . 36 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 41
D.1. Comparison to full PCAPS . . . . . . . . . . . . . . . . 36 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 44
D.2. Block size choices . . . . . . . . . . . . . . . . . . . 36 C.2. Simple versus block coding . . . . . . . . . . . . . . . 45
D.3. Blocking vs more simple output . . . . . . . . . . . . . 36 C.3. Binary versus text formats . . . . . . . . . . . . . . . 45
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 45
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 46
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 46
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 47
1. Introduction 1. Introduction
There has long been a need to collect DNS queries and responses on There has long been a need to collect DNS queries and responses on
authoritative and recursive name servers for monitoring and analysis. authoritative and recursive name servers for monitoring and analysis.
This data is used in a number of ways including traffic monitoring, This data is used in a number of ways including traffic monitoring,
analyzing network attacks and DITL [ditl]. analyzing network attacks and "day in the life" (DITL) [ditl]
analysis.
A wide variety of tools already exist to facilitate the collection of A wide variety of tools already exist that facilitate the collection
DNS traffic data. DSC [dsc], packetq [packetq], dnscap [dnscap] and of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap
dnstap [dnstap]. However, there is no standard exchange format for [dnscap] and dnstap [dnstap]. However, there is no standard exchange
large DNS packet captures and PCAP [pcap] or PCAP-NG [pcapng] are format for large DNS packet captures. The PCAP [pcap] or PCAP-NG
typically used in practice. Such file formats can contain much [pcapng] formats are typically used in practice for packet captures,
additional information not directly pertinent to DNS traffic analysis but these file formats can contain a great deal of additional
which unnecessarily increases the capture file size. information that is not directly pertinent to DNS traffic analysis
and thus unnecessarily increases the capture file size.
There has also been work on using other text based formats to There has also been work on using text based formats to describe DNS
describe DNS packets [I-D.daley-dnsxml], [I-D.hoffman-dns-in-json] packets such as [I-D.daley-dnsxml], [I-D.hoffman-dns-in-json], but
but these are largely aimed at producing convenient representations these are largely aimed at producing convenient representations of
of single messages. single messages.
Many DNS operators may receive 100's of thousands of queries per Many DNS operators may receive hundreds of thousands of queries per
second on a single name server instance so a mechanism to minimize second on a single name server instance so a mechanism to minimize
the storage size (and therefore upload overhead) of the data the storage size (and therefore upload overhead) of the data
collected is highly desirable. collected is highly desirable.
This documents focusses on the problem of capturing and storing large The format described in this document, C-DNS (Compacted-DNS),
packet capture files of DNS traffic. with the following goals in focusses on the problem of capturing and storing large packet capture
mind: files of DNS traffic. with the following goals in mind:
o Minimize the file size for storage and transmission o Minimize the file size for storage and transmission
o Minimizing the overhead of producing the packet capture file and o Minimizing the overhead of producing the packet capture file and
the cost of any further (general purpose) compression of the file the cost of any further (general purpose) compression of the file
to minimise the size
This document contains This document contains:
o A discussion of the some common use cases in which such DNS data o A discussion of the some common use cases in which such DNS data
is collected. See Section 3. is collected Section 3
o A discussion of the major design considerations in developing an o A discussion of the major design considerations in developing an
efficient data representation for collections of DNS messages. efficient data representation for collections of DNS messages
See Section 4. Section 4
o A definition of a CBOR [RFC7049] representation of a collection of o A conceptual overview of the C-DNS format Section 5
DNS messages. This will be referred to as the C-DNS format
(Compacted-DNS). See Section 7.
o Notes on converting C-DNS back to PCAP format. See Section 8. o A description of why CBOR [RFC7049] was chosen for this format
Section 6
o The definition of the C-DNS format for the collection of DNS
messages Section 7.
o Notes on converting C-DNS data to PCAP format Section 9
o Some high level implementation considerations for applications o Some high level implementation considerations for applications
designed to produce C-DNS, e.g. a query response matching designed to produce C-DNS Section 10
algorithm. See Section 9.
2. Requirements Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
The parts of DNS messages are named as they are in [RFC1035]. In
specific, the DNS message has five sections: Header, Question,
Answer, Authority, and Additional.
Pairs of DNS messages are called a Query and a Response.
3. Data Collection Use Cases 3. Data Collection Use Cases
In an ideal world it would be optimal to collect full packet captures In an ideal world, it would be optimal to collect full packet
of all packets going in or out of a name server. However, there are captures of all packets going in or out of a name server. However,
several design choices or other limitations that are common to many there are several design choices or other limitations that are common
DNS installations and operators. to many DNS installations and operators.
o Servers are hosted in a variety of situations o DNS servers are hosted in a variety of situations
* Operator self hosted servers * Self-hosted servers
* Third party hosting (including multiple third parties) * Third party hosting (including multiple third parties)
* Third party hardware (including multiple third parties) * Third party hardware (including multiple third parties)
o Data is collected under different conditions o Data is collected under different conditions
* On well provisioned servers running in a steady state.
* On well-provisioned servers running in a steady state
* On heavily loaded servers * On heavily loaded servers
* On virtualized servers * On virtualized servers
* On servers that are under attack * On servers that are under DoS attack
* On servers that are unwitting intermediaries in attacks * On servers that are unwitting intermediaries in DoS attacks
o Traffic can be collected via a variety of mechanisms o Traffic can be collected via a variety of mechanisms
* On the same hardware as the name server itself * On the same hardware as the name server itself
* Using a network tap to listen in from another server * Using a network tap on an adjacent host to listen to DNS
traffic
* Using port mirroring to listen in from another server * Using port mirroring to listen from another host
o The capabilities of data collection (and upload) networks vary o The capabilities of data collection (and upload) networks vary
* Out-of-band networks with the same capacity as the in-band * Out-of-band networks with the same capacity as the in-band
network network
* Out-of-band networks with less capacity than the in-band * Out-of-band networks with less capacity than the in-band
network network
* Everything on the in-band network * Everything being on the in-band network
Clearly, there is a wide range of use cases from very limited data Thus, there is a wide range of use cases from very limited data
collection environments (third party hardware, servers that are under collection environments (third party hardware, servers that are under
attack, packet capture on the name server itself and no out-of-band attack, packet capture on the name server itself and no out-of-band
network) to 'limitless' environments (self hosted, well provisioned network) to "limitless" environments (self hosted, well provisioned
servers, using a network tap or port mirroring with an out-of-band servers, using a network tap or port mirroring with an out-of-band
networks with the same capacity as the in-band network). In the networks with the same capacity as the in-band network). In the
former, it is unfeasible to reliably collect full PCAPS especially if former, it is infeasible to reliably collect full packet captures,
the server is under attack. In the latter case, collection of full especially if the server is under attack. In the latter case,
PCAPs may be reasonable. collection of full packet captures may be reasonable.
As a result of these restrictions the data format discussed below was As a result of these restrictions, the C-DNS data format was designed
designed with the most limited use case in mind such that with the most limited use case in mind such that:
o Data collection will occur on the same hardware as the name server o data collection will occur on the same hardware as the name server
itself. itself
o Collected data will be stored on the same hardware as the name o collected data will be stored on the same hardware as the name
server itself, at least temporarily. server itself, at least temporarily
o Collected data being returned to some central analysis system will o collected data being returned to some central analysis system will
use the same network interface as the DNS queries and responses. use the same network interface as the DNS queries and responses
o There are multiple third party servers involved. o there can be multiple third party servers involved
and therefore minimal storage size of the capture files is a major Because of these considerations, a major factor in the design of the
factor. format is minimal storage size of the capture files.
Another consideration for any application that records DNS traffic is Another significant consideration for any application that records
that the running of the name server software and the transmission of DNS traffic is that the running of the name server software and the
DNS queries and responses is the most important job of a name server. transmission of DNS queries and responses are the most important jobs
Any data collection system co-located with the name server will need of a name server; capturing data is not. Any data collection system
to be intelligent enough to carefully manage its CPU, disk, memory co-located with the name server needs to be intelligent enough to
and network utilization. Hence this use case benefits from a format carefully manage its CPU, disk, memory and network utilization. This
that has a relatively low overhead to produce and minimizes the leads to designing a format that requires a relatively low overhead
requirement for further potentially costly compression. to produce and minimizes the requirement for further potentially
costly compression.
However, it was also essential that interoperability with less However, it was also essential that interoperability with less
restricted infrastructure was maintained. In particular it is highly restricted infrastructure was maintained. In particular, it is
desirable that the resulting collection format should facilitate the highly desirable that the collection format should facilitate the re-
re-creation of common formats (such as PCAPs) that are as close to creation of common formats (such as PCAP) that are as close to the
the original as is realistic given the restrictions above. original as is realistic given the restrictions above.
4. Design Considerations 4. Design Considerations
This section presents some of the major design considerations used in This section presents some of the major design considerations used in
the development of the C-DNS format. the development of the C-DNS format.
o The basic unit of data is a combined DNS Query and the associated 1. The basic unit of data is a combined DNS Query and the associated
Response (a 'Q/R data item'). The same structure will be used for Response (a "Q/R data item"). The same structure will be used
unmatched queries and responses. Queries without responses will for unmatched Queries and Responses. Queries without Responses
be captured omitting the Response data. Responses without queries will be captured omitting the response data. Responses without
will be captured omitting the Query data (but using the Query queries will be captured omitting the Query data (but using the
section from the Response, if present, as an identifying QNAME). Question section from the response, if present, as an identifying
QNAME).
Rationale: A Query and Response represents the basic level of a
clients interaction with the server. Also, combining the Query and
Response into one item lowers storage requirements due to commonality
in the data in most cases.
o Each Q/R data item will comprise a default Q/R data description
and a set of optional sections. Inclusion of optional sections
shall be configurable.
Rationale: Different users will have different requirements for data
to be available for analysis. Users with minimal requirements should
not have to pay the cost of recording full data, however this will
limit the ability to reconstruct PCAPS. For example omitting the
Resource Records from a Response will reduce the files size, and in
principle responses can be synthesized if there is enough context.
o Multiple Q/R items will be collected into blocks in the format.
Common data in a block will be abstracted and referenced from
individual Q/R items by indexing. The maximum number of Q/R items
in a block will be configurable.
Rationale: This blocking and indexing provides a significant * Rationale: A Query and Response represents the basic level of
reduction in the volume of file data generated. Whilst introducing a clients interaction with the server. Also, combining the
complexity it provides compression of the data that makes use of Query and Response into one item often reduces storage
knowledge of the DNS packet structure. requirements due to commonality in the data of the two
messages.
[TODO: Further discussion on commonality between DNS packets e.g. 2. Each Q/R data item will comprise a default Q/R data description
and a set of optional sections. Inclusion of optional sections
shall be configurable.
o common query signatures * Rationale: Different users will have different requirements
for data to be available for analysis. Users with minimal
requirements should not have to pay the cost of recording full
data, however this will limit the ability to reconstruct
packet captures. For example, omitting the resource records
from a Response will reduce the files size, and in principle
responses can be synthesized if there is enough context.
o for the authoritative case there are a finite set of valid 3. Multiple Q/R data items will be collected into blocks in the
responses and much commonality in NXDOMAIN responses] format. Common data in a block will be abstracted and referenced
from individual Q/R data items by indexing. The maximum number
of Q/R data items in a block will be configurable.
It is anticipated that the files produced will be subject to further * Rationale: This blocking and indexing provides a significant
compression using general purpose compression tools. Measurements reduction in the volume of file data generated. Although this
show that blocking significantly reduces the CPU required to perform introduces complexity, it provides compression of the data
such strong compression. See Appendix D. that makes use of knowledge of the DNS packet structure.
o Meta-data about other packets received should also be included in * It is anticipated that the files produced can be subject to
each block. For example counts of malformed DNS packets and non- further compression using general purpose compression tools.
DNS packets (e.g. ICMP, TCP resets) sent to the server are of Measurements show that blocking significantly reduces the CPU
interest. required to perform such strong compression. See
Appendix C.2.
It should be noted that any structured capture format that does not * [TODO: Further discussion of commonality between DNS packets
capture the DNS payload byte for byte will likely be limited to some e.g. common query signatures, a finite set of valid responses
extent in that it cannot represent 'malformed' DNS packets. Only from authoritatives]
those packets that can be transformed reasonably into the structured
format can be represented by it. So if a query is malformed this
will lead to the (well formed) DNS responses with error code FORMERR
appearing as 'unmatched'.
[TODO: Need further discussion of well-formed vs malformed packets 4. Metadata about other packets received can optionally be included
and how name servers view this definition.] in each block. For example, counts of malformed DNS packets and
non-DNS packets (e.g. ICMP, TCP resets) sent to the server may
be of interest.
[TODO: Need to develop optional representation of malformed packets 5. The wire format content of malformed DNS packets can optionally
within CBOR and what this means for packet matching. This may be recorded.
influence which fields are optional in the rest of the
representation.]
Packets such as those described above can be separately recorded in a * Rationale: Any structured capture format that does not capture
PCAP file for later analysis. the DNS payload byte for byte will be limited to some extent
in that it cannot represent "malformed" DNS packets (see
Section 8). Only those packets that can be transformed
reasonably into the structured format can be represented by
the format. However this can result in rather misleading
statistics. For example, a malformed query which cannot be
represented in the C-DNS format will lead to the (well formed)
DNS responses with error code FORMERR appearing as
'unmatched'. Therefore it can greatly aid downstream analysis
to have the wire format of the malformed DNS packets available
directly in the C-DNS file.
5. C-DNS conceptual overview 5. Conceptual Overview
The following figures show purely schematic representations of the The following figures show purely schematic representations of the
C-DNS format to convey the high-level structure of the C-DNS format. C-DNS format to convey the high-level structure of the C-DNS format.
Section 7 provides a detailed discussion of the CBOR representation Section 7 provides a detailed discussion of the CBOR representation
and individual elements. and individual elements.
Figure showing the C-DNS format (PNG) [1] Figure showing the C-DNS format (PNG) [1]
Figure showing the C-DNS format (SVG) [2] Figure showing the C-DNS format (SVG) [2]
skipping to change at page 8, line 27 skipping to change at page 8, line 43
Figure showing the Q/R data item and Block tables format (SVG) [4] Figure showing the Q/R data item and Block tables format (SVG) [4]
6. Choice of CBOR 6. Choice of CBOR
This document presents a detailed format description using CBOR, the This document presents a detailed format description using CBOR, the
Concise Binary Object Representation defined in [RFC7049]. Concise Binary Object Representation defined in [RFC7049].
The choice of CBOR was made taking a number of factors into account. The choice of CBOR was made taking a number of factors into account.
o CBOR is a binary representation, and so economical in storage o CBOR is a binary representation, and thus is economical in storage
space. space.
o Other similar representations were investigated, and whilst all o Other binary representations were investigated, and whilst all had
had attractive features, none had a significant advantage over attractive features, none had a significant advantage over CBOR.
CBOR. See Appendix C and Appendix D - for some discussion of See Appendix C for some discussion of this.
this.
o CBOR is an IETF Standard and familiar to IETF participants, and o CBOR is an IETF standard and familiar to IETF participants. It is
being based on the successful JSON text format, requires very based on the now-common ideas of lists and objects, and thus
little familiarization for those in the wider industry. requires very little familiarization for those in the wider
industry.
o CBOR can also be easily converted to JSON for debugging and other o CBOR is a simple format, and can easily be implemented from
human inspection requirements. scratch if necessary. More complex formats require library
support which may present problems on unusual platforms.
o CBOR can also be easily converted to text formats such as JSON
([RFC7159]) for debugging and other human inspection requirements.
o CBOR data schemas can be described using CDDL o CBOR data schemas can be described using CDDL
[I-D.greevenbosch-appsawg-cbor-cddl]. [I-D.greevenbosch-appsawg-cbor-cddl].
7. C-DNS CBOR format 7. The C-DNS format
7.1. CDDL definition 7.1. CDDL definition
The CDDL definition for the C-DNS format is given in Appendix A. The CDDL definition for the C-DNS format is given in Appendix A.
7.2. Format overview 7.2. Format overview
A C-DNS file begins with a file header containing a file type A C-DNS file begins with a file header containing a file type
identifier and preamble. The preamble contains information on the identifier and a preamble. The preamble contains information on the
collection settings. collection settings.
This is followed by a series of data blocks. The file header is followed by a series of data blocks.
A block consists of a block header, containing various tables of A block consists of a block header, containing various tables of
common data, and some statistics for the traffic received over the common data, and some statistics for the traffic received over the
block. The block header is then followed by a list of the Q/R pairs block. The block header is then followed by a list of the Q/R data
detailing the queries and responses received during the block. The items detailing the queries and responses received during processing
list of Q/R pairs is in turn followed by a list of per-client counts of the block input. The list of Q/R data items is in turn followed
of particular IP events that occurred during collection of the block by a list of per-client counts of particular IP events that occurred
data. during collection of the block data.
The exact nature of the DNS data will affect what block size is the The exact nature of the DNS data will affect what block size is the
best fit, however sample data for a root server indicated that block best fit, however sample data for a root server indicated that block
sizes in the low 1000's give good results. See Appendix D.2 for more sizes up to 10,000 Q/R data items give good results. See
details. Appendix C.6 for more details.
If no field type is specified then the field is unsigned.
In the following
o For all quantities that contain bit flags, bit 0 indicates the If no field type is specified, then the field is unsigned.
least significant bit.
o Items described as indexes are the index of the data item in the In all quantities that contain bit flags, bit 0 indicates the least
referenced table. Indexes are 1-based. An index value of 0 is significant bit. An item described as an index is the index of the
reserved to mean not present. Q/R data item in the referenced table. Indexes are 1-based. An
index value of 0 is reserved to mean "not present".
7.3. File header contents 7.3. File header contents
The file header contains the following: The file header contains the following:
+-------------+---------------+-------------------------------------+ +---------------+---------------+-----------------------------------+
| Field | Type | Description | | Field | Type | Description |
+-------------+---------------+-------------------------------------+ +---------------+---------------+-----------------------------------+
| File type | Text string | String identifying the file type | | file-type-id | Text string | String "C-DNS" identifying the |
| ID | | | | | | file type. |
| | | | | | | |
| File | Map of items | Collection information for the | | file-preamble | Map of items | Collection information for the |
| preamble | | whole file. | | | | whole file. |
| | | | | | | |
| File Blocks | Array of | The data blocks | | file-blocks | Array of | The data blocks. |
| | Blocks | | | | Blocks | |
+-------------+---------------+-------------------------------------+ +---------------+---------------+-----------------------------------+
7.4. File preamble contents 7.4. File preamble contents
The file preamble contains the following: The file preamble contains the following:
+---------------+----------+----------------------------------------+ +----------------------+----------+---------------------------------+
| Field | Type | Description | | Field | Type | Description |
+---------------+----------+----------------------------------------+ +----------------------+----------+---------------------------------+
| Format | Unsigned | Indicates version of format used in | | major-format-version | Unsigned | Unsigned integer '1'. The major |
| version | | file. | | | | version of format used in file. |
| | | | | | | |
| Configuration | Map of | The collection configuration. | | minor-format-version | Unsigned | Unsigned integer '0'. The minor |
| | items | Optional. | | | | version of format used in file. |
| | | | | | | |
| Generator ID | Text | String identifying the collection | | private-version | Unsigned | Version indicator available for |
| | string | program. Optional. | | | | private use by applications. |
| | | | | | | Optional. |
| Host ID | Text | String identifying the collecting | | | | |
| | string | host. Blank if converting an existing | | configuration | Map of | The collection configuration. |
| | | PCAP file. Optional. | | | items | Optional. |
+---------------+----------+----------------------------------------+ | | | |
| generator-id | Text | String identifying the |
| | string | collection program. Optional. |
| | | |
| host-id | Text | String identifying the |
| | string | collecting host. Empty if |
| | | converting an existing packet |
| | | capture file. Optional. |
+----------------------+----------+---------------------------------+
7.5. Configuration contents 7.5. Configuration contents
The collection configuration contains the following items. All are The collection configuration contains the following items. All are
optional. optional.
+-------------+----------+------------------------------------------+ +--------------------+----------+-----------------------------------+
| Field | Type | Description | | Field | Type | Description |
+-------------+----------+------------------------------------------+ +--------------------+----------+-----------------------------------+
| Query | Unsigned | To be matched with a query, a response | | query-timeout | Unsigned | To be matched with a query, a |
| timeout | | must arrive within this number of | | | | response must arrive within this |
| | | seconds. | | | | number of seconds. |
| | | | | | | |
| Skew | Unsigned | The network stack may report a response | | skew-timeout | Unsigned | The network stack may report a |
| timeout | | before the corresponding query. A | | | | response before the corresponding |
| | | response is not considered to be missing | | | | query. A response is not |
| | | a query until after this many micro- | | | | considered to be missing a query |
| | | seconds. | | | | until after this many micro- |
| | | | | | | seconds. |
| Snap length | Unsigned | Collect up to this many bytes per | | | | |
| | | packet. | | snaplen | Unsigned | Collect up to this many bytes per |
| | | | | | | packet. |
| Promiscuous | Unsigned | 1 if promiscuous mode was enabled on the | | | | |
| mode | | interface, 0 otherwise. | | promisc | Unsigned | 1 if promiscuous mode was enabled |
| | | | | | | on the interface, 0 otherwise. |
| Interfaces | Array of | Identifiers of the interfaces used for | | | | |
| | text | collection. | | interfaces | Array of | Identifiers of the interfaces |
| | strings | | | | text | used for collection. |
| | | | | | strings | |
| VLAN IDs | Array of | Identifiers of VLANs selected for | | | | |
| | unsigned | collection. | | server-addresses | Array of | Server collection IP addresses. |
| | | | | | byte | Hint for downstream analysers; |
| Filter | Text | "tcpdump" [pcap] style filter for input. | | | strings | does not affect collection. |
| | string | | | | | |
| | | | | vlan-ids | Array of | Identifiers of VLANs selected for |
| Query | Unsigned | Bit flags indicating sections in Query | | | unsigned | collection. |
| collection | | packets to be collected. | | | | |
| options | | | | filter | Text | 'tcpdump' [pcap] style filter for |
| | | Bit 0. Collect second and subsequent | | | string | input. |
| | | question sections. | | | | |
| | | Bit 1. Collect Answer sections. | | query-options | Unsigned | Bit flags indicating sections in |
| | | Bit 2. Collect Authority sections. | | | | Query packets to be collected. |
| | | Bit 3. Collection Additional sections. | | | | Bit 0. Collect second and |
| | | | | | | subsequent question sections. |
| Response | Unsigned | Bit flags indicating sections in | | | | Bit 1. Collect Answer sections. |
| collection | | Response packets to be collected. | | | | Bit 2. Collect Authority |
| options | | | | | | sections. |
| | | Bit 0. Collect second and subsequent | | | | Bit 3. Collection Additional |
| | | question sections. | | | | sections. |
| | | Bit 1. Collect Answer sections. | | | | |
| | | Bit 2. Collect Authority sections. | | response-options | Unsigned | Bit flags indicating sections in |
| | | Bit 3. Collection Additional sections. | | | | Response packets to be collected. |
| | | | | | | Bit 0. Collect second and |
| Accept RR | Array of | A set of RR type names [rrtypes]. If not | | | | subsequent question sections. |
| types | text | empty, only the nominated RR types are | | | | Bit 1. Collect Answer sections. |
| | strings | collected. | | | | Bit 2. Collect Authority |
| | | | | | | sections. |
| Ignore RR | Array of | A set of RR type names [rrtypes]. If not | | | | Bit 3. Collection Additional |
| types | text | empty, all RR types are collected except | | | | sections. |
| | strings | those listed. If present, this item must | | | | |
| | | be empty if a non-empty list of Accept | | accept-rr-types | Array of | A set of RR type names [rrtypes]. |
| | | RR types is present. | | | text | If not empty, only the nominated |
+-------------+----------+------------------------------------------+ | | strings | RR types are collected. |
| | | |
| ignore-rr-types | Array of | A set of RR type names [rrtypes]. |
| | text | If not empty, all RR types are |
| | strings | collected except those listed. If |
| | | present, this item must be empty |
| | | if a non-empty list of Accept RR |
| | | types is present. |
| | | |
| max-block-qr-items | Unsigned | Maximum number of Q/R data items |
| | | in a block. |
| | | |
| collect-malformed | Unsigned | 1 if malformed packet contents |
| | | are collected, 0 otherwise. |
+--------------------+----------+-----------------------------------+
7.6. Block contents 7.6. Block contents
Each block contains the following: Each block contains the following:
+-------------+------------------+----------------------------------+ +-----------------------+--------------+----------------------------+
| Field | Type | Description | | Field | Type | Description |
+-------------+------------------+----------------------------------+ +-----------------------+--------------+----------------------------+
| Block | Map of items | Overall information for the | | preamble | Map of items | Overall information for |
| preamble | | block. | | | | the block. |
| | | | | | | |
| Block | Map of | Statistics about the block. | | statistics | Map of | Statistics about the |
| statistics | statistics | | | | statistics | block. Optional. |
| | | | | | | |
| Block | Map of tables | The tables containing data | | tables | Map of | The tables containing data |
| tables | | referenced by individual Q/R | | | tables | referenced by individual |
| | | entries. | | | | Q/R data items. |
| | | | | | | |
| Q/Rs | Array of Q/Rs | Details of individual Q/R pairs. | | queries | Array of Q/R | Details of individual Q/R |
| | | | | | data items | data items. |
| Address | Array of Address | Per client counts of ICMP | | | | |
| Event | Event counts | messages and TCP resets. | | address-event-counts | Array of | Per client counts of ICMP |
| Counts | | | | | Address | messages and TCP resets. |
+-------------+------------------+----------------------------------+ | | Event counts | Optional. |
| | | |
| malformed-packet-data | Array of | Wire contents of malformed |
| | malformed | packets. Optional. |
| | packets | |
+-----------------------+--------------+----------------------------+
7.7. Block preamble map 7.7. Block preamble map
The block preamble map contains overall information for the block. The block preamble map contains overall information for the block.
+-----------+----------+--------------------------------------------+ +---------------+----------+----------------------------------------+
| Field | Type | Description | | Field | Type | Description |
+-----------+----------+--------------------------------------------+ +---------------+----------+----------------------------------------+
| Timestamp | Array of | A timestamp for the earliest record in the | | earliest-time | Array of | A timestamp for the earliest record in |
| | unsigned | block. The timestamp is specified as a | | | unsigned | the block. The timestamp is specified |
| | | CBOR array with two elements as in Posix | | | | as a CBOR array with two or three |
| | | struct timeval. The first element is an | | | | elements. The first two elements are |
| | | unsigned integer time_t and the second is | | | | as in Posix struct timeval. The first |
| | | an unsigned integer number of | | | | element is an unsigned integer time_t |
| | | microseconds. The latter is always a value | | | | and the second is an unsigned integer |
| | | between 0 and 999,999. | | | | number of microseconds. The third, if |
+-----------+----------+--------------------------------------------+ | | | present, is an unsigned integer number |
| | | of picoseconds. The microsecond and |
[TODO: Extend to support pico/nano. Also do this for Time offset and | | | picosecond items always have a value |
Response delay] | | | between 0 and 999,999. |
+---------------+----------+----------------------------------------+
7.8. Block statistics 7.8. Block statistics
[TODO: Add block statistics] The block statistics section contains some basic statistical
information about the block. All are optional.
+---------------------+----------+----------------------------------+
| Field | Type | Description |
+---------------------+----------+----------------------------------+
| total-packets | Unsigned | Total number of packets |
| | | processed from the input traffic |
| | | stream during collection of the |
| | | block data. |
| total-pairs | Unsigned | Total number of Q/R data items |
| | | in the block. |
| unmatched-queries | Unsigned | Number of unmatched queries in |
| | | the block. |
| unmatched-responses | Unsigned | Number of unmatched responses in |
| | | the block. |
| malformed-packets | Unsigned | Number of malformed packets |
| | | found in input for the block. |
+---------------------+----------+----------------------------------+
Implementations may choose to add additional implementation-specific
fields to the statistics.
7.9. Block table map 7.9. Block table map
The block table map contains the block tables. Each element, or The block table map contains the block tables. Each element, or
table, is an array. The following tables detail the contents of each table, is an array. The following tables detail the contents of each
block table. block table.
The Present column in the following tables indicates the The Present column in the following tables indicates the
circumstances when an optional field will be present. A Q/R pair may circumstances when an optional field will be present. A Q/R data
be: item may be:
o A Query plus a Response. o A Query plus a Response.
o A Query without a Response. o A Query without a Response.
o A Response without a Query. o A Response without a Query.
Also: Also:
o A Query and/or a Response may contain an OPT section. o A Query and/or a Response may contain an OPT section.
o A Question may or may not be present. If the Query is available, o A Question may or may not be present. If the Query is available,
the Question section of the Query is used. If no Query is the Question section of the Query is used. If no Query is
available, the Question section of the Response is used. Unless available, the Question section of the Response is used. Unless
otherwise noted, a Question refers to the first Question in the otherwise noted, a Question refers to the first Question in the
Question section. Question section.
So, for example, a field listed with a Present value of QUERY is So, for example, a field listed with a Present value of QUERY is
present whenever the Q/R pair contains a Query. If the pair contains present whenever the Q/R data item contains a Query. If the pair
a Response only, the field will not be present. contains a Response only, the field will not be present.
7.10. IP address table 7.10. IP address table
This table holds all client and server IP addresses in the block. The table "ip-address" holds all client and server IP addresses in
Each item in the table is a single IP address. the block. Each item in the table is a single IP address.
+---------+--------+------------------------------------------------+ +------------+--------+---------------------------------------------+
| Field | Type | Description | | Field | Type | Description |
+---------+--------+------------------------------------------------+ +------------+--------+---------------------------------------------+
| Address | Byte | The IP address, in network byte order. The | | ip-address | Byte | The IP address, in network byte order. The |
| | string | string is 4 bytes long for an IPv4 address, 16 | | | string | string is 4 bytes long for an IPv4 address, |
| | | bytes long for an IPv6 address. | | | | 16 bytes long for an IPv6 address. |
+---------+--------+------------------------------------------------+ +------------+--------+---------------------------------------------+
7.11. Class/Type table 7.11. Class/Type table
This table holds pairs of RR CLASS and TYPE values. Each item in the The table "classtype" holds pairs of RR CLASS and TYPE values. Each
table is a CBOR map. item in the table is a CBOR map.
+-------+--------------+ +-------+--------------+
| Field | Description | | Field | Description |
+-------+--------------+ +-------+--------------+
| Class | CLASS value. | | type | TYPE value. |
| | | | | |
| Type | TYPE value. | | class | CLASS value. |
+-------+--------------+ +-------+--------------+
[TODO: Can this be optimized? Should a class of IN be inferred if
not present?]
7.12. Name/RDATA table 7.12. Name/RDATA table
This table holds the contents of all NAME or RDATA items in the The table "name-rdata" holds the contents of all NAME or RDATA items
block. Each item in the table is the content of a single NAME or in the block. Each item in the table is the content of a single NAME
RDATA. or RDATA.
+-------+--------+--------------------------------------------------+ +------------+--------+---------------------------------------------+
| Field | Type | Description | | Field | Type | Description |
+-------+--------+--------------------------------------------------+ +------------+--------+---------------------------------------------+
| Data | Byte | The NAME or RDATA contents. NAMEs, and labels | | name-rdata | Byte | The NAME or RDATA contents. NAMEs, and |
| | string | within RDATA contents, are in uncompressed label | | | string | labels within RDATA contents, are in |
| | | format. | | | | uncompressed label format. |
+-------+--------+--------------------------------------------------+ +------------+--------+---------------------------------------------+
7.13. Query Signature table 7.13. Query Signature table
This table holds elements of the Q/R data that are often common to The table "query-sig" holds elements of the Q/R data item that are
between different individual Q/R records. Each item in the table is often common between multiple individual Q/R data items. Each item
a CBOR map. Each item in the map has an unsigned value and an in the table is a CBOR map. Each item in the map has an unsigned
unsigned key. value and an unsigned key.
The following abbreviations are used in the Present (P) column The following abbreviations are used in the Present (P) column
o Q = QUERY o Q = QUERY
o A = Always o A = Always
o QT = QUESTION o QT = QUESTION
o QO = QUERY, OPT o QO = QUERY, OPT
o QR = QUERY & RESPONSE o QR = QUERY & RESPONSE
o R = RESPONSE o R = RESPONSE
+------------+----+-------------------------------------------------+ +-----------------------+----+--------------------------------------+
| Field | P | Description | | Field | P | Description |
+------------+----+-------------------------------------------------+ +-----------------------+----+--------------------------------------+
| Server | A | The index in the IP address table of the server | | server-address-index | A | The index in the IP address table of |
| address | | IP address. | | | | the server IP address. |
| | | | | | | |
| Server | A | The server port. | | server-port | A | The server port. |
| port | | | | | | |
| | | | | transport-flags | A | Bit flags describing the transport |
| Transport | A | Bit flags describing the protocol used to | | | | used to service the query. Bit 0 is |
| flags | | service the query. Bit 0 is the least | | | | the least significant bit. |
| | | significant bit. | | | | Bit 0. Transport type. 0 = UDP, 1 = |
| | | Bit 0. Transport type. 0 = UDP, 1 = TCP. | | | | TCP. |
| | | Bit 1. IP type. 0 = IPv4, 1 = IPv6. | | | | Bit 1. IP type. 0 = IPv4, 1 = IPv6. |
| | | | | | | Bit 2. Trailing bytes in query |
| Q/R | A | Bit flags indicating information present in | | | | payload. The DNS query message in |
| signature | | this Q/R pair. Bit 0 is the least significant | | | | the UDP payload was followed by some |
| flags | | bit. | | | | additional bytes, which were |
| | | Bit 0. 1 if a Query is present. | | | | discarded. |
| | | Bit 1. 1 if a Response is present. | | | | |
| | | Bit 2. 1 if one or more Question is present. | | qr-sig-flags | A | Bit flags indicating information |
| | | Bit 3. 1 if a Query is present and it has an | | | | present in this Q/R data item. Bit 0 |
| | | OPT Resource Record. | | | | is the least significant bit. |
| | | Bit 4. 1 if a Response is present and it has an | | | | Bit 0. 1 if a Query is present. |
| | | OPT Resource Record. | | | | Bit 1. 1 if a Response is present. |
| | | Bit 5. 1 if a Response is present but has no | | | | Bit 2. 1 if one or more Question is |
| | | Question. | | | | present. |
| | | | | | | Bit 3. 1 if a Query is present and |
| Query | Q | Query OPCODE. | | | | it has an OPT Resource Record. |
| OPCODE | | | | | | Bit 4. 1 if a Response is present |
| | | | | | | and it has an OPT Resource Record. |
| Q/R DNS | A | Bit flags with values from the Query and | | | | Bit 5. 1 if a Response is present |
| flags | | Response DNS flags. Bit 0 is the least | | | | but has no Question. |
| | | significant bit. Flag values are 0 if the Query | | | | |
| | | or Response is not present. | | query-opcode | Q | Query OPCODE. Optional. |
| | | Bit 0. Query Checking Disabled (CD) flag. | | | | |
| | | Bit 1. Query Authenticated Data (AD) flag. | | qr-dns-flags | A | Bit flags with values from the Query |
| | | Bit 2. Query reserved (Z) flag. | | | | and Response DNS flags. Bit 0 is the |
| | | Bit 3. Query Recursion Available (RA) flag. | | | | least significant bit. Flag values |
| | | Bit 4. Query Recursion Desired (RD) flag. | | | | are 0 if the Query or Response is |
| | | Bit 5. Query TrunCation (TC) flag. | | | | not present. |
| | | Bit 6. Query Authoritative Answer (AA) flag. | | | | Bit 0. Query Checking Disabled (CD). |
| | | Bit 7. Query DNSSEC answer OK (D0) flag. | | | | Bit 1. Query Authenticated Data |
| | | Bit 8. Response Checking Disabled (CD) flag. | | | | (AD). |
| | | Bit 9. Response Authenticated Data (AD) flag. | | | | Bit 2. Query reserved (Z). |
| | | Bit 10. Response reserved (Z) flag. | | | | Bit 3. Query Recursion Available |
| | | Bit 11. Response Recursion Available (RA) flag. | | | | (RA). |
| | | Bit 12. Response Recursion Desired (RD) flag. | | | | Bit 4. Query Recursion Desired (RD). |
| | | Bit 13. Response TrunCation (TC) flag. | | | | Bit 5. Query TrunCation (TC). |
| | | Bit 14. Response Authoritative Answer (AA) | | | | Bit 6. Query Authoritative Answer |
| | | flag. | | | | (AA). |
| | | | | | | Bit 7. Query DNSSEC answer OK (DO). |
| Query | Q | Query RCODE. If the Query contains OPT, this | | | | Bit 8. Response Checking Disabled |
| RCODE | | value incorporates any EXTENDED_RCODE_VALUE. | | | | (CD). |
| | | | | | | Bit 9. Response Authenticated Data |
| Question | QT | The index in the Class/Type table of the CLASS | | | | (AD). |
| Class/Type | | and TYPE of the first Question. | | | | Bit 10. Response reserved (Z). |
| | | | | | | Bit 11. Response Recursion Available |
| Question | QT | The QDCOUNT in the Query, or Response if no | | | | (RA). |
| QDCOUNT | | Query present. | | | | Bit 12. Response Recursion Desired |
| | | | | | | (RD). |
| Query | Q | Query ANCOUNT. | | | | Bit 13. Response TrunCation (TC). |
| ANCOUNT | | | | | | Bit 14. Response Authoritative |
| | | | | | | Answer (AA). |
| Query | Q | Query ARCOUNT. | | | | |
| ARCOUNT | | | | query-rcode | Q | Query RCODE. If the Query contains |
| | | | | | | OPT, this value incorporates any |
| Query | Q | Query NSCOUNT. | | | | EXTENDED_RCODE_VALUE. Optional. |
| NSCOUNT | | | | | | |
| | | | | query-classtype-index | QT | The index in the Class/Type table of |
| Query EDNS | QO | The Query EDNS version. | | | | the CLASS and TYPE of the first |
| version | | | | | | Question. Optional. |
| | | | | | | |
| EDNS UDP | QO | The Query EDNS sender's UDO payload size | | query-qd-count | QT | The QDCOUNT in the Query, or |
| size | | | | | | Response if no Query present. |
| | | | | | | Optional. |
| Query OPT | QO | The index in the NAME/RDATA table of the OPT | | | | |
| RDATA | | RDATA. | | query-an-count | Q | Query ANCOUNT. Optional. |
| | | | | | | |
| Response | R | Response RCODE. If the Response contains OPT, | | query-ar-count | Q | Query ARCOUNT. Optional. |
| RCODE | | this value incorporates any | | | | |
| | | EXTENDED_RCODE_VALUE. | | query-ns-count | Q | Query NSCOUNT. Optional. |
+------------+----+-------------------------------------------------+ | | | |
| edns-version | QO | The Query EDNS version. Optional. |
| | | |
| udp-buf-size | QO | The Query EDNS sender's UDP payload |
| | | size. Optional. |
| | | |
| opt-rdata-index | QO | The index in the NAME/RDATA table of |
| | | the OPT RDATA. Optional. |
| | | |
| response-rcode | R | Response RCODE. If the Response |
| | | contains OPT, this value |
| | | incorporates any |
| | | EXTENDED_RCODE_VALUE. Optional. |
+-----------------------+----+--------------------------------------+
7.14. Question table 7.14. Question table
This table holds details on individual Questions in a Question The table "qrr" holds details on individual Questions in a Question
section. Each item in the table is a CBOR map containing a single section. Each item in the table is a CBOR map containing a single
Question. Each item in the map has an unsigned value and an unsigned Question. Each item in the map has an unsigned value and an unsigned
key. This data is optionally collected. key. This data is optionally collected.
+------------+------------------------------------------------------+ +-----------------+-------------------------------------------------+
| Field | Description | | Field | Description |
+------------+------------------------------------------------------+ +-----------------+-------------------------------------------------+
| QNAME | The index in the NAME/RDATA table of the QNAME. | | name-index | The index in the NAME/RDATA table of the QNAME. |
| | | | | |
| Class/Type | The index in the Class/Type table of the CLASS and | | classtype-index | The index in the Class/Type table of the CLASS |
| | TYPE of the Question. | | | and TYPE of the Question. |
+------------+------------------------------------------------------+ +-----------------+-------------------------------------------------+
7.15. Resource Record (RR) table 7.15. Resource Record (RR) table
This table holds details on individual Resource Records in RR The table "rr" holds details on individual Resource Records in RR
sections. Each item in the table is a CBOR map containing a single sections. Each item in the table is a CBOR map containing a single
Resource Record. This data is optionally collected. Resource Record. This data is optionally collected.
+------------+------------------------------------------------------+ +-----------------+-------------------------------------------------+
| Field | Description | | Field | Description |
+------------+------------------------------------------------------+ +-----------------+-------------------------------------------------+
| NAME | The index in the NAME/RDATA table of the NAME. | | name-index | The index in the NAME/RDATA table of the NAME. |
| | | | | |
| Class/Type | The index in the Class/Type table of the CLASS and | | classtype-index | The index in the Class/Type table of the CLASS |
| | TYPE of the RR. | | | and TYPE of the RR. |
| | | | | |
| TTL | The RR Time to Live. | | ttl | The RR Time to Live. |
| | | | | |
| RDATA | The index in the NAME/RDATA table of the RR RDATA. | | rdata-index | The index in the NAME/RDATA table of the RR |
+------------+------------------------------------------------------+ | | RDATA. |
+-----------------+-------------------------------------------------+
7.16. Question list table 7.16. Question list table
This table holds a list of second and subsequent individual Questions The table "qlist" holds a list of second and subsequent individual
in a Question section. Each item in the table is a CBOR unsigned. Questions in a Question section. Each item in the table is a CBOR
This data is optionally collected. unsigned integer. This data is optionally collected.
+----------+--------------------------------------------------------+ +----------+--------------------------------------------------------+
| Field | Description | | Field | Description |
+----------+--------------------------------------------------------+ +----------+--------------------------------------------------------+
| Question | The index in the Question table of the individual | | question | The index in the Question table of the individual |
| | Question. | | | Question. |
+----------+--------------------------------------------------------+ +----------+--------------------------------------------------------+
7.17. Resource Record list table 7.17. Resource Record list table
This table holds a list of individual Resource Records in a Answer, The table "rrlist" holds a list of individual Resource Records in a
Authority or Additional section. Each item in the table is a CBOR Answer, Authority or Additional section. Each item in the table is a
unsigned. This data is optionally collected. CBOR unsigned integer. This data is optionally collected.
+-------+-----------------------------------------------------------+ +-------+-----------------------------------------------------------+
| Field | Description | | Field | Description |
+-------+-----------------------------------------------------------+ +-------+-----------------------------------------------------------+
| RR | The index in the Resource Record table of the individual | | rr | The index in the Resource Record table of the individual |
| | Resource Record. | | | Resource Record. |
+-------+-----------------------------------------------------------+ +-------+-----------------------------------------------------------+
7.18. Query/Response data 7.18. Query/Response data
The block Q/R data is a CBOR array of individual Q/R items. Each The block Q/R data is a CBOR array of individual Q/R data items.
item in the array is a CBOR map containing details on the individual Each item in the array is a CBOR map containing details on the
Q/R pair. individual Q/R data item.
Note that there is no requirement that the elements of the Q/R array Note that there is no requirement that the elements of the Q/R array
are presented in strict chronological order. are presented in strict chronological order.
The following abbreviations are used in the Present (P) column The following abbreviations are used in the Present (P) column
o Q = QUERY o Q = QUERY
o A = Always o A = Always
skipping to change at page 18, line 38 skipping to change at page 20, line 22
o QO = QUERY, OPT o QO = QUERY, OPT
o QR = QUERY & RESPONSE o QR = QUERY & RESPONSE
o R = RESPONSE o R = RESPONSE
Each item in the map has an unsigned value (with the exception of Each item in the map has an unsigned value (with the exception of
those listed below) and an unsigned key. those listed below) and an unsigned key.
o Query extended information and Response extended information which o query-extended and response-extended which are of type Extended
are of Type Extended Information. Information.
o Response delay which is an integer (This can be negative if the o delay-useconds and delay-pseconds which are integers (The delay
network stack/capture library returns them out of order.) can be negative if the network stack/capture library returns them
out of order.)
+-------------+----+------------------------------------------------+ +-----------------------+----+--------------------------------------+
| Field | P | Description | | Field | P | Description |
+-------------+----+------------------------------------------------+ +-----------------------+----+--------------------------------------+
| Time offset | A | Q/R timestamp as an offset in microseconds | | time-useconds | A | Q/R timestamp as an offset in |
| | | from the Block pre-amble Timestamp. The | | | | microseconds from the Block preamble |
| | | timestamp is the timestamp of the Query, or | | | | Timestamp. The timestamp is the |
| | | the Response if there is no Query. | | | | timestamp of the Query, or the |
| | | | | | | Response if there is no Query. |
| Client | A | The index in the IP address table of the | | | | |
| address | | client IP address. | | time-pseconds | A | Picosecond component of the |
| | | | | | | timestamp. Optional. |
| Client port | A | The client port. | | | | |
| | | | | client-address-index | A | The index in the IP address table of |
| Transaction | A | DNS transaction identifier. | | | | the client IP address. |
| ID | | | | | | |
| | | | | client-port | A | The client port. |
| Query | A | The index of the more information on the Q/R | | | | |
| signature | | in the Query Signature table. | | transaction-id | A | DNS transaction identifier. |
| | | | | | | |
| Client | Q | The IPv4 TTL or IPv6 Hoplimit from the Query | | query-signature-index | A | The index of the Query Signature |
| hoplimit | | packet. | | | | table record for this data item. |
| | | | | | | |
| Response | QR | The time different between Query and Response, | | client-hoplimit | Q | The IPv4 TTL or IPv6 Hoplimit from |
| delay | | in microseconds. | | | | the Query packet. Optional. |
| | | | | | | |
| Question | QT | The index in the NAME/RDATA table of the QNAME | | delay-useconds | QR | The time difference between Query |
| NAME | | for the first Question. | | | | and Response, in microseconds. Only |
| | | | | | | present if there is a query and a |
| Response | R | The size of the DNS message (not the packet | | | | response. |
| size | | containing the message, just the DNS message) | | | | |
| | | that forms the Response. | | delay-pseconds | QR | Picosecond component of the time |
| | | | | | | different between Query and |
| Query | Q | Extended Query information. This item is only | | | | Response. If delay-useconds is non- |
| extended | | present if collection of extra Query | | | | zero then delay-pseconds (if |
| information | | information is configured. | | | | present) MUST be of the same sign as |
| | | | | | | delay-useconds, or be 0. Optional. |
| Response | R | Extended Response information. This item is | | | | |
| extended | | only present if collection of extra Query | | query-name-index | QT | The index in the NAME/RDATA table of |
| information | | information is configured. | | | | the QNAME for the first Question. |
+-------------+----+------------------------------------------------+ | | | Optional. |
| | | |
| query-size | R | DNS query message size (see below). |
| | | Optional. |
| | | |
| response-size | R | DNS query message size (see below). |
| | | Optional. |
| | | |
| query-extended | Q | Extended Query information. This |
| | | item is only present if collection |
| | | of extra Query information is |
| | | configured. Optional. |
| | | |
| response-extended | R | Extended Response information. This |
| | | item is only present if collection |
| | | of extra Response information is |
| | | configured. Optional. |
+-----------------------+----+--------------------------------------+
The collector always collects basic Q/R information. It may be An implementation must always collect basic Q/R information. It may
configured to collect details on Question, Answer, Authority and be configured to collect details on Question, Answer, Authority and
Additional sections of the Query, the Response or both. Note that Additional sections of the Query, the Response or both. Note that
only the second and subsequent Questions of any Question section are only the second and subsequent Questions of any Question section are
collected (the details of the first are in the basic information), collected (the details of the first are in the basic information),
and that OPT Records are not collected in the Additional section. and that OPT Records are not collected in the Additional section.
The query-size and response-size fields hold the DNS message size.
For UDP this is the size of the UDP payload that contained the DNS
message and will therefore include any trailing bytes if present.
Trailing bytes with queries are routinely observed in traffic to
authoritative servers and this value allows a calculation of how many
trailing bytes were present. For TCP it is the size of the DNS
message as specified in the two-byte message length header.
The Extended information is a CBOR map as follows. Each item in the The Extended information is a CBOR map as follows. Each item in the
map is present only of collection of the relevant details is map is present only if collection of the relevant details is
configured. Each item in the map has an unsigned value and an configured. Each item in the map has an unsigned value and an
unsigned key. unsigned key.
+------------+------------------------------------------------------+ +------------------+------------------------------------------------+
| Field | Description | | Field | Description |
+------------+------------------------------------------------------+ +------------------+------------------------------------------------+
| Question | The index in the Questions list table of the entry | | question-index | The index in the Questions list table of the |
| | listing the second and subsequent Question sections | | | entry listing the second and subsequent |
| | for the Query or Response. | | | Question sections for the Query or Response. |
| | | | | |
| Answer | The index in the RR list table of the entry listing | | answer-index | The index in the RR list table of the entry |
| | the Answer Resource Record sections for the Query or | | | listing the Answer Resource Record sections |
| | Response. | | | for the Query or Response. |
| | | | | |
| Authority | The index in the RR list table of the entry listing | | authority-index | The index in the RR list table of the entry |
| | the Authority Resource Record sections for the Query | | | listing the Authority Resource Record sections |
| | or Response. | | | for the Query or Response. |
| | | | | |
| Additional | The index in the RR list table of the entry listing | | additional-index | The index in the RR list table of the entry |
| | the Additional Resource Record sections for the | | | listing the Additional Resource Record |
| | Query or Response. | | | sections for the Query or Response. |
+------------+------------------------------------------------------+ +------------------+------------------------------------------------+
7.19. Address Event counts 7.19. Address Event counts
This table holds counts of various IP related events relating to This table holds counts of various IP related events relating to
traffic with individual client addresses. traffic with individual client addresses.
+----------+----------+---------------------------------------------+ +------------------+----------+-------------------------------------+
| Field | Type | Description | | Field | Type | Description |
+----------+----------+---------------------------------------------+ +------------------+----------+-------------------------------------+
| Event | Unsigned | The type of event. The following events | | ae-type | Unsigned | The type of event. The following |
| type | | types are currently defined: | | | | events types are currently defined: |
| | | 0. TCP reset. | | | | 0. TCP reset. |
| | | 1. ICMP time exceeded. | | | | 1. ICMP time exceeded. |
| | | 2. ICMP destination unreachable. | | | | 2. ICMP destination unreachable. |
| | | 3. ICMPv6 time exceeded. | | | | 3. ICMPv6 time exceeded. |
| | | 4. ICMPv6 destination unreachable. | | | | 4. ICMPv6 destination unreachable. |
| | | 5. ICMPv6 packet too big. | | | | 5. ICMPv6 packet too big. |
| | | | | | | |
| Event | Unsigned | A code relating to the event. Optional. | | ae-code | Unsigned | A code relating to the event. |
| code | | | | | | Optional. |
| | | | | | | |
| Address | Unsigned | The index in the IP address table of the | | ae-address-index | Unsigned | The index in the IP address table |
| index | | client address. | | | | of the client address. |
| | | | | | | |
| Count | Unsigned | The number of occurrences of this event | | ae-count | Unsigned | The number of occurrences of this |
| | | during the block collection period. | | | | event during the block collection |
+----------+----------+---------------------------------------------+ | | | period. |
+------------------+----------+-------------------------------------+
8. C-DNS to PCAP 7.20. Malformed packet records
It is possible to re-construct PCAP files from the C-DNS format. This optional table records the original wire format content of
However this is a lossy process and some of the issues with malformed packets (see Section 8).
reconstructing both the DNS payload and the full packet stream are
outlined here.
Firstly the reconstruction depends on whether or not all the optional +----------------+--------+-----------------------------------------+
| Field | Type | Description |
+----------------+--------+-----------------------------------------+
| time-useconds | A | Packet timestamp as an offset in |
| | | microseconds from the Block preamble |
| | | Timestamp. |
| | | |
| time-pseconds | A | Picosecond component of the timestamp. |
| | | Optional. |
| | | |
| packet-content | Byte | The packet content in wire format. |
| | string | |
+----------------+--------+-----------------------------------------+
8. Malformed Packets
In the context of generating a C-DNS file it is assumed that only
those packets which can be parsed to produce a well-formed DNS
message are stored in the C-DNS format. This means as a minimum:
o The packet has a well-formed 12 bytes DNS Header
o The section counts are consistent with the section contents
o All of the resource records can be parsed
In principle, packets that do not meet these criteria could be
classified into two categories:
o Partially malformed: those packets which can be decoded
sufficiently to extract
* a DNS header (and therefore a DNS transaction ID)
* a QDCOUNT
* the first question in the QUESTION section if QDCOUNT is
greater than 0
but suffer other issues while parsing. This is the minimum
information required to attempt packet matching as described in
Section 10.1
o Completely malformed: those packets that cannot be decoded to this
extent.
An open question is whether there is value in attempting to process
partially malformed packets in an analogous manner to well formed
packets in terms of attempting to match them with the corresponding
query or response. This could be done by creating 'placeholder'
records during packet matching with just the information extracted as
above. If the packet were then matched the resulting C-DNS Q/R data
item would include a flag to indicate a malformed record (in addition
to capturing the wire format of the packet).
An advantage of this would be that it would result in more meaningful
statistics about matched packets because, for example, some partially
malformed queries could be matched to responses. However it would
only apply to those queries where the first QUESTION is well formed.
It could also simplify the downstream analysis of C-DNS files and the
reconstruction of packet streams from C-DNS.
A disadvantage is that this adds complexity to the packet matching
and data representation, could potentially lead to false matches and
some additional statistics would be required (e.g. counts for
matched-partially-malformed, unmatched-partially-malformed,
completely-malformed).
9. C-DNS to PCAP
It is possible to re-construct PCAP files from the C-DNS format in a
lossy fashion. Some of the issues with reconstructing both the DNS
payload and the full packet stream are outlined here.
The reconstruction depends on whether or not all the optional
sections of both the query and response were captured in the C-DNS sections of both the query and response were captured in the C-DNS
file. Clearly if they were not all captured the reconstruction is file. Clearly, if they were not all captured, the reconstruction
imperfect. will be imperfect.
Secondly, even if all sections of the response were captured name Even if all sections of the response were captured, one cannot
compression presents a challenge in reconstructing the DNS response reconstruct the DNS response payload exactly due to the fact that
payload byte for byte. Section 8.1 discusses this is more detail. some DNS names in the message on the wire may have been compressed.
Section 9.1 discusses this is more detail.
Thirdly, not all transport information is captured in the C-DNS Some transport information is not captured in the C-DNS format. For
format. For example, the following aspects of the original packet example, the following aspects of the original packet stream cannot
stream cannot be re-constructed from the C-DNS format: be re-constructed from the C-DNS format:
o IP Fragmentation o IP fragmentation
o TCP stream information: o TCP stream information:
* Multiple DNS messages may have been sent in a single TCP * Multiple DNS messages may have been sent in a single TCP
segment segment
* A DNS payload may have be split across multiple TCP segments * A DNS payload may have be split across multiple TCP segments
* Multiple DNS messages may have be sent on a single TCP session * Multiple DNS messages may have be sent on a single TCP session
o Malformed DNS messages and non-DNS packets o Malformed DNS messages if the wire format is not recorded
Simple assumptions can be made on the reconstruction - fragmented and o Any Non-DNS messages that were in the original packet stream e.g.
DNS-over-TCP messages can be reconstructed into 'single' packets and ICMP
a single TCP session can be constructed for each TCP packet.
Additionally if the malformed and non-DNS packets are captured Simple assumptions can be made on the reconstruction: fragmented and
separately into PCAPs they can be merged with PCAPs reconstructed DNS-over-TCP messages can be reconstructed into single packets and a
single TCP session can be constructed for each TCP packet.
Additionally, if malformed packets and Non-DNS packets are captured
separately, they can be merged with packet captures reconstructed
from C-DNS to produce a more complete packet stream. from C-DNS to produce a more complete packet stream.
8.1. Name Compression 9.1. Name Compression
All the names stored in the C-DNS format are full domain names; no All the names stored in the C-DNS format are full domain names; no
DNS style name compression is used on the individual names within the DNS style name compression is used on the individual names within the
format. Therefore when reconstructing a packet name compression must format. Therefore when reconstructing a packet, name compression
be used in order to re-produce the on the wire representation of the must be used in order to reproduce the on the wire representation of
packet. the packet.
[RFC1035] name compression works by substituting trailing sections of [RFC1035] name compression works by substituting trailing sections of
a name with a reference back to the occurrence of those sections a name with a reference back to the occurrence of those sections
earlier in the packet. Not all name server software uses the same earlier in the packet. Not all name server software uses the same
algorithm when compressing domain names within the responses. Some algorithm when compressing domain names within the responses. Some
attempt maximum recompression at the expense of runtime resources, attempt maximum recompression at the expense of runtime resources,
others use heuristics to balance compression and speed and others use others use heuristics to balance compression and speed and others use
different rules for what is a valid compression target. different rules for what is a valid compression target.
This means that responses to the same question from different name This means that responses to the same question from different name
server software which match in terms of DNS payload content (header, server software which match in terms of DNS payload content (header,
counts, RRs with name compression removed) do not necessarily match counts, RRs with name compression removed) do not necessarily match
byte for byte on the wire. byte-for-byte on the wire.
From the C-DNS format it is not possible to ensure that the DNS Therefore, it is not possible to ensure that the DNS response payload
response payload is reconstructed byte for byte. However it can at is reconstructed byte-for-byte from C-DNS data. However, it can at
least, in principle, be reconstructed to have the correct payload least, in principle, be reconstructed to have the correct payload
length (since the original response length is captured) if there is length (since the original response length is captured) if there is
enough knowledge of the commonly implemented name compression enough knowledge of the commonly implemented name compression
algorithms. For example, a simplistic approach would be to try each algorithms. For example, a simplistic approach would be to try each
algorithm in turn to see if it reproduces the original length, algorithm in turn to see if it reproduces the original length,
stopping at the first match. This would not guarantee the correct stopping at the first match. This would not guarantee the correct
algorithm has been used as it is possible to match the length whilst algorithm has been used as it is possible to match the length whilst
still not matching the on the wire bytes but without further still not matching the on the wire bytes but, without further
information added to the C-BOR this is the best that can be achieved. information added to the C-DNS data, this is the best that can be
achieved.
Appendix B presents an example of two differing compression Appendix B presents an example of two different compression
algorithms used by well known name server software. algorithms used by well-known name server software.
9. Data Collection 10. Data Collection
This section describes a non-normative proposed algorithm for the This section describes a non-normative proposed algorithm for the
processing of a captured stream of DNS queries and responses and processing of a captured stream of DNS queries and responses and
matching queries/responses where possible. matching queries/responses where possible.
For the purposes of this discussion, it is assumed that the input has For the purposes of this discussion, it is assumed that the input has
been pre-processed such that: been pre-processed such that:
1. All IP fragmentation reassembly, TCP stream reassembly etc. has 1. All IP fragmentation reassembly, TCP stream reassembly, and so
already been performed on, has already been performed
2. Each message is associated with transport meta-data required to 2. Each message is associated with transport metadata required to
generate the Primary ID (see below) generate the Primary ID (see Section 10.2.1)
3. Each message has a well-formed DNS header of 12 bytes and (if 3. Each message has a well-formed DNS header of 12 bytes and (if
present) the first RR in the query section can be parsed to present) the first RR in the Question section can be parsed to
generate the Secondary ID (see below). generate the Secondary ID (see below). As noted earlier, this
requirement can result in a malformed query being removed in the
* As noted earlier, this requirement can result in a malformed pre-processing stage, but the correctly formed response with
query being removed in the pre-processing stage, but the RCODE of FORMERR being present.
correctly formed response with RCODE of FORMERR being present
DNS messages are processed in the order they are delivered to the DNS messages are processed in the order they are delivered to the
application. application. It should be noted that packet capture libraries do not
necessary provide packets in strict chronological order.
o It should be noted that packet capture libraries do not necessary
provide packets in strict chronological order.
[TODO: Discuss the corner cases resulting from this in more detail.] TODO: Discuss the corner cases resulting from this in more detail.
9.1. Matching algorithm 10.1. Matching algorithm
A schematic representation of the algorithm for matching Q/R pairs is A schematic representation of the algorithm for matching Q/R data
shown in the following diagram: items is shown in the following diagram:
Figure showing the packet matching algorithm format (PNG) [5] Figure showing the packet matching algorithm format (PNG) [5]
Figure showing the packet matching algorithm format (SVG) [6] Figure showing the packet matching algorithm format (SVG) [6]
and further details of the algorithm are given in the following Further details of the algorithm are given in the following sections.
sections.
9.2. Message identifiers 10.2. Message identifiers
9.2.1. Primary ID (required) 10.2.1. Primary ID (required)
A Primary ID can be constructed for each message which is composed of A Primary ID is constructed for each message. It is composed of the
the following data: following data:
1. Source IP Address 1. Source IP Address
2. Destination IP Address 2. Destination IP Address
3. Source Port 3. Source Port
4. Destination Port 4. Destination Port
5. Transport 5. Transport
6. DNS Message ID 6. DNS Message ID
9.2.2. Secondary ID (optional) 10.2.2. Secondary ID (optional)
If present, the first question in the Question section is used as a If present, the first question in the Question section is used as a
secondary ID for each message. Note that there may be well formed secondary ID for each message. Note that there may be well formed
DNS queries that have a QDCOUNT of 0, and some responses may have a DNS queries that have a QDCOUNT of 0, and some responses may have a
QDCOUNT of 0 (for example, RCODE=FORMERR or NOTIMP) QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP).
In this case the secondary ID is not used in matching.
9.3. Algorithm Parameters 10.3. Algorithm Parameters
1. Configurable timeout 1. Query timeout
9.4. Algorithm Requirements 2. Skew timeout
10.4. Algorithm Requirements
The algorithm is designed to handle the following input data: The algorithm is designed to handle the following input data:
1. Multiple queries with the same Primary ID (but different 1. Multiple queries with the same Primary ID (but different
Secondary ID) arriving before any responses for these queries are Secondary ID) arriving before any responses for these queries are
seen. seen.
2. Multiple queries with the same Primary and Secondary ID arriving 2. Multiple queries with the same Primary and Secondary ID arriving
before any responses for these queries are seen. before any responses for these queries are seen.
3. Queries for which no later response can be found within the 3. Queries for which no later response can be found within the
specified timeout. specified timeout.
4. Responses for which no previous query can be found within the 4. Responses for which no previous query can be found within the
specified timeout. specified timeout.
9.5. Algorithm Limitations 10.5. Algorithm Limitations
For cases 1 and 2 listed in the above requirements, it is not For cases 1 and 2 listed in the above requirements, it is not
possible to unambiguously match queries with responses. The solution possible to unambiguously match queries with responses. This
to this employed in this algorithm is to match to the earliest query algorithm chooses to match to the earliest query with the correct
with the correct Primary and Secondary ID. Primary and Secondary ID.
9.6. Workspace 10.6. Workspace
A FIFO structure is used to hold the Q/R items during processing. A FIFO structure is used to hold the Q/R data items during
processing.
9.7. Output 10.7. Output
The output is a list of Q/R data items. Both the Query and Response The output is a list of Q/R data items. Both the Query and Response
elements are optional in these items, therefore Q/R items have one of elements are optional in these items, therefore Q/R data items have
three types of content: one of three types of content:
1. Paired Q/R messages 1. A matched pair of query and response messages
2. A query message (no response) 2. A query message with no response
3. A response message (no query) 3. A response message with no query
The timestamp of a list item is that of the query for cases 1 and 2 The timestamp of a list item is that of the query for cases 1 and 2
and that of the response for case 3. and that of the response for case 3.
9.8. Post Processing 10.8. Post Processing
When ending capture, all remaining entries in the Q/R FIFO should be When ending capture, all remaining entries in the Q/R data item FIFO
treated as timed out queries. should be treated as timed out queries.
10. IANA Considerations 11. IANA Considerations
None None
11. Security Considerations 12. Security Considerations
Any control interface MUST perform authentication and encryption. Any control interface MUST perform authentication and encryption.
Any data upload MUST be authenticated and encrypted. Any data upload MUST be authenticated and encrypted.
12. Acknowledgements 13. Acknowledgements
The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for
many useful discussions on binary formats, compression and packet many useful discussions on binary formats, compression and packet
matching. Also Jan Vcelak and Wouter Wijngaards for discussions on matching. Also Jan Vcelak and Wouter Wijngaards for discussions on
name compression. name compression and Paul Hoffman for a detailed review of the
document and the C-DNS CDDL.
Thanks to Robert Edmonds, Paul Hoffman and Jerry Lundstroem for Thanks also to Robert Edmonds and Jerry Lundstroem for review.
review.
Also, Miek Gieben for mmark [7] Also, Miek Gieben for mmark [7]
13. Changelog 14. Changelog
draft-ietf-dnsop-dns-capture-format-01
o Many editorial improvements by Paul Hoffman
o Included discussion of malformed packet handling
o Improved Appendix C on Comparison of Binary Formats
o Now using C-DNS field names in the tables in section 8
o A handful of new fields included (CDDL updated)
o Timestamps now include optional picoseconds
o Added details of block statistics
draft-ietf-dnsop-dns-capture-format-00 draft-ietf-dnsop-dns-capture-format-00
o Changed dnstap.io to dnstap.info o Changed dnstap.io to dnstap.info
o qr_data_format.png was cut off at the bottom o qr_data_format.png was cut off at the bottom
o Update authors address o Update authors address
o Improve wording in Abstract o Improve wording in Abstract
skipping to change at page 26, line 32 skipping to change at page 30, line 30
o Changed DNS-STAT to C-DNS in CDDL o Changed DNS-STAT to C-DNS in CDDL
o Set the format version in the CDDL o Set the format version in the CDDL
o Added a TODO: Add block statistics o Added a TODO: Add block statistics
o Added a TODO: Add extend to support pico/nano. Also do this for o Added a TODO: Add extend to support pico/nano. Also do this for
Time offset and Response delay Time offset and Response delay
o Added a TODO: Need to develop optional representation of malformed o Added a TODO: Need to develop optional representation of malformed
packets within CBOR and what this means for packet matching. This packets within C-DNS and what this means for packet matching.
may influence which fields are optional in the rest of the This may influence which fields are optional in the rest of the
representation. representation.
o Added section on design goals to Introduction o Added section on design goals to Introduction
o Added a TODO: Can Class be optimised? Should a class of IN be o Added a TODO: Can Class be optimised? Should a class of IN be
inferred if not present? inferred if not present?
draft-dickinson-dnsop-dns-capture-format-00 draft-dickinson-dnsop-dns-capture-format-00
o Initial commit o Initial commit
14. References 15. References
14.1. Normative References 15.1. Normative References
[RFC1035] Mockapetris, P., "Domain names - implementation and [RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <http://www.rfc-editor.org/info/rfc1035>. November 1987, <http://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <http://www.rfc-editor.org/info/rfc7049>. October 2013, <http://www.rfc-editor.org/info/rfc7049>.
14.2. Informative References 15.2. Informative References
[ditl] DNS-OARC, "DITL", 2016, <https://www.dns- [ditl] DNS-OARC, "DITL", 2016, <https://www.dns-
oarc.net/oarc/data/ditl>. oarc.net/oarc/data/ditl>.
[dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/ [dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
dnscap>. dnscap>.
[dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>.
[dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
skipping to change at page 27, line 40 skipping to change at page 31, line 40
dnsxml-00 (work in progress), July 2013. dnsxml-00 (work in progress), July 2013.
[I-D.greevenbosch-appsawg-cbor-cddl] [I-D.greevenbosch-appsawg-cbor-cddl]
Vigano, C. and H. Birkholz, "CBOR data definition language Vigano, C. and H. Birkholz, "CBOR data definition language
(CDDL): a notational convention to express CBOR data (CDDL): a notational convention to express CBOR data
structures", draft-greevenbosch-appsawg-cbor-cddl-09 (work structures", draft-greevenbosch-appsawg-cbor-cddl-09 (work
in progress), September 2016. in progress), September 2016.
[I-D.hoffman-dns-in-json] [I-D.hoffman-dns-in-json]
Hoffman, P., "Representing DNS Messages in JSON", draft- Hoffman, P., "Representing DNS Messages in JSON", draft-
hoffman-dns-in-json-09 (work in progress), October 2016. hoffman-dns-in-json-10 (work in progress), October 2016.
[packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
2014, <https://github.com/dotse/PacketQ>. 2014, <https://github.com/dotse/PacketQ>.
[pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.
[pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
Harris, "pcap-ng", 2016, <https://github.com/pcapng/ Harris, "pcap-ng", 2016, <https://github.com/pcapng/
pcapng>. pcapng>.
[RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March
2014, <http://www.rfc-editor.org/info/rfc7159>.
[rrtypes] IANA, "RR types", 2016, <http://www.iana.org/assignments/ [rrtypes] IANA, "RR types", 2016, <http://www.iana.org/assignments/
dns-parameters/dns-parameters.xhtml#dns-parameters-4>. dns-parameters/dns-parameters.xhtml#dns-parameters-4>.
14.3. URIs 15.3. URIs
[1] https://github.com/dns-stats/draft-dns-capture- [1] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/cdns_format.png format/blob/master/cdns_format.png
[2] https://github.com/dns-stats/draft-dns-capture- [2] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/cdns_format.svg format/blob/master/cdns_format.svg
[3] https://github.com/dns-stats/draft-dns-capture- [3] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/qr_data_format.png format/blob/master/qr_data_format.png
skipping to change at page 28, line 31 skipping to change at page 32, line 38
[6] https://github.com/dns-stats/draft-dns-capture- [6] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/packet_matching.svg format/blob/master/packet_matching.svg
[7] https://github.com/miekg/mmark [7] https://github.com/miekg/mmark
[8] https://www.nlnetlabs.nl/projects/nsd/ [8] https://www.nlnetlabs.nl/projects/nsd/
[9] https://www.knot-dns.cz/ [9] https://www.knot-dns.cz/
[10] https://avro.apache.org/
[11] https://developers.google.com/protocol-buffers/
[12] http://cbor.io
[13] https://github.com/kubo/snzip
[14] http://google.github.io/snappy/
[15] http://lz4.github.io/lz4/
[16] http://www.gzip.org/
[17] http://facebook.github.io/zstd/
[18] http://tukaani.org/xz/
[19] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.png
[20] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.svg
Appendix A. CDDL Appendix A. CDDL
; CDDL specification of the file format for C-DNS, ; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS Query/Response pairs. ; which describes a collection of DNS messages and
; traffic meta-data.
File = [ File = [
file-type-id : tstr, ; "C-DNS" file-type-id : tstr, ; = "C-DNS"
file-preamble : FilePreamble, file-preamble : FilePreamble,
file-blocks : [* Block], file-blocks : [* Block],
] ]
FilePreamble = { FilePreamble = {
format-version => uint, ; 1 major-format-version => uint, ; = 1
? configuration => Configuration, minor-format-version => uint, ; = 0
? generator-id => tstr, ? private-version => uint,
? host-id => tstr, ? configuration => Configuration,
} ? generator-id => tstr,
? host-id => tstr,
}
format-version = 0 major-format-version = 0
configuration = 1 minor-format-version = 1
generator-id = 2 private-version = 2
host-id = 3 configuration = 3
Configuration = { generator-id = 4
? query-timeout => uint, host-id = 5
? skew-timeout => uint,
? snaplen => uint,
? promisc => uint,
? interfaces => [* tstr],
? vlan-ids => [* uint],
? filter => tstr,
? query-options => uint, ; See below
? response-options => uint,
? accept-rr-types => [* tstr],
? ignore-rr-types => [* tstr],
}
; query-options and response-options are bitmasks. A bit set adds in the Configuration = {
; specified sections. ? query-timeout => uint,
; ? skew-timeout => uint,
; second & subsequent question sections = 1 ? snaplen => uint,
; answer sections = 2 ? promisc => uint,
; authority sections = 4 ? interfaces => [* tstr],
; additional sections = 8 ? server-addresses => [* IPAddress], ; Hint for later analysis
? vlan-ids => [* uint],
? filter => tstr,
? query-options => QRCollectionSections,
? response-options => QRCollectionSections,
? accept-rr-types => [* uint],
? ignore-rr-types => [* uint],
? max-block-qr-items => uint,
? collect-malformed => uint,
}
query-timeout = 0 QRCollectionSectionValues = &(
skew-timeout = 1 question : 0, ; Second & subsequent question sections
snaplen = 2 answer : 1,
promisc = 3 authority : 2,
interfaces = 4 additional: 3,
vlan-ids = 5 )
filter = 6 QRCollectionSections = uint .bits QRCollectionSectionValues
query-options = 7
response-options = 8
accept-rr-types = 9;
ignore-rr-types = 10;
Block = { query-timeout = 0
preamble => BlockPreamble, skew-timeout = 1
? statistics => BlockStatistics, ; Much of this could be derived snaplen = 2
tables => BlockTables, promisc = 3
queries => [* QueryResponse], interfaces = 4
address-event-counts => [* AddressEventCount], vlan-ids = 5
} filter = 6
query-options = 7
response-options = 8
accept-rr-types = 9
ignore-rr-types = 10
server-addresses = 11
max-block-qr-items = 12
collect-malformed = 13
preamble = 0 Block = {
statistics = 1 preamble => BlockPreamble,
tables = 2 ? statistics => BlockStatistics,
queries = 3 tables => BlockTables,
address-event-counts = 4 queries => [* QueryResponse],
BlockPreamble = { ? address-event-counts => [* AddressEventCount],
start-time => Timeval ? malformed-packet-data => [* MalformedPacket],
} }
start-time = 1 preamble = 0
statistics = 1
tables = 2
queries = 3
address-event-counts = 4
malformed-packet-data = 5
Timeval = [ BlockPreamble = {
seconds : uint, earliest-time => Timeval
microseconds : uint, }
] earliest-time = 1
BlockStatistics = { Timeval = [
? total-packets => uint, seconds : uint,
? total-pairs => uint, microseconds : uint,
? unmatched_queries => uint, ? picoseconds : uint,
? unmatched_responses => uint, ]
? malformed-packets => uint,
? non-dns-packets => uint,
? out-of-order-packets => uint,
? missing-pairs => uint,
? missing-packets => uint,
? missing-non-dns => uint,
}
total-packets = 0 BlockStatistics = {
total-pairs = 1 ? total-packets => uint,
unmatched_queries = 2 ? total-pairs => uint,
unmatched_responses = 3 ? unmatched-queries => uint,
malformed-packets = 4 ? unmatched-responses => uint,
non-dns-packets = 5 ? malformed-packets => uint,
out-of-order-packets = 6 }
missing-pairs = 7
missing-packets = 8
missing-non-dns = 9
BlockTables = { total-packets = 0
ip-address => [* bstr], total-pairs = 1
classtype => [* ClassType], unmatched-queries = 2
name-rdata => [* bstr], ; Holds both Name RDATA and RDATA unmatched-responses = 3
query_sig => [* QuerySignature] malformed-packets = 4
? qlist => [* QuestionList],
? qrr => [* Question],
? rrlist => [* RRList],
? rr => [* RR],
}
ip-address = 0 BlockTables = {
classtype = 1 ip-address => [* IPAddress],
name-rdata = 2 classtype => [* ClassType],
query_sig = 3 name-rdata => [* bstr], ; Holds both Name RDATA and RDATA
qlist = 4 query-sig => [* QuerySignature]
qrr = 5 ? qlist => [* QuestionList],
rrlist = 6 ? qrr => [* Question],
rr = 7 ? rrlist => [* RRList],
? rr => [* RR],
}
QueryResponse = { ip-address = 0
time-useconds => uint, ; Time offset from earliest record classtype = 1
client-address-index => uint, name-rdata = 2
client-port => uint, query-sig = 3
transaction-id => uint, qlist = 4
query-signature-index => uint, qrr = 5
? client-hoplimit => uint, rrlist = 6
? delay-useconds => int, ; Times may be -ve at capture rr = 7
? query-name-index => uint,
? response-size => uint, ; DNS size of response
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
time-useconds = 0 QueryResponse = {
client-address-index = 1 time-useconds => uint, ; Time offset from start of block
client-port = 2 ? time-pseconds => uint, ; in microseconds and picoseconds
transaction-id = 3 client-address-index => uint,
query-signature-index = 4 client-port => uint,
client-hoplimit = 5 transaction-id => uint,
delay-useconds = 6 query-signature-index => uint,
query-name-index = 7 ? client-hoplimit => uint,
response-size = 8 ? delay-useconds => int,
query-extended = 9 ? delay-pseconds => int, ; Has same sign as delay-useconds
response-extended = 10 ? query-name-index => uint,
? query-size => uint, ; DNS size of query
? response-size => uint, ; DNS size of response
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
ClassType = { time-useconds = 0
type => uint, time-pseconds = 1
class => uint, client-address-index = 2
} client-port = 3
transaction-id = 4
query-signature-index = 5
client-hoplimit = 6
delay-useconds = 7
delay-pseconds = 8
query-name-index = 9
query-size = 10
response-size = 11
query-extended = 12
response-extended = 13
type = 0 ClassType = {
class = 1 type => uint,
class => uint,
}
QuerySignature = { type = 0
server-address-index => uint, class = 1
server-port => uint,
transport-flags => uint,
qr-sig-flags => uint,
? query-opcode => uint,
qr-dns-flags => uint,
? query-rcode => uint,
? query-classtype-index => uint,
? query-qd-count => uint,
? query-an-count => uint,
? query-ar-count => uint,
? query-ns-count => uint,
? edns-version => uint,
? udp-buf-size => uint,
? opt-rdata-index => uint,
? response-rcode => uint,
}
server-address-index = 0 DNSFlagValues = &(
server-port = 1 query-cd : 0,
transport-flags = 2 query-ad : 1,
qr-sig-flags = 3 query-z : 2,
query-opcode = 4 query-ra : 3,
qr-dns-flags = 5 query-rd : 4,
query-rcode = 6 query-tc : 5,
query-classtype-index = 7 query-aa : 6,
query-qd-count = 8 query-d0 : 7,
query-an-count = 9 response-cd: 8,
query-ar-count = 10 response-ad: 9,
query-ns-count = 11 response-z : 10,
edns-version = 12 response-ra: 11,
udp-buf-size = 13 response-rd: 12,
opt-rdata-index = 14 response-tc: 13,
response-rcode = 15 response-aa: 14,
)
DNSFlags = uint .bits DNSFlagValues
QuestionList = [ QueryResponseFlagValues = &(
* uint, ; Index of Question has-query : 0,
] has-reponse : 1,
query-has-question : 2,
query-has-opt : 3,
response-has-opt : 4,
response-has-no-question: 5,
)
QueryResponseFlags = uint .bits QueryResponseFlagValues
Question = { ; Second and subsequent questions TransportFlagValues = &(
name-index => uint, ; Index to a name in the name-rdata table tcp : 0,
classtype-index => uint, ipv6 : 1,
} query-trailingdata: 2,
)
TransportFlags = uint .bits TransportFlagValues
name-index = 0 QuerySignature = {
classtype-index = 1 server-address-index => uint,
server-port => uint,
transport-flags => TransportFlags,
qr-sig-flags => QueryResponseFlags,
? query-opcode => uint,
qr-dns-flags => DNSFlags,
? query-rcode => uint,
? query-classtype-index => uint,
? query-qd-count => uint,
? query-an-count => uint,
? query-ar-count => uint,
? query-ns-count => uint,
? edns-version => uint,
? udp-buf-size => uint,
? opt-rdata-index => uint,
? response-rcode => uint,
}
RRList = [ server-address-index = 0
* uint, ; Index of RR server-port = 1
] transport-flags = 2
qr-sig-flags = 3
query-opcode = 4
qr-dns-flags = 5
query-rcode = 6
query-classtype-index = 7
query-qd-count = 8
query-an-count = 9
query-ar-count = 10
query-ns-count = 11
edns-version = 12
udp-buf-size = 13
opt-rdata-index = 14
response-rcode = 15
RR = { QuestionList = [
name-index => uint, ; Index to a name in the name-rdata table * uint, ; Index of Question
classtype-index => uint, ]
ttl => uint,
rdata-index => uint, ; Index to RDATA in the name-rdata table
}
ttl = 2 Question = { ; Second and subsequent questions
rdata-index = 3 name-index => uint, ; Index to a name in the name-rdata table
classtype-index => uint,
}
QueryResponseExtended = { name-index = 0
? question-index => uint, ; Index of QuestionList classtype-index = 1
? answer-index => uint, ; Index of RRList
? authority-index => uint,
? additional-index => uint,
}
question-index = 0 RRList = [
answer-index = 1 * uint, ; Index of RR
authority-index = 2 ]
additional-index = 3
AddressEventCount = { RR = {
ae-type => &AddressEventType, name-index => uint, ; Index to a name in the name-rdata table
? ae-code => uint, classtype-index => uint,
ae-address-index => uint, ttl => uint,
ae-count => uint, rdata-index => uint, ; Index to RDATA in the name-rdata table
} }
ae-type = 0 ttl = 2
ae-code = 1 rdata-index = 3
ae-address-index = 2
ae-count = 3
AddressEventType = ( QueryResponseExtended = {
tcp-reset: 0, ? question-index => uint, ; Index of QuestionList
icmp-time-exceeded : 1, ? answer-index => uint, ; Index of RRList
icmp-dest-unreachable : 2, ? authority-index => uint,
icmpv6-time-exceeded : 3, ? additional-index => uint,
icmpv6-dest-unreachable: 4, }
icmpv6-packet-too-big : 5,
) question-index = 0
answer-index = 1
authority-index = 2
additional-index = 3
AddressEventCount = {
ae-type => &AddressEventType,
? ae-code => uint,
ae-address-index => uint,
ae-count => uint,
}
ae-type = 0
ae-code = 1
ae-address-index = 2
ae-count = 3
AddressEventType = (
tcp-reset : 0,
icmp-time-exceeded : 1,
icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5,
)
MalformedPacket = {
time-useconds => uint, ; Time offset from start of block
? time-pseconds => uint, ; in microseconds and picoseconds
packet-content => bstr, ; Raw packet contents
}
time-useconds = 0
time-pseconds = 1
packet-content = 2
IPv4Address = bstr .size 4
IPv6Address = bstr .size 16
IPAddress = IPv4Address / IPv6Address
Appendix B. DNS Name compression example Appendix B. DNS Name compression example
The basic algorithm which follows the guidance in [RFC1035] is simply The basic algorithm, which follows the guidance in [RFC1035], is
to collect each name, and the offset in the packet at which it simply to collect each name, and the offset in the packet at which it
starts, during packet construction. As each name is added, it is starts, during packet construction. As each name is added, it is
offered to each of the collected names in order of collection, offered to each of the collected names in order of collection,
starting from the first name. If labels at the end of the name can starting from the first name. If labels at the end of the name can
be replaced with a reference back to part (or all) of the earlier be replaced with a reference back to part (or all) of the earlier
name, and if the uncompressed part of the name is shorter than any name, and if the uncompressed part of the name is shorter than any
compression already found, the earlier name is noted as the compression already found, the earlier name is noted as the
compression target for the name. compression target for the name.
The following tables illustrate the process. In an example packet, The following tables illustrate the process. In an example packet,
the first name is example.com. the first name is example.com.
skipping to change at page 34, line 43 skipping to change at page 40, line 40
name is www, this is an improved compression, and so it is adopted. name is www, this is an improved compression, and so it is adopted.
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| 1 | example.com | | | | 1 | example.com | | |
| 2 | bar.com | bar | 1 + offset to com | | 2 | bar.com | bar | 1 + offset to com |
| 3 | www.bar.com | www | 2 | | 3 | www.bar.com | www | 2 |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
As an optimization, if a name is already perfectly compressed - in As an optimization, if a name is already perfectly compressed (in
other words, the uncompressed part of the name is empty - no further other words, the uncompressed part of the name is empty), then no
names will be considered for compression. further names will be considered for compression.
B.1. NSD compression algorithm B.1. NSD compression algorithm
Using the above basic algorithm the packet lengths of responses Using the above basic algorithm the packet lengths of responses
generated by NSD [8] can be matched almost exactly. At the time of generated by NSD [8] can be matched almost exactly. At the time of
writing, a tiny number (<.01%) of the reconstructed packets had writing, a tiny number (<.01%) of the reconstructed packets had
incorrect lengths. incorrect lengths.
B.2. Knot Authoritative compression algorithm B.2. Knot Authoritative compression algorithm
skipping to change at page 35, line 26 skipping to change at page 41, line 24
A set of smart heuristics as described below can be implemented to A set of smart heuristics as described below can be implemented to
mimic this and while not perfect it produces output nearly, but not mimic this and while not perfect it produces output nearly, but not
quite, as good a match as with NSD. The heuristics are: quite, as good a match as with NSD. The heuristics are:
1. A match is only perfect if the name is completely compressed AND 1. A match is only perfect if the name is completely compressed AND
the TYPE of the section in which the name occurs matches the TYPE the TYPE of the section in which the name occurs matches the TYPE
of the name used as the compression target. of the name used as the compression target.
2. If the name occurs in RDATA: 2. If the name occurs in RDATA:
a If the compression target name is in a query, then only the * If the compression target name is in a query, then only the
first RR in an RRSET can use that name as a compression first RR in an RRSET can use that name as a compression
target. target.
b The compression target name MUST be in RDATA. * The compression target name MUST be in RDATA.
c The name section TYPE must match the compression target name * The name section TYPE must match the compression target name
section TYPE. section TYPE.
d The compression target name MUST be in the immediately * The compression target name MUST be in the immediately
preceding RR in the RRSET. preceding RR in the RRSET.
Using this algorithm less than 0.1% of the reconstructed packets had Using this algorithm less than 0.1% of the reconstructed packets had
incorrect lengths. incorrect lengths.
B.3. Observed differences B.3. Observed differences
In sample traffic collected on a root name server around 2-4% of In sample traffic collected on a root name server around 2-4% of
responses generated by Knot had different packet lengths to those responses generated by Knot had different packet lengths to those
produced by NSD. produced by NSD.
Appendix C. Comparison of Binary Formats Appendix C. Comparison of Binary Formats
Several binary representations were considered in particular CBOR, Several binary serialisation formats were considered, and for
Apache Avro and Protocol Buffers. completeness were also compared to JSON.
Protocol Buffers and Avro both require a data schema, and validate o Apache Avro [10]. Data is stored according to a pre-defined
data being stored against that schema. schema. The schema itself is always included in the data file.
[TODO: Finish pros and cons of CBOR vs Avro vs Protocol buffers - Data can therefore be stored untagged, for a smaller serialisation
tools, schema, adoption, etc.] size, and be written and read by an Avro library.
The difference in file sizes were mostly minimal See Appendix D.3. * At the time of writing, Avro libraries are available for C,
C++, C#, Java, Python, Ruby and PHP. Optionally tools are
available for C++, Java and C# to generate code for encoding
and decoding.
Appendix D. Sample data on the C-DNS format o Google Protocol Buffers [11]. Data is stored according to a pre-
defined schema. The schema is used by a generator to generate
code for encoding and decoding the data. Data can therefore be
stored untagged, for a smaller serialisation size. The schema is
not stored with the data, so unlike Avro cannot be read with a
generic library.
This section presents some example figures for the output size of * Code must be generated for a particular data schema to to read
capture files when using different block sizes, data representations and write data using that schema. At the time of writing, the
and binary formats. The data is sample data for a root instance. Google code generator can currently generate code for encoding
and decoding a schema for C++, Go, Java, Python, Ruby, C#,
Objective-C, Javascript and PHP.
[TODO: This section needs more work..] o CBOR [12]. Defined in [RFC7049], this serialisation format is
comparable to JSON but with a binary representation. It does not
use a pre-defined schema, so data is always stored tagged.
However, CBOR data schemas can be described using CDDL
[I-D.greevenbosch-appsawg-cbor-cddl] and tools exist to verify
data files conform to the schema.
D.1. Comparison to full PCAPS * CBOR is a simple format, and simple to implement. At the time
of writing, the CBOR website lists implementations for 16
languages.
As can be seen in more detail below for this sample data the Avro and Protocol Buffers both allow storage of untagged data, but
compressed C-DNS files are around 30% the size of the full compressed because they rely on the data schema for this, their implementation
PCAPs. It should also be noted that experiments showed that is considerably more complex than CBOR. Using Avro or Protocol
compression of the C-DNS format required very roughly an order of Buffers in an unsupported environment would require notably greater
magnitude less CPU resources than compression of full PCAPSs when development effort compared to CBOR.
using one core from a 3.5GHz i7 processor.
D.2. Block size choices A test program was written which reads input from a PCAP file and
writes output using one of two basic structures; either a simple
structure, where each query/response pair is represented in a single
record entry, or the C-DNS block structure.
[TODO: Discuss trade-off of file block size vs memory consumption.] The resulting output files were then compressed using a variety of
common general-purpose lossless compression tools to explore the
compressibility of the formats. The compression tools employed were:
[TODO: Add graph that demonstrates block size of 5000 is optimal for o snzip [13]. A command line compression tool based on the Google
the sample data used.] Snappy [14] library.
D.3. Blocking vs more simple output o lz4 [15]. The command line compression tool from the reference C
LZ4 implementation.
Some experiments were conducted producing output in a very simple o gzip [16]. The ubiquitous GNU zip tool.
format involving a single record per Q/R data item (akin to a .csv
representation). The aim here was to examine whether the blocking
mechanism (using a block size of 5000) was worth the complexity,
particularly after compression of the output file using several
general purpose compression tools. The original PCAP file was
325.79M and compressed using xz to 24.3Mb.
+-------------+-------------+--------+--------+-------+ o zstd [17]. Compression using the Zstandard algorithm.
| Format | Output size | lz4 | gzip | xz |
+-------------+-------------+--------+--------+-------+
| cbor-simple | 44.23M | 16.06M | 11.50M | 7.51M |
| cbor-block | 22.44M | 15.14M | 10.70M | 7.23M |
+-------------+-------------+--------+--------+-------+
It might be expected that blocking is exploiting commonality that a o xz [18]. A popular compression tool noted for high compression.
general purpose compression engine could also exploit, and the
figures do indeed bear this out. The more powerful (and resource-
consuming) the compression, the closer the compressed simple file
size gets to the compressed chunk file size. With no compression,
the blocked output size is typically half that of the simple output,
but as greater degrees of compression are applied the gap shrinks.
However, even with the stronger compressor, the chunked output
remains roughly 5-10% smaller than the simple output. This, and the
higher gains at lower compression, might be significant, depending on
the target environment.
[TODO: Add data on reduction in CPU overhead of compressing blocked In all cases the compression tools were run using their default
output vs simple output.] settings.
This was repeated using some other binary representations: Note that this draft does not mandate the use of compression, nor any
particular compression scheme, but it anticipates that in practice
output data will be subject to general-purpose compression, and so
this should be taken into consideration.
+-----------------+-------------+--------+--------+-------+ "test.pcap", a 662Mb capture of sample data from a root instance was
| Format | Output size | lz4 | gzip | xz | used for the comparison. The following table shows the formatted
+-----------------+-------------+--------+--------+-------+ size and size after compression (abbreviated to Comp. in the table
| json-simple | 189.85M | 25.59M | 16.03M | 9.74M | headers), together with the task resident set size (RSS) and the user
| avro-simple | 43.31M | 16.07M | 11.92M | 7.99M | time taken by the compression. File sizes are in Mb, RSS in kb and
| avro-block | 17.44M | 12.94M | 10.08M | 7.18M | user time in seconds.
| protobuf-simple | 46.02M | 15.79M | 11.59M | 7.94M |
| protobuf-block | 22.08M | 15.43M | 10.91M | 7.40M |
+-----------------+-------------+--------+--------+-------+
There's not a lot to choose between the three contenders with simple +-------------+-----------+-------+------------+-------+-----------+
output. Avro produces the smaller output, CBOR the next and Protocol | Format | File size | Comp. | Comp. size | RSS | User time |
Buffers the largest, but the different is under 10%. However, with +-------------+-----------+-------+------------+-------+-----------+
blocking, while CBOR and Protocol Buffers are again within a few | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 |
percentage points of each other (though Protocol Buffers now has a | | | lz4 | 181.58 | 6336 | 1.35 |
slight advantage), Avro produces files in the region of 20% smaller, | | | gzip | 153.46 | 1428 | 18.20 |
and holds a diminishing advantage through increased compression. | | | zstd | 87.07 | 3544 | 4.27 |
| | | xz | 49.09 | 97416 | 160.79 |
| | | | | | |
| JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 |
| | | lz4 | 386.42 | 5636 | 5.25 |
| | | gzip | 271.11 | 1492 | 73.00 |
| | | zstd | 133.43 | 3284 | 8.68 |
| | | xz | 51.98 | 97412 | 600.74 |
| | | | | | |
| Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 |
| | | lz4 | 111.92 | 5828 | 0.99 |
| | | gzip | 103.07 | 1540 | 11.52 |
| | | zstd | 49.08 | 3524 | 2.50 |
| | | xz | 22.87 | 97308 | 90.34 |
| | | | | | |
| CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 |
| | | lz4 | 120.98 | 5892 | 1.13 |
| | | gzip | 110.61 | 1428 | 12.88 |
| | | zstd | 54.14 | 3224 | 2.77 |
| | | xz | 23.43 | 97276 | 111.48 |
| | | | | | |
| PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 |
| | | lz4 | 123.09 | 5824 | 1.14 |
| | | gzip | 112.05 | 1424 | 12.75 |
| | | zstd | 53.39 | 3388 | 2.76 |
| | | xz | 23.99 | 97348 | 106.47 |
| | | | | | |
| JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 |
| | | lz4 | 104.34 | 6080 | 0.97 |
| | | gzip | 57.97 | 1604 | 12.70 |
| | | zstd | 61.51 | 3396 | 3.45 |
| | | xz | 27.67 | 97524 | 169.10 |
| | | | | | |
| Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 |
| | | lz4 | 48.78 | 8540 | 0.22 |
| | | gzip | 39.62 | 1576 | 2.92 |
| | | zstd | 29.63 | 3612 | 1.25 |
| | | xz | 18.28 | 97564 | 25.81 |
| | | | | | |
| CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 |
| | | lz4 | 51.88 | 8008 | 0.28 |
| | | gzip | 41.17 | 1548 | 4.36 |
| | | zstd | 30.61 | 3476 | 1.48 |
| | | xz | 18.15 | 97556 | 38.78 |
| | | | | | |
| PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 |
| | | lz4 | 52.39 | 8304 | 0.24 |
| | | gzip | 40.19 | 1520 | 3.63 |
| | | zstd | 31.61 | 3576 | 1.40 |
| | | xz | 17.94 | 97440 | 33.99 |
+-------------+-----------+-------+------------+-------+-----------+
The above results are discussed in the following sections.
C.1. Comparison with full PCAP files
An important first consideration is whether moving away from PCAP
offers significant benefits.
The simple binary formats are typically larger than PCAP, even though
they omit some information such as Ethernet MAC addresses. But not
only do they require less CPU to compress than PCAP, the resulting
compressed files are smaller than compressed PCAP.
C.2. Simple versus block coding
The intention of the block coding is to perform data de-duplication
on query/response records within the block. The simple and block
formats above store exactly the same information for each query/
response record. This information is parsed from the DNS traffic in
the input PCAP file, and in all cases each field has an identifier
and the field data is typed.
The data de-duplication on the block formats show an order of
magnitude reduction in the size of the format file size against the
simple formats. As would be expected, the compression tools are able
to find and exploit a lot of this duplication, but as the de-
duplication process uses knowledge of DNS traffic, it is able to
retain a size advantage. This advantage reduces as stronger
compression is applied, as again would be expected, but even with the
strongest compression applied the block formatted data remains around
75% of the size of the simple format and its compression requires
roughly a third of the CPU time.
C.3. Binary versus text formats
Text data formats offer many advantages over binary formats,
particularly in the areas of ad-hoc data inspection and extraction.
It was therefore felt worthwhile to carry out a direct comparison,
implementing JSON versions of the simple and block formats.
Concentrating on JSON block format, the format files produced are a
significant fraction of an order of magnitude larger than binary
formats. The impact on file size after compression is as might be
expected from that starting point; the stronger compression produces
files that are 150% of the size of similarly compressed binary
format, and require over 4x more CPU to compress.
C.4. Performance
Concentrating again on the block formats, all three produce format
files that are close to an order of magnitude smaller that the
original "test.pcap" file. CBOR produces the largest files and Avro
the smallest, 20% smaller than CBOR.
However, once compression is taken into account, the size difference
narrows. At medium compression (with gzip), the size difference is
4%. Using strong compression (with xz) the difference reduces to 2%,
with Avro the largest and Protocol Buffers the smallest, although
CBOR and Protocol Buffers require slightly more compression CPU.
The measurements presented above do not include data on the CPU
required to generate the format files. Measurements indicate that
writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It
appears, therefore, that Avro's advantage in compression CPU usage is
probably offset by a larger CPU requirement in writing Avro.
C.5. Conclusions
The above assessments lead us to the choice of a binary format file
using blocking.
As noted previously, this draft anticipates that output data will be
subject to compression. There is no compelling case for one
particular binary serialisation format in terms of either final file
size or machine resources consumed, so the choice must be largely
based on other factors. CBOR was therefore chosen as the binary
serialisation format for the reasons listed in Section 6.
C.6. Block size choice
Given the choice of a CBOR format using blocking, the question arises
of what an appropriate default value for the maximum number of query/
response pairs in a block should be. This has two components; what
is the impact on performance of using different block sizes in the
format file, and what is the impact on the size of the format file
before and after compression.
The following table addresses the performance question, showing the
impact on the performance of a C++ program converting "test.pcap" to
C-DNS. File size is in Mb, resident set size (RSS) in kb.
+------------+-----------+--------+-----------+
| Block size | File size | RSS | User time |
+------------+-----------+--------+-----------+
| 1000 | 133.46 | 612.27 | 15.25 |
| 5000 | 89.85 | 676.82 | 14.99 |
| 10000 | 76.87 | 752.40 | 14.53 |
| 20000 | 67.86 | 750.75 | 14.49 |
| 40000 | 61.88 | 736.30 | 14.29 |
| 80000 | 58.08 | 694.16 | 14.28 |
| 160000 | 55.94 | 733.84 | 14.44 |
| 320000 | 54.41 | 799.20 | 13.97 |
+------------+-----------+--------+-----------+
Increasing block size, therefore, tends to increase maximum RSS a
little, with no significant effect (if anything a small reduction) on
CPU consumption.
The following figure plots the effect of increasing block size on
output file size for different compressions.
Figure showing effect of block size on file size (PNG) [19]
Figure showing effect of block size on file size (SVG) [20]
From the above, there is obviously scope for tuning the default block
size to the compression being employed, traffic characteristics,
frequency of output file rollover etc. Using a strong compression,
block sizes over 10,000 query/response pairs would seem to offer
limited improvements.
Authors' Addresses Authors' Addresses
John Dickinson John Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
Email: jad@sinodun.com Email: jad@sinodun.com
 End of changes. 218 change blocks. 
960 lines changed or deleted 1412 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/