draft-ietf-dnsop-dns-capture-format-07.txt   draft-ietf-dnsop-dns-capture-format-08.txt 
dnsop J. Dickinson dnsop J. Dickinson
Internet-Draft J. Hague Internet-Draft J. Hague
Intended status: Standards Track S. Dickinson Intended status: Standards Track S. Dickinson
Expires: November 9, 2018 Sinodun IT Expires: February 11, 2019 Sinodun IT
T. Manderson T. Manderson
J. Bond J. Bond
ICANN ICANN
May 8, 2018 August 10, 2018
C-DNS: A DNS Packet Capture Format C-DNS: A DNS Packet Capture Format
draft-ietf-dnsop-dns-capture-format-07 draft-ietf-dnsop-dns-capture-format-08
Abstract Abstract
This document describes a data representation for collections of DNS This document describes a data representation for collections of DNS
messages. The format is designed for efficient storage and messages. The format is designed for efficient storage and
transmission of large packet captures of DNS traffic; it attempts to transmission of large packet captures of DNS traffic; it attempts to
minimize the size of such packet capture files but retain the full minimize the size of such packet capture files but retain the full
DNS message contents along with the most useful transport metadata. DNS message contents along with the most useful transport metadata.
It is intended to assist with the development of DNS traffic It is intended to assist with the development of DNS traffic
monitoring applications. monitoring applications.
skipping to change at page 1, line 40 skipping to change at page 1, line 40
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 9, 2018. This Internet-Draft will expire on February 11, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 19 skipping to change at page 2, line 19
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Data collection use cases . . . . . . . . . . . . . . . . . . 5 3. Data collection use cases . . . . . . . . . . . . . . . . . . 5
4. Design considerations . . . . . . . . . . . . . . . . . . . . 7 4. Design considerations . . . . . . . . . . . . . . . . . . . . 7
5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8
6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9 6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9
6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 10 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13
6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 10 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13
6.2.1. Optional data items . . . . . . . . . . . . . . . . . 11 6.2.1. Optional data items . . . . . . . . . . . . . . . . . 13
6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 12 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 14
6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 12 6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15
6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 12 6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 15
7. C-DNS format detailed description . . . . . . . . . . . . . . 13 7. C-DNS format detailed description . . . . . . . . . . . . . . 15
7.1. Map quantities and indexes . . . . . . . . . . . . . . . 13 7.1. Map quantities and indexes . . . . . . . . . . . . . . . 15
7.2. Tabular representation . . . . . . . . . . . . . . . . . 13 7.2. Tabular representation . . . . . . . . . . . . . . . . . 16
7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 14 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 14 7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 17
7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 15 7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 18
7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 18 7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 21
7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 19 7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 20 7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 23
7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 21 7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 24
7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 22 7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 25
7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 28 7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 31
7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 30 7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 33
7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 30 7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 33
7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 31 7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 34
7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 32 7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 35
8. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 33 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 36
8.1. Name compression . . . . . . . . . . . . . . . . . . . . 34 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 36
9. Data collection . . . . . . . . . . . . . . . . . . . . . . . 34 9.1. Name compression . . . . . . . . . . . . . . . . . . . . 37
9.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 35 10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 38
9.2. Message identifiers . . . . . . . . . . . . . . . . . . . 36 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 39
9.2.1. Primary ID (required) . . . . . . . . . . . . . . . . 36 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 41
9.2.2. Secondary ID (optional) . . . . . . . . . . . . . . . 36 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 41
9.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 36 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 42
9.4. Algorithm requirements . . . . . . . . . . . . . . . . . 36 10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 42
9.5. Algorithm limitations . . . . . . . . . . . . . . . . . . 37 10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 42
9.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . . 37 10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 42
9.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 37 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 43
9.8. Post processing . . . . . . . . . . . . . . . . . . . . . 37 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 43
10. Implementation guidance . . . . . . . . . . . . . . . . . . . 38 10.8. Post processing . . . . . . . . . . . . . . . . . . . . 43
10.1. Optional data . . . . . . . . . . . . . . . . . . . . . 38 11. Implementation guidance . . . . . . . . . . . . . . . . . . . 43
10.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 38 11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 44
10.3. Limiting collection of RDATA . . . . . . . . . . . . . . 39 11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 44
11. Implementation status . . . . . . . . . . . . . . . . . . . . 39 11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 44
11.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 39 12. Implementation status . . . . . . . . . . . . . . . . . . . . 44
12. IANA considerations . . . . . . . . . . . . . . . . . . . . . 40 12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 45
13. Security considerations . . . . . . . . . . . . . . . . . . . 40 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 45
14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 40 14. Security considerations . . . . . . . . . . . . . . . . . . . 46
15. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 40 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46
16. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 16. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 46
16.1. Normative References . . . . . . . . . . . . . . . . . . 43 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 49
16.2. Informative References . . . . . . . . . . . . . . . . . 43 17.1. Normative References . . . . . . . . . . . . . . . . . . 49
16.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 44 17.2. Informative References . . . . . . . . . . . . . . . . . 49
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 45 17.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Appendix B. DNS Name compression example . . . . . . . . . . . . 55 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 51
B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 56 Appendix B. DNS Name compression example . . . . . . . . . . . . 61
B.2. Knot Authoritative compression algorithm . . . . . . . . 56 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 62
B.3. Observed differences . . . . . . . . . . . . . . . . . . 57 B.2. Knot Authoritative compression algorithm . . . . . . . . 62
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 57 B.3. Observed differences . . . . . . . . . . . . . . . . . . 63
C.1. Comparison with full PCAP files . . . . . . . . . . . . . 60 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 63
C.2. Simple versus block coding . . . . . . . . . . . . . . . 60 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 66
C.3. Binary versus text formats . . . . . . . . . . . . . . . 61 C.2. Simple versus block coding . . . . . . . . . . . . . . . 66
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 61 C.3. Binary versus text formats . . . . . . . . . . . . . . . 67
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 61 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 67
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 62 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 67
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 63 C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 68
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 68
1. Introduction 1. Introduction
There has long been a need to collect DNS queries and responses on There has long been a need to collect DNS queries and responses on
authoritative and recursive name servers for monitoring and analysis. authoritative and recursive name servers for monitoring and analysis.
This data is used in a number of ways including traffic monitoring, This data is used in a number of ways including traffic monitoring,
analyzing network attacks and "day in the life" (DITL) [ditl] analyzing network attacks and "day in the life" (DITL) [ditl]
analysis. analysis.
A wide variety of tools already exist that facilitate the collection A wide variety of tools already exist that facilitate the collection
skipping to change at page 4, line 41 skipping to change at page 4, line 41
Section 4. Section 4.
o A description of why CBOR [RFC7049] was chosen for this format, o A description of why CBOR [RFC7049] was chosen for this format,
see Section 5. see Section 5.
o A conceptual overview of the C-DNS format, see Section 6. o A conceptual overview of the C-DNS format, see Section 6.
o The definition of the C-DNS format for the collection of DNS o The definition of the C-DNS format for the collection of DNS
messages, see Section 7. messages, see Section 7.
o Notes on converting C-DNS data to PCAP format, see Section 8. o Notes on converting C-DNS data to PCAP format, see Section 9.
o Some high level implementation considerations for applications o Some high level implementation considerations for applications
designed to produce C-DNS, see Section 9. designed to produce C-DNS, see Section 10.
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
"Packet" refers to an individual IPv4 or IPv6 packet. Typically "Packet" refers to an individual IPv4 or IPv6 packet. Typically
packets are UDP datagrams, but may also be part of a TCP data stream. packets are UDP datagrams, but may also be part of a TCP data stream.
"Message", unless otherwise qualified, refers to a DNS payload "Message", unless otherwise qualified, refers to a DNS payload
skipping to change at page 9, line 34 skipping to change at page 9, line 34
o CBOR data schemas can be described using CDDL o CBOR data schemas can be described using CDDL
[I-D.ietf-cbor-cddl]. [I-D.ietf-cbor-cddl].
6. C-DNS format conceptual overview 6. C-DNS format conceptual overview
The following figures show purely schematic representations of the The following figures show purely schematic representations of the
C-DNS format to convey the high-level structure of the C-DNS format. C-DNS format to convey the high-level structure of the C-DNS format.
Section 7 provides a detailed discussion of the CBOR representation Section 7 provides a detailed discussion of the CBOR representation
and individual elements. and individual elements.
Figure showing the C-DNS format (PNG) [1] Figure 1 shows the C-DNS format at the top level including the file
header and data blocks. The Query/Response data items, Address/Event
Count data items and Malformed Message data items link to various
Block tables.
Figure showing the C-DNS format (SVG) [2] +-------+
+ C-DNS |
+-------+--------------------------+
| File type identifier |
+----------------------------------+
| File preamble |
| +--------------------------------+
| | Format version info |
| +--------------------------------+
| | Block parameters |
+-+--------------------------------+
| Block |
| +--------------------------------+
| | Block preamble |
| +--------------------------------+
| | Block statistics |
| +--------------------------------+
| | Block tables |
| +--------------------------------+
| | Query/Response data items |
| +--------------------------------+
| | Address/Event Count data items |
| +--------------------------------+
| | Malformed Message data items |
+-+--------------------------------+
| Block |
| +--------------------------------+
| | Block preamble |
| +--------------------------------+
| | Block statistics |
| +--------------------------------+
| | Block tables |
| +--------------------------------+
| | Query/Response data items |
| +--------------------------------+
| | Address/Event Count data items |
| +--------------------------------+
| | Malformed Message data items |
+-+--------------------------------+
| Further Blocks... |
+----------------------------------+
Figure showing the Query/Response data item and Block Tables format Figure 1: The C-DNS format.
(PNG) [3]
Figure showing the Query/Response item and Block Tables format (SVG) Figure 2 shows some more detailed relationships within each block,
[4] specifically those between the Query/Response data item and the
relevant Block tables.
+----------------+
| Query/Response |
+-------------------------+
| Time offset |
+-------------------------+ +------------------+
| Client address |------------>| IP address array |
+-------------------------+ +------------------+
| Client port |
+-------------------------+ +------------------+
| Transaction ID | +------>| Name/RDATA array |<------+
+-------------------------+ | +------------------+ |
| Query signature |--+ | |
+-------------------------+ | | +-----------------+ |
| Client hoplimit (q) | +--)------>| Query Signature | |
+-------------------------+ | +-----------------+------+ |
| Response delay (r) | | | Server address | |
+-------------------------+ | +------------------------+ |
| Query name (q) |--+--+ | Server port | |
+-------------------------+ | +------------------------+ |
| Query size (q) | | | Transport flags | |
+-------------------------+ | +------------------------+ |
| Response size (r) | | | QR type | |
+-------------------------+ | +------------------------+ |
| Response processing (r) | | | QR signature flags | |
| +-----------------------+ | +------------------------+ |
| | Bailiwick index |--+ | Query OPCODE (q) | |
| +-----------------------+ +------------------------+ |
| | Flags | | QR DNS flags | |
+-+-----------------------+ +------------------------+ |
| Extra query info (q) | | Query RCODE (q) | |
| +-----------------------+ +------------------------+ |
| | Question |--+---+ +--+-Query Class/Type (q) | |
| +-----------------------+ | | +------------------------+ |
| | Answer |--+ | | | Query QD count (q) | |
| +-----------------------+ | | | +------------------------+ |
| | Authority |--+ | | | Query AN count (q) | |
| +-----------------------+ | | | +------------------------+ |
| | Additional |--+ | | | Query NS count (q) | |
+-+-----------------------+ | | | +------------------------+ |
| Extra response info (r) | |-+ | | | Query EDNS version (q) | |
| +-----------------------+ | | | | +------------------------+ |
| | Answer |--+ | | | | EDNS UDP size (q) | |
| +-----------------------+ | | | | +------------------------+ |
| | Authority |--+ | | | | Query Opt RDATA (q) | |
| +-----------------------+ | | | | +------------------------+ |
| | Additional |--+ | | | | Response RCODE (r) | |
+-+-----------------------+ | | | +------------------------+ |
| | | |
| | | |
+ -----------------------------+ | +----------+ |
| | | |
| + -----------------------------+ | |
| | +---------------+ +----------+ | |
| +->| Question list |->| Question | | |
| | array | | array | | |
| +---------------+ +----------+--+ | |
| | Name |--+------)------------------+
| +-------------+ | | +------------+
| | Class/type |--)---+--+->| Class/Type |
| +-------------+ | | | array |
| | | +------------+--+
| | | | Class |
| +---------------+ +----------+ | | +---------------+
+--->| RR list array |->| RR array | | | | Type |
+---------+-----+ +----------+--+ | | +---------------+
| Name |--+ |
+-------------+ |
| Class/type |------+
+-------------+
Figure 2: The Query/Response data item and subsidiary tables.
In Figure 2 data items annotated (q) are only present when a query/
response has a query, and those annotated (r) are only present when a
query/response response is present.
A C-DNS file begins with a file header containing a File Type A C-DNS file begins with a file header containing a File Type
Identifier and a File Preamble. The File Preamble contains Identifier and a File Preamble. The File Preamble contains
information on the file Format Version and an array of Block information on the file Format Version and an array of Block
Parameters items (the contents of which include Collection and Parameters items (the contents of which include Collection and
Storage Parameters used for one or more blocks). Storage Parameters used for one or more blocks).
The file header is followed by a series of data Blocks. The file header is followed by a series of data Blocks.
A Block consists of a Block Preamble item, some Block Statistics for A Block consists of a Block Preamble item, some Block Statistics for
skipping to change at page 24, line 40 skipping to change at page 27, line 40
| | | | | | | | | |
| qr-transport-flags | O | U | Bit flags describing the transport | | qr-transport-flags | O | U | Bit flags describing the transport |
| | | | used to service the query. | | | | | used to service the query. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if | | | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6 | | | | | IPv6 |
| | | | Bit 1-4. Transport. 4 bit unsigned | | | | | Bit 1-4. Transport. 4 bit unsigned |
| | | | value where 0 = UDP, 1 = TCP, 2 = | | | | | value where 0 = UDP, 1 = TCP, 2 = |
| | | | TLS, 3 = DTLS. Values 4-15 are | | | | | TLS, 3 = DTLS. Values 4-15 are |
| | | | reserved for future use. | | | | | reserved for future use. |
| | | | Bit 5. 1 if trailing bytes in query | | | | | Bit 5. 1 if trailing bytes in query |
| | | | packet. See Section 10.2. | | | | | packet. See Section 11.2. |
| | | | | | | | | |
| qr-type | O | U | Type of Query/Response transaction. | | qr-type | O | U | Type of Query/Response transaction. |
| | | | 0 = Stub. A query from a stub | | | | | 0 = Stub. A query from a stub |
| | | | resolver. | | | | | resolver. |
| | | | 1 = Client. An incoming query to a | | | | | 1 = Client. An incoming query to a |
| | | | recursive resolver. | | | | | recursive resolver. |
| | | | 2 = Resolver. A query sent from a | | | | | 2 = Resolver. A query sent from a |
| | | | recursive resolver to an authorative | | | | | recursive resolver to an authorative |
| | | | resolver. | | | | | resolver. |
| | | | 3 = Authorative. A query to an | | | | | 3 = Authorative. A query to an |
skipping to change at page 33, line 5 skipping to change at page 36, line 5
| | | | See Section 7.5.3. | | | | | See Section 7.5.3. |
| | | | | | | | | |
| client-port | O | U | The client port. | | client-port | O | U | The client port. |
| | | | | | | | | |
| message-data-index | O | U | The index in the "malformed- | | message-data-index | O | U | The index in the "malformed- |
| | | | message-data" array of the message | | | | | message-data" array of the message |
| | | | data for this message. See Section | | | | | data for this message. See Section |
| | | | 7.5.3. | | | | | 7.5.3. |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
8. C-DNS to PCAP 8. Versioning
The C-DNS file preamble includes a file format version; a major and
minor version number are required fields. The document defines
version 1.0 of the C-DNS specification. This section describes the
intended use of these version numbers in future specifications.
It is noted that version 1.0 includes many optional fields and
therefore consumers of version 1.0 should be inherently robust to
parsing files with variable data content.
Within a major version, a new minor version MUST be a strict superset
of the previous minor version, with no semantic changes to existing
fields. New keys MAY be added to existing maps, and new maps MAY be
added. A consumer capable of reading a particular major.minor
version MUST also be capable of reading all previous minor versions
of the same major version. It SHOULD also be capable of parsing all
subsequent minor versions ignoring any keys or maps that it does not
recognise.
A new major version indicates changes to the format that are not
backwards compatible with previous major versions. A consumer
capable of only reading a particular major version (greater than 1)
is not required to and has no expectation to be capable of reading a
previous major version.
9. C-DNS to PCAP
It is possible to re-construct PCAP files from the C-DNS format in a It is possible to re-construct PCAP files from the C-DNS format in a
lossy fashion. Some of the issues with reconstructing both the DNS lossy fashion. Some of the issues with reconstructing both the DNS
payload and the full packet stream are outlined here. payload and the full packet stream are outlined here.
The reconstruction depends on whether or not all the optional The reconstruction depends on whether or not all the optional
sections of both the query and response were captured in the C-DNS sections of both the query and response were captured in the C-DNS
file. Clearly, if they were not all captured, the reconstruction file. Clearly, if they were not all captured, the reconstruction
will be imperfect. will be imperfect.
Even if all sections of the response were captured, one cannot Even if all sections of the response were captured, one cannot
reconstruct the DNS response payload exactly due to the fact that reconstruct the DNS response payload exactly due to the fact that
some DNS names in the message on the wire may have been compressed. some DNS names in the message on the wire may have been compressed.
Section 8.1 discusses this is more detail. Section 9.1 discusses this is more detail.
Some transport information is not captured in the C-DNS format. For Some transport information is not captured in the C-DNS format. For
example, the following aspects of the original packet stream cannot example, the following aspects of the original packet stream cannot
be re-constructed from the C-DNS format: be re-constructed from the C-DNS format:
o IP fragmentation o IP fragmentation
o TCP stream information: o TCP stream information:
* Multiple DNS messages may have been sent in a single TCP * Multiple DNS messages may have been sent in a single TCP
segment segment
* A DNS payload may have be split across multiple TCP segments * A DNS payload may have be split across multiple TCP segments
* Multiple DNS messages may have be sent on a single TCP session * Multiple DNS messages may have be sent on a single TCP session
o Malformed DNS messages if the wire format is not recorded o Malformed DNS messages if the wire format is not recorded
skipping to change at page 34, line 5 skipping to change at page 37, line 26
ICMP ICMP
Simple assumptions can be made on the reconstruction: fragmented and Simple assumptions can be made on the reconstruction: fragmented and
DNS-over-TCP messages can be reconstructed into single packets and a DNS-over-TCP messages can be reconstructed into single packets and a
single TCP session can be constructed for each TCP packet. single TCP session can be constructed for each TCP packet.
Additionally, if malformed messages and Non-DNS packets are captured Additionally, if malformed messages and Non-DNS packets are captured
separately, they can be merged with packet captures reconstructed separately, they can be merged with packet captures reconstructed
from C-DNS to produce a more complete packet stream. from C-DNS to produce a more complete packet stream.
8.1. Name compression 9.1. Name compression
All the names stored in the C-DNS format are full domain names; no All the names stored in the C-DNS format are full domain names; no
DNS style name compression is used on the individual names within the DNS style name compression is used on the individual names within the
format. Therefore when reconstructing a packet, name compression format. Therefore when reconstructing a packet, name compression
must be used in order to reproduce the on the wire representation of must be used in order to reproduce the on the wire representation of
the packet. the packet.
[RFC1035] name compression works by substituting trailing sections of [RFC1035] name compression works by substituting trailing sections of
a name with a reference back to the occurrence of those sections a name with a reference back to the occurrence of those sections
earlier in the message. Not all name server software uses the same earlier in the message. Not all name server software uses the same
skipping to change at page 34, line 42 skipping to change at page 38, line 15
algorithm in turn to see if it reproduces the original length, algorithm in turn to see if it reproduces the original length,
stopping at the first match. This would not guarantee the correct stopping at the first match. This would not guarantee the correct
algorithm has been used as it is possible to match the length whilst algorithm has been used as it is possible to match the length whilst
still not matching the on the wire bytes but, without further still not matching the on the wire bytes but, without further
information added to the C-DNS data, this is the best that can be information added to the C-DNS data, this is the best that can be
achieved. achieved.
Appendix B presents an example of two different compression Appendix B presents an example of two different compression
algorithms used by well-known name server software. algorithms used by well-known name server software.
9. Data collection 10. Data collection
This section describes a non-normative proposed algorithm for the This section describes a non-normative proposed algorithm for the
processing of a captured stream of DNS queries and responses and processing of a captured stream of DNS queries and responses and
production of a stream of query/response items, matching queries/ production of a stream of query/response items, matching queries/
responses where possible. responses where possible.
For the purposes of this discussion, it is assumed that the input has For the purposes of this discussion, it is assumed that the input has
been pre-processed such that: been pre-processed such that:
1. All IP fragmentation reassembly, TCP stream reassembly, and so 1. All IP fragmentation reassembly, TCP stream reassembly, and so
on, has already been performed. on, has already been performed.
2. Each message is associated with transport metadata required to 2. Each message is associated with transport metadata required to
generate the Primary ID (see Section 9.2.1). generate the Primary ID (see Section 10.2.1).
3. Each message has a well-formed DNS header of 12 bytes and (if 3. Each message has a well-formed DNS header of 12 bytes and (if
present) the first Question in the Question section can be parsed present) the first Question in the Question section can be parsed
to generate the Secondary ID (see below). As noted earlier, this to generate the Secondary ID (see below). As noted earlier, this
requirement can result in a malformed query being removed in the requirement can result in a malformed query being removed in the
pre-processing stage, but the correctly formed response with pre-processing stage, but the correctly formed response with
RCODE of FORMERR being present. RCODE of FORMERR being present.
DNS messages are processed in the order they are delivered to the DNS messages are processed in the order they are delivered to the
implementation. implementation.
skipping to change at page 35, line 39 skipping to change at page 39, line 11
choronological; a response can appear in the capture stream before choronological; a response can appear in the capture stream before
the query that provoked the response. For this discussion, this non- the query that provoked the response. For this discussion, this non-
chronological delivery is termed "skew". chronological delivery is termed "skew".
In the presence of skew, a response packets can arrive for matching In the presence of skew, a response packets can arrive for matching
before the corresponding query. To avoid generating false instances before the corresponding query. To avoid generating false instances
of responses without a matching query, and queries without a matching of responses without a matching query, and queries without a matching
response, the matching algorithm must take account of the possibility response, the matching algorithm must take account of the possibility
of skew. of skew.
9.1. Matching algorithm 10.1. Matching algorithm
A schematic representation of the algorithm for matching Q/R data A schematic representation of the algorithm for matching Q/R data
items is shown in the following diagram: items is shown in Figure 3. It takes individual DNS query or
response messages as input, and outputs matched Q/R items. The
numbers in the figure identify matching operations listed in Table 1.
Specific details of the algorithm, for example queues, timers and
identifiers, are given in the following sections.
Figure showing the Query/Response matching algorithm format (PNG) [5] .----------------------.
| Process next message |<------------------+
`----------------------' |
| |
+------------------------------+ |
| Generate message identifiers | |
+------------------------------+ |
| |
Response | Query |
+--------------< >---------------+ |
| | |
+--------------------+ +--------------------+ |
| Find earliest QR | | Create QR item [2] | |
| item in OFIFO [1] | +--------------------+ |
+--------------------+ | |
| +---------------+ |
Match | No match | Append new QR | |
+--------< >------+ | item to OFIFO | |
| | +---------------+ |
+-----------+ +--------+ | |
| Update QR | | Add to | +-------------------+ |
| item [3] | | RFIFO | | Find earliest QR | |
+-----------+ +--------+ | item in RFIFO [1] | |
| | +-------------------+ |
+-----------------+ | |
| | |
| +----------------+ Match | No match |
| | Remove R |-------< >-----+ |
| | from RFIFO [3] | | |
| +----------------+ | |
| | | |
+--------------+-----------------------+ |
| |
+----------------------------------------------+ |
| Update all timed out (QT) OFIFO QR items [4] | |
+----------------------------------------------+ |
| |
+--------------------------------+ |
| Remove all timed out (ST) R | |
| from RFIFO, create QR item [5] | |
+--------------------------------+ |
____________________|_______________________ |
/ / |
/ Remove all consecutive done entries from /-------+
/ front of OFIFO for further processing /
/____________________________________________/
Figure showing the Query/Response matching algorithm format (SVG) [6] Figure 3: Query/Response matching algorithm
Further details of the algorithm are given in the following sections. +-----+-------------------------------------------+
| Ref | Operation |
+-----+-------------------------------------------+
| [1] | Find earliest QR item in FIFO where: |
| | * QR.done = false |
| | * QR.Q.PrimaryID == R.PrimaryID |
| | and, if both QR.Q and R have SecondaryID: |
| | * QR.Q.SecondaryID == R.SecondaryID |
| | |
| [2] | Set: |
| | QR.Q := Q |
| | QR.R := nil |
| | QR.done := false |
| | |
| [3] | Set: |
| | QR.R := R |
| | QR.done := true |
| | |
| [4] | Set: |
| | QR.done := true |
| | |
| [5] | Set: |
| | QR.Q := nil |
| | QR.R := R |
| | QR.done := true |
+-----+-------------------------------------------+
9.2. Message identifiers Table 1: Operations used in the matching algorithm
9.2.1. Primary ID (required) 10.2. Message identifiers
10.2.1. Primary ID (required)
A Primary ID is constructed for each message. It is composed of the A Primary ID is constructed for each message. It is composed of the
following data: following data:
1. Source IP Address 1. Source IP Address
2. Destination IP Address 2. Destination IP Address
3. Source Port 3. Source Port
4. Destination Port 4. Destination Port
5. Transport 5. Transport
6. DNS Message ID 6. DNS Message ID
9.2.2. Secondary ID (optional) 10.2.2. Secondary ID (optional)
If present, the first Question in the Question section is used as a If present, the first Question in the Question section is used as a
secondary ID for each message. Note that there may be well formed secondary ID for each message. Note that there may be well formed
DNS queries that have a QDCOUNT of 0, and some responses may have a DNS queries that have a QDCOUNT of 0, and some responses may have a
QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP).
In this case the secondary ID is not used in matching. In this case the secondary ID is not used in matching.
9.3. Algorithm parameters 10.3. Algorithm parameters
1. Query timeout, QT. A query arrives with timestamp t1. If no 1. Query timeout, QT. A query arrives with timestamp t1. If no
response matching that query has arrived before other input response matching that query has arrived before other input
arrives timestamped later than (t1 + QT), a query/response item arrives timestamped later than (t1 + QT), a query/response item
containing only a query item is recorded. The query timeout containing only a query item is recorded. The query timeout
value is typically of the order of 5 seconds. value is typically of the order of 5 seconds.
2. Skew timeout, ST. A response arrives with timestamp t2. If a 2. Skew timeout, ST. A response arrives with timestamp t2. If a
response has not been matched by a query before input arrives response has not been matched by a query before input arrives
timestamped later than (t2 + ST), a query/response item timestamped later than (t2 + ST), a query/response item
containing only a response is recorded. The skew timeout value containing only a response is recorded. The skew timeout value
is typically a few microseconds. is typically a few microseconds.
9.4. Algorithm requirements 10.4. Algorithm requirements
The algorithm is designed to handle the following input data: The algorithm is designed to handle the following input data:
1. Multiple queries with the same Primary ID (but different 1. Multiple queries with the same Primary ID (but different
Secondary ID) arriving before any responses for these queries are Secondary ID) arriving before any responses for these queries are
seen. seen.
2. Multiple queries with the same Primary and Secondary ID arriving 2. Multiple queries with the same Primary and Secondary ID arriving
before any responses for these queries are seen. before any responses for these queries are seen.
3. Queries for which no later response can be found within the 3. Queries for which no later response can be found within the
specified timeout. specified timeout.
4. Responses for which no previous query can be found within the 4. Responses for which no previous query can be found within the
specified timeout. specified timeout.
9.5. Algorithm limitations 10.5. Algorithm limitations
For cases 1 and 2 listed in the above requirements, it is not For cases 1 and 2 listed in the above requirements, it is not
possible to unambiguously match queries with responses. This possible to unambiguously match queries with responses. This
algorithm chooses to match to the earliest query with the correct algorithm chooses to match to the earliest query with the correct
Primary and Secondary ID. Primary and Secondary ID.
9.6. Workspace 10.6. Workspace
A FIFO structure is used to hold the Q/R data items during The algorithm employs two FIFO queues:
processing. A secondary responses FIFO holds responses awaiting
matching queries.
9.7. Output o OFIFO, an output FIFO containing Q/R items in chronological order,
o RFIFO, a FIFO holding responses without a matching query in order
of arrival.
10.7. Output
The output is a list of Q/R data items. Both the Query and Response The output is a list of Q/R data items. Both the Query and Response
elements are optional in these items, therefore Q/R data items have elements are optional in these items, therefore Q/R data items have
one of three types of content: one of three types of content:
1. A matched pair of query and response messages 1. A matched pair of query and response messages
2. A query message with no response 2. A query message with no response
3. A response message with no query 3. A response message with no query
The timestamp of a list item is that of the query for cases 1 and 2 The timestamp of a list item is that of the query for cases 1 and 2
and that of the response for case 3. and that of the response for case 3.
9.8. Post processing 10.8. Post processing
When ending capture, all items in the responses FIFO are timed out When ending capture, all items in the responses FIFO are timed out
immediately, generating response-only entries to the Q/R data item immediately, generating response-only entries to the Q/R data item
FIFO. These and all other remaining entries in the Q/R data item FIFO. These and all other remaining entries in the Q/R data item
FIFO should be treated as timed out queries. FIFO should be treated as timed out queries.
10. Implementation guidance 11. Implementation guidance
Whilst this document makes no specific recommendations with respect Whilst this document makes no specific recommendations with respect
to Canonical CBOR (see Section 3.9 of [RFC7049]) the following to Canonical CBOR (see Section 3.9 of [RFC7049]) the following
guidance may be of use to implementors. guidance may be of use to implementors.
Adherence to the first two rules given in Section 3.9 of [RFC7049] Adherence to the first two rules given in Section 3.9 of [RFC7049]
will minimise file sizes. will minimise file sizes.
Adherence to the last two rules given in Section 3.9 of [RFC7049] for Adherence to the last two rules given in Section 3.9 of [RFC7049] for
all maps and arrays would unacceptably constrain implementations, for all maps and arrays would unacceptably constrain implementations, for
example, in the use case of real-time data collection in constrained example, in the use case of real-time data collection in constrained
environments. environments.
10.1. Optional data 11.1. Optional data
When decoding C-DNS data some of the items required for a particular When decoding C-DNS data some of the items required for a particular
function that the consumer wishes to perform may be missing. function that the consumer wishes to perform may be missing.
Consumers should consider providing configurable default values to be Consumers should consider providing configurable default values to be
used in place of the missing values in their output. used in place of the missing values in their output.
10.2. Trailing bytes 11.2. Trailing bytes
A DNS query message in a UDP or TCP payload can be followed by some A DNS query message in a UDP or TCP payload can be followed by some
additional (spurious) bytes, which are not stored in C-DNS. additional (spurious) bytes, which are not stored in C-DNS.
When DNS traffic is sent over TCP, each message is prefixed with a When DNS traffic is sent over TCP, each message is prefixed with a
two byte length field which gives the message length, excluding the two byte length field which gives the message length, excluding the
two byte length field. In this context, trailing bytes can occur in two byte length field. In this context, trailing bytes can occur in
two circumstances with different results: two circumstances with different results:
1. The number of bytes consumed by fully parsing the message is less 1. The number of bytes consumed by fully parsing the message is less
skipping to change at page 39, line 5 skipping to change at page 44, line 39
2. There are surplus bytes between the end of a well-formed message 2. There are surplus bytes between the end of a well-formed message
and the start of the length field for the next message. In this and the start of the length field for the next message. In this
case the first of the surplus bytes will be processed as the case the first of the surplus bytes will be processed as the
first byte of the next length field, and parsing will proceed first byte of the next length field, and parsing will proceed
from there, almost certainly leading to the next and any from there, almost certainly leading to the next and any
subsequent messages in the packet being considered malformed. subsequent messages in the packet being considered malformed.
This will not generate a trailing bytes record for the processed This will not generate a trailing bytes record for the processed
well-formed message. well-formed message.
10.3. Limiting collection of RDATA 11.3. Limiting collection of RDATA
Implementations should consider providing a configurable maximum Implementations should consider providing a configurable maximum
RDATA size for capture, for example, to avoid memory issues when RDATA size for capture, for example, to avoid memory issues when
confronted with large XFR records. confronted with large XFR records.
11. Implementation status 12. Implementation status
[Note to RFC Editor: please remove this section and reference to [Note to RFC Editor: please remove this section and reference to
[RFC7942] prior to publication.] [RFC7942] prior to publication.]
This section records the status of known implementations of the This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942]. Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to The description of implementations in this section is intended to
assist the IETF in its decision processes in progressing drafts to assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation RFCs. Please note that the listing of any individual implementation
here does not imply endorsement by the IETF. Furthermore, no effort here does not imply endorsement by the IETF. Furthermore, no effort
has been spent to verify the information presented here that was has been spent to verify the information presented here that was
supplied by IETF contributors. This is not intended as, and must not supplied by IETF contributors. This is not intended as, and must not
be construed to be, a catalog of available implementations or their be construed to be, a catalog of available implementations or their
features. Readers are advised to note that other implementations may features. Readers are advised to note that other implementations may
exist. exist.
skipping to change at page 39, line 36 skipping to change at page 45, line 22
features. Readers are advised to note that other implementations may features. Readers are advised to note that other implementations may
exist. exist.
According to [RFC7942], "this will allow reviewers and working groups According to [RFC7942], "this will allow reviewers and working groups
to assign due consideration to documents that have the benefit of to assign due consideration to documents that have the benefit of
running code, which may serve as evidence of valuable experimentation running code, which may serve as evidence of valuable experimentation
and feedback that have made the implemented protocols more mature. and feedback that have made the implemented protocols more mature.
It is up to the individual working groups to use this information as It is up to the individual working groups to use this information as
they see fit". they see fit".
11.1. DNS-STATS Compactor 12.1. DNS-STATS Compactor
ICANN/Sinodun IT have developed an open source implementation called ICANN/Sinodun IT have developed an open source implementation called
DNS-STATS Compactor. The Compactor is a suite of tools which can DNS-STATS Compactor. The Compactor is a suite of tools which can
capture DNS traffic (from either a network interface or a PCAP file) capture DNS traffic (from either a network interface or a PCAP file)
and store it in the Compacted-DNS (C-DNS) file format. PCAP files and store it in the Compacted-DNS (C-DNS) file format. PCAP files
for the captured traffic can also be reconstructed. See Compactor for the captured traffic can also be reconstructed. See Compactor
[7]. [1].
This implementation: This implementation:
o covers the whole of the specification described in the -03 draft o covers the whole of the specification described in the -03 draft
with the exception of support for malformed messages and pico with the exception of support for malformed messages and pico
second time resolution. (Note: this implementation does allow second time resolution. (Note: this implementation does allow
malformed messages to be recorded separately in a PCAP file). malformed messages to be recorded separately in a PCAP file).
o is released under the Mozilla Public License Version 2.0. o is released under the Mozilla Public License Version 2.0.
o has a users mailing list available, see dns-stats-users [8]. o has a users mailing list available, see dns-stats-users [2].
There is also some discussion of issues encountered during There is also some discussion of issues encountered during
development available at Compressing Pcap Files [9] and Packet development available at Compressing Pcap Files [3] and Packet
Capture [10]. Capture [4].
This information was last updated on 3rd of May 2018. This information was last updated on 3rd of May 2018.
12. IANA considerations 13. IANA considerations
None None
13. Security considerations 14. Security considerations
Any control interface MUST perform authentication and encryption. Any control interface MUST perform authentication and encryption.
Any data upload MUST be authenticated and encrypted. Any data upload MUST be authenticated and encrypted.
14. Acknowledgements 15. Acknowledgements
The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for
many useful discussions on binary formats, compression and packet many useful discussions on binary formats, compression and packet
matching. Also Jan Vcelak and Wouter Wijngaards for discussions on matching. Also Jan Vcelak and Wouter Wijngaards for discussions on
name compression and Paul Hoffman for a detailed review of the name compression and Paul Hoffman for a detailed review of the
document and the C-DNS CDDL. document and the C-DNS CDDL.
Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson, Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson,
Stephane Bortzmeyer and many other members of DNSOP for review. Stephane Bortzmeyer and many other members of DNSOP for review.
Also, Miek Gieben for mmark [11] Also, Miek Gieben for mmark [5]
15. Changelog 16. Changelog
draft-ietf-dnsop-dns-capture-format-08
o Convert diagrams to ASCII
o Describe versioning
o Fix unused group warning in CDDL
draft-ietf-dnsop-dns-capture-format-07 draft-ietf-dnsop-dns-capture-format-07
o Resolve outstanding questions and TODOs o Resolve outstanding questions and TODOs
o Make RR RDATA optional o Make RR RDATA optional
o Update matching diagram and explain skew o Update matching diagram and explain skew
o Add count of discarded messages to block statistics o Add count of discarded messages to block statistics
skipping to change at page 42, line 4 skipping to change at page 47, line 47
o Clarify that map keys are unsigned integers o Clarify that map keys are unsigned integers
o Add Type to Class/Type table o Add Type to Class/Type table
o Clarify storage format in section 7.12 o Clarify storage format in section 7.12
draft-ietf-dnsop-dns-capture-format-03 draft-ietf-dnsop-dns-capture-format-03
o Added an Implementation Status section o Added an Implementation Status section
o Update qr_data_format.png to match CDDL
draft-ietf-dnsop-dns-capture-format-02
o Update qr_data_format.png to match CDDL
o Editorial clarifications and improvements o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-01 draft-ietf-dnsop-dns-capture-format-01
o Many editorial improvements by Paul Hoffman o Many editorial improvements by Paul Hoffman
o Included discussion of malformed message handling o Included discussion of malformed message handling
o Improved Appendix C on Comparison of Binary Formats o Improved Appendix C on Comparison of Binary Formats
skipping to change at page 43, line 4 skipping to change at page 48, line 47
o Added a TODO: Add extend to support pico/nano. Also do this for o Added a TODO: Add extend to support pico/nano. Also do this for
Time offset and Response delay Time offset and Response delay
o Added a TODO: Need to develop optional representation of malformed o Added a TODO: Need to develop optional representation of malformed
messages within C-DNS and what this means for packet matching. messages within C-DNS and what this means for packet matching.
This may influence which fields are optional in the rest of the This may influence which fields are optional in the rest of the
representation. representation.
o Added section on design goals to Introduction o Added section on design goals to Introduction
o Added a TODO: Can Class be optimised? Should a class of IN be o Added a TODO: Can Class be optimised? Should a class of IN be
inferred if not present? inferred if not present?
draft-dickinson-dnsop-dns-capture-format-00 draft-dickinson-dnsop-dns-capture-format-00
o Initial commit o Initial commit
16. References 17. References
16.1. Normative References 17.1. Normative References
[RFC1035] Mockapetris, P., "Domain names - implementation and [RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>. November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, <https://www.rfc- DOI 10.17487/RFC2119, March 1997, <https://www.rfc-
editor.org/info/rfc2119>. editor.org/info/rfc2119>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <https://www.rfc-editor.org/info/rfc7049>. October 2013, <https://www.rfc-editor.org/info/rfc7049>.
16.2. Informative References 17.2. Informative References
[ditl] DNS-OARC, "DITL", 2016, <https://www.dns- [ditl] DNS-OARC, "DITL", 2016, <https://www.dns-
oarc.net/oarc/data/ditl>. oarc.net/oarc/data/ditl>.
[dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/ [dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
dnscap>. dnscap>.
[dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>.
[dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
<https://www.dns-oarc.net/tools/dsc>. <https://www.dns-oarc.net/tools/dsc>.
[I-D.daley-dnsxml] [I-D.daley-dnsxml]
Daley, J., Morris, S., and J. Dickinson, "dnsxml - A Daley, J., Morris, S., and J. Dickinson, "dnsxml - A
standard XML representation of DNS data", draft-daley- standard XML representation of DNS data", draft-daley-
dnsxml-00 (work in progress), July 2013. dnsxml-00 (work in progress), July 2013.
[I-D.hoffman-dns-in-json] [I-D.hoffman-dns-in-json]
Hoffman, P., "Representing DNS Messages in JSON", draft- Hoffman, P., "Representing DNS Messages in JSON", draft-
hoffman-dns-in-json-14 (work in progress), April 2018. hoffman-dns-in-json-16 (work in progress), May 2018.
[I-D.ietf-cbor-cddl] [I-D.ietf-cbor-cddl]
Birkholz, H., Vigano, C., and C. Bormann, "Concise data Birkholz, H., Vigano, C., and C. Bormann, "Concise data
definition language (CDDL): a notational convention to definition language (CDDL): a notational convention to
express CBOR data structures", draft-ietf-cbor-cddl-02 express CBOR data structures", draft-ietf-cbor-cddl-03
(work in progress), February 2018. (work in progress), July 2018.
[packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
2014, <https://github.com/dotse/PacketQ>. 2014, <https://github.com/dotse/PacketQ>.
[pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.
[pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
Harris, "pcap-ng", 2016, <https://github.com/pcapng/ Harris, "pcap-ng", 2016, <https://github.com/pcapng/
pcapng>. pcapng>.
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205, Code: The Implementation Status Section", BCP 205,
RFC 7942, DOI 10.17487/RFC7942, July 2016, RFC 7942, DOI 10.17487/RFC7942, July 2016,
<https://www.rfc-editor.org/info/rfc7942>. <https://www.rfc-editor.org/info/rfc7942>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259, Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017, <https://www.rfc- DOI 10.17487/RFC8259, December 2017, <https://www.rfc-
editor.org/info/rfc8259>. editor.org/info/rfc8259>.
16.3. URIs 17.3. URIs
[1] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/cdns_format.png
[2] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/cdns_format.svg
[3] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/qr_data_format.png
[4] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/qr_data_format.svg
[5] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/packet_matching.png
[6] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-07/packet_matching.svg
[7] https://github.com/dns-stats/compactor/wiki [1] https://github.com/dns-stats/compactor/wiki
[8] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users [2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
[9] https://www.sinodun.com/2017/06/compressing-pcap-files/ [3] https://www.sinodun.com/2017/06/compressing-pcap-files/
[10] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu- [4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu-
trusty-packet-capture-woes/ trusty-packet-capture-woes/
[11] https://github.com/miekg/mmark [5] https://github.com/miekg/mmark
[12] https://www.nlnetlabs.nl/projects/nsd/ [6] https://www.nlnetlabs.nl/projects/nsd/
[13] https://www.knot-dns.cz/ [7] https://www.knot-dns.cz/
[14] https://avro.apache.org/ [8] https://avro.apache.org/
[15] https://developers.google.com/protocol-buffers/ [9] https://developers.google.com/protocol-buffers/
[16] http://cbor.io [10] http://cbor.io
[17] https://github.com/kubo/snzip [11] https://github.com/kubo/snzip
[18] http://google.github.io/snappy/ [12] http://google.github.io/snappy/
[19] http://lz4.github.io/lz4/ [13] http://lz4.github.io/lz4/
[20] http://www.gzip.org/ [14] http://www.gzip.org/
[21] http://facebook.github.io/zstd/ [15] http://facebook.github.io/zstd/
[22] http://tukaani.org/xz/ [16] http://tukaani.org/xz/
[23] https://github.com/dns-stats/draft-dns-capture- [17] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.png format/blob/master/file-size-versus-block-size.png
[24] https://github.com/dns-stats/draft-dns-capture- [18] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.svg format/blob/master/file-size-versus-block-size.svg
Appendix A. CDDL Appendix A. CDDL
This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for
C-DNS.
CDDL does not permit a range of allowed values to be specified for a
bitfield. Where necessary, those values are given as a CDDL group,
but the group definition is commented out to prevent CDDL tooling
from warning that the group is unused.
; CDDL specification of the file format for C-DNS, ; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS messages and ; which describes a collection of DNS messages and
; traffic meta-data. ; traffic meta-data.
; ;
; The overall structure of a file. ; The overall structure of a file.
; ;
File = [ File = [
file-type-id : tstr .regexp "C-DNS", file-type-id : tstr .regexp "C-DNS",
file-preamble : FilePreamble, file-preamble : FilePreamble,
skipping to change at page 51, line 38 skipping to change at page 57, line 22
query-classtype-index = 8 query-classtype-index = 8
query-qd-count = 9 query-qd-count = 9
query-an-count = 10 query-an-count = 10
query-ns-count = 12 query-ns-count = 12
query-ar-count = 12 query-ar-count = 12
edns-version = 13 edns-version = 13
udp-buf-size = 14 udp-buf-size = 14
opt-rdata-index = 15 opt-rdata-index = 15
response-rcode = 16 response-rcode = 16
Transport = &( ; Transport gives the values that may appear in bits 1..4 of
udp : 0, ; TransportFlags. There is currently no way to express this in
tcp : 1, ; CDDL, so Transport is unused. To avoid confusion when used
tls : 2, ; with CDDL tools, it is commented out.
dtls : 3, ;
) ; Transport = &(
; udp : 0,
; tcp : 1,
; tls : 2,
; dtls : 3,
; )
TransportFlagValues = &( TransportFlagValues = &(
ip-version : 0, ; 0=IPv4, 1=IPv6 ip-version : 0, ; 0=IPv4, 1=IPv6
; Transport value bits 1-4
) / (1..4) ) / (1..4)
TransportFlags = uint .bits TransportFlagValues TransportFlags = uint .bits TransportFlagValues
QueryResponseTransportFlagValues = &( QueryResponseTransportFlagValues = &(
query-trailingdata : 5, query-trailingdata : 5,
) / TransportFlagValues ) / TransportFlagValues
QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues
QueryResponseType = &( QueryResponseType = &(
stub : 0, stub : 0,
skipping to change at page 56, line 38 skipping to change at page 62, line 27
| 3 | www.bar.com | www | 2 | | 3 | www.bar.com | www | 2 |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
As an optimization, if a name is already perfectly compressed (in As an optimization, if a name is already perfectly compressed (in
other words, the uncompressed part of the name is empty), then no other words, the uncompressed part of the name is empty), then no
further names will be considered for compression. further names will be considered for compression.
B.1. NSD compression algorithm B.1. NSD compression algorithm
Using the above basic algorithm the packet lengths of responses Using the above basic algorithm the packet lengths of responses
generated by NSD [12] can be matched almost exactly. At the time of generated by NSD [6] can be matched almost exactly. At the time of
writing, a tiny number (<.01%) of the reconstructed packets had writing, a tiny number (<.01%) of the reconstructed packets had
incorrect lengths. incorrect lengths.
B.2. Knot Authoritative compression algorithm B.2. Knot Authoritative compression algorithm
The Knot Authoritative [13] name server uses different compression The Knot Authoritative [7] name server uses different compression
behavior, which is the result of internal optimization designed to behavior, which is the result of internal optimization designed to
balance runtime speed with compression size gains. In brief, and balance runtime speed with compression size gains. In brief, and
omitting complications, Knot Authoritative will only consider the omitting complications, Knot Authoritative will only consider the
QNAME and names in the immediately preceding RR section in an RRSET QNAME and names in the immediately preceding RR section in an RRSET
as compression targets. as compression targets.
A set of smart heuristics as described below can be implemented to A set of smart heuristics as described below can be implemented to
mimic this and while not perfect it produces output nearly, but not mimic this and while not perfect it produces output nearly, but not
quite, as good a match as with NSD. The heuristics are: quite, as good a match as with NSD. The heuristics are:
skipping to change at page 57, line 41 skipping to change at page 63, line 27
In sample traffic collected on a root name server around 2-4% of In sample traffic collected on a root name server around 2-4% of
responses generated by Knot had different packet lengths to those responses generated by Knot had different packet lengths to those
produced by NSD. produced by NSD.
Appendix C. Comparison of Binary Formats Appendix C. Comparison of Binary Formats
Several binary serialisation formats were considered, and for Several binary serialisation formats were considered, and for
completeness were also compared to JSON. completeness were also compared to JSON.
o Apache Avro [14]. Data is stored according to a pre-defined o Apache Avro [8]. Data is stored according to a pre-defined
schema. The schema itself is always included in the data file. schema. The schema itself is always included in the data file.
Data can therefore be stored untagged, for a smaller serialisation Data can therefore be stored untagged, for a smaller serialisation
size, and be written and read by an Avro library. size, and be written and read by an Avro library.
* At the time of writing, Avro libraries are available for C, * At the time of writing, Avro libraries are available for C,
C++, C#, Java, Python, Ruby and PHP. Optionally tools are C++, C#, Java, Python, Ruby and PHP. Optionally tools are
available for C++, Java and C# to generate code for encoding available for C++, Java and C# to generate code for encoding
and decoding. and decoding.
o Google Protocol Buffers [15]. Data is stored according to a pre- o Google Protocol Buffers [9]. Data is stored according to a pre-
defined schema. The schema is used by a generator to generate defined schema. The schema is used by a generator to generate
code for encoding and decoding the data. Data can therefore be code for encoding and decoding the data. Data can therefore be
stored untagged, for a smaller serialisation size. The schema is stored untagged, for a smaller serialisation size. The schema is
not stored with the data, so unlike Avro cannot be read with a not stored with the data, so unlike Avro cannot be read with a
generic library. generic library.
* Code must be generated for a particular data schema to to read * Code must be generated for a particular data schema to to read
and write data using that schema. At the time of writing, the and write data using that schema. At the time of writing, the
Google code generator can currently generate code for encoding Google code generator can currently generate code for encoding
and decoding a schema for C++, Go, Java, Python, Ruby, C#, and decoding a schema for C++, Go, Java, Python, Ruby, C#,
Objective-C, Javascript and PHP. Objective-C, Javascript and PHP.
o CBOR [16]. Defined in [RFC7049], this serialisation format is o CBOR [10]. Defined in [RFC7049], this serialisation format is
comparable to JSON but with a binary representation. It does not comparable to JSON but with a binary representation. It does not
use a pre-defined schema, so data is always stored tagged. use a pre-defined schema, so data is always stored tagged.
However, CBOR data schemas can be described using CDDL However, CBOR data schemas can be described using CDDL
[I-D.ietf-cbor-cddl] and tools exist to verify data files conform [I-D.ietf-cbor-cddl] and tools exist to verify data files conform
to the schema. to the schema.
* CBOR is a simple format, and simple to implement. At the time * CBOR is a simple format, and simple to implement. At the time
of writing, the CBOR website lists implementations for 16 of writing, the CBOR website lists implementations for 16
languages. languages.
Avro and Protocol Buffers both allow storage of untagged data, but Avro and Protocol Buffers both allow storage of untagged data, but
because they rely on the data schema for this, their implementation because they rely on the data schema for this, their implementation
skipping to change at page 58, line 41 skipping to change at page 64, line 28
A test program was written which reads input from a PCAP file and A test program was written which reads input from a PCAP file and
writes output using one of two basic structures; either a simple writes output using one of two basic structures; either a simple
structure, where each query/response pair is represented in a single structure, where each query/response pair is represented in a single
record entry, or the C-DNS block structure. record entry, or the C-DNS block structure.
The resulting output files were then compressed using a variety of The resulting output files were then compressed using a variety of
common general-purpose lossless compression tools to explore the common general-purpose lossless compression tools to explore the
compressibility of the formats. The compression tools employed were: compressibility of the formats. The compression tools employed were:
o snzip [17]. A command line compression tool based on the Google o snzip [11]. A command line compression tool based on the Google
Snappy [18] library. Snappy [12] library.
o lz4 [19]. The command line compression tool from the reference C o lz4 [13]. The command line compression tool from the reference C
LZ4 implementation. LZ4 implementation.
o gzip [20]. The ubiquitous GNU zip tool. o gzip [14]. The ubiquitous GNU zip tool.
o zstd [21]. Compression using the Zstandard algorithm. o zstd [15]. Compression using the Zstandard algorithm.
o xz [22]. A popular compression tool noted for high compression. o xz [16]. A popular compression tool noted for high compression.
In all cases the compression tools were run using their default In all cases the compression tools were run using their default
settings. settings.
Note that this draft does not mandate the use of compression, nor any Note that this draft does not mandate the use of compression, nor any
particular compression scheme, but it anticipates that in practice particular compression scheme, but it anticipates that in practice
output data will be subject to general-purpose compression, and so output data will be subject to general-purpose compression, and so
this should be taken into consideration. this should be taken into consideration.
"test.pcap", a 662Mb capture of sample data from a root instance was "test.pcap", a 662Mb capture of sample data from a root instance was
skipping to change at page 62, line 45 skipping to change at page 68, line 38
| 320000 | 54.41 | 799.20 | 13.97 | | 320000 | 54.41 | 799.20 | 13.97 |
+------------+-----------+--------+-----------+ +------------+-----------+--------+-----------+
Increasing block size, therefore, tends to increase maximum RSS a Increasing block size, therefore, tends to increase maximum RSS a
little, with no significant effect (if anything a small reduction) on little, with no significant effect (if anything a small reduction) on
CPU consumption. CPU consumption.
The following figure plots the effect of increasing block size on The following figure plots the effect of increasing block size on
output file size for different compressions. output file size for different compressions.
Figure showing effect of block size on file size (PNG) [23] Figure showing effect of block size on file size (PNG) [17]
Figure showing effect of block size on file size (SVG) [24] Figure showing effect of block size on file size (SVG) [18]
From the above, there is obviously scope for tuning the default block From the above, there is obviously scope for tuning the default block
size to the compression being employed, traffic characteristics, size to the compression being employed, traffic characteristics,
frequency of output file rollover etc. Using a strong compression, frequency of output file rollover etc. Using a strong compression,
block sizes over 10,000 query/response pairs would seem to offer block sizes over 10,000 query/response pairs would seem to offer
limited improvements. limited improvements.
Authors' Addresses Authors' Addresses
John Dickinson John Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom United Kingdom
Email: jad@sinodun.com Email: jad@sinodun.com
Jim Hague Jim Hague
 End of changes. 93 change blocks. 
180 lines changed or deleted 409 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/