draft-ietf-dnsop-dns-capture-format-08.txt   draft-ietf-dnsop-dns-capture-format-09.txt 
dnsop J. Dickinson dnsop J. Dickinson
Internet-Draft J. Hague Internet-Draft J. Hague
Intended status: Standards Track S. Dickinson Intended status: Standards Track S. Dickinson
Expires: February 11, 2019 Sinodun IT Expires: June 3, 2019 Sinodun IT
T. Manderson T. Manderson
J. Bond J. Bond
ICANN ICANN
August 10, 2018 November 30, 2018
C-DNS: A DNS Packet Capture Format C-DNS: A DNS Packet Capture Format
draft-ietf-dnsop-dns-capture-format-08 draft-ietf-dnsop-dns-capture-format-09
Abstract Abstract
This document describes a data representation for collections of DNS This document describes a data representation for collections of DNS
messages. The format is designed for efficient storage and messages. The format is designed for efficient storage and
transmission of large packet captures of DNS traffic; it attempts to transmission of large packet captures of DNS traffic; it attempts to
minimize the size of such packet capture files but retain the full minimize the size of such packet capture files but retain the full
DNS message contents along with the most useful transport metadata. DNS message contents along with the most useful transport metadata.
It is intended to assist with the development of DNS traffic It is intended to assist with the development of DNS traffic
monitoring applications. monitoring applications.
skipping to change at page 1, line 40 skipping to change at page 1, line 40
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 11, 2019. This Internet-Draft will expire on June 3, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Data collection use cases . . . . . . . . . . . . . . . . . . 5 3. Data collection use cases . . . . . . . . . . . . . . . . . . 5
4. Design considerations . . . . . . . . . . . . . . . . . . . . 7 4. Design considerations . . . . . . . . . . . . . . . . . . . . 7
5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 9
6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9 6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9
6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13
6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13
6.2.1. Optional data items . . . . . . . . . . . . . . . . . 13 6.2.1. Optional data items . . . . . . . . . . . . . . . . . 14
6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 14 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 15
6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15 6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15
6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 15 6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 16
7. C-DNS format detailed description . . . . . . . . . . . . . . 15 7. C-DNS format detailed description . . . . . . . . . . . . . . 16
7.1. Map quantities and indexes . . . . . . . . . . . . . . . 15 7.1. Map quantities and indexes . . . . . . . . . . . . . . . 16
7.2. Tabular representation . . . . . . . . . . . . . . . . . 16 7.2. Tabular representation . . . . . . . . . . . . . . . . . 17
7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 17 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 18
7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 17 7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 18
7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 18 7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 19
7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 21 7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 22
7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 22 7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 24
7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 23 7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 24
7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 24 7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 25
7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 25 7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 26
7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 31 7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 32
7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 33 7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 34
7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 33 7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 34
7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 34 7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 35
7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 35 7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 36
8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 36 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 37
9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 36 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 37
9.1. Name compression . . . . . . . . . . . . . . . . . . . . 37 9.1. Name compression . . . . . . . . . . . . . . . . . . . . 38
10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 38 10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 39
10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 39 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 40
10.2. Message identifiers . . . . . . . . . . . . . . . . . . 41 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 42
10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 41 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 42
10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 42 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 43
10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 42 10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 43
10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 42 10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 43
10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 42 10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 43
10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 43 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 44
10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 43 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 44
10.8. Post processing . . . . . . . . . . . . . . . . . . . . 43 10.8. Post processing . . . . . . . . . . . . . . . . . . . . 44
11. Implementation guidance . . . . . . . . . . . . . . . . . . . 43 11. Implementation guidance . . . . . . . . . . . . . . . . . . . 44
11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 44 11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 45
11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 44 11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 45
11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 44 11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 45
12. Implementation status . . . . . . . . . . . . . . . . . . . . 44 11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 45
12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 45 12. Implementation status . . . . . . . . . . . . . . . . . . . . 46
13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 45 12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 46
14. Security considerations . . . . . . . . . . . . . . . . . . . 46 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 47
15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 14. Security considerations . . . . . . . . . . . . . . . . . . . 47
16. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 46 15. Privacy considerations . . . . . . . . . . . . . . . . . . . 47
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 49 16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 48
17.1. Normative References . . . . . . . . . . . . . . . . . . 49 17. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 48
17.2. Informative References . . . . . . . . . . . . . . . . . 49 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 51
17.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 50 18.1. Normative References . . . . . . . . . . . . . . . . . . 51
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 51 18.2. Informative References . . . . . . . . . . . . . . . . . 52
Appendix B. DNS Name compression example . . . . . . . . . . . . 61 18.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 54
B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 62 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 55
B.2. Knot Authoritative compression algorithm . . . . . . . . 62 Appendix B. DNS Name compression example . . . . . . . . . . . . 65
B.3. Observed differences . . . . . . . . . . . . . . . . . . 63 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 66
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 63 B.2. Knot Authoritative compression algorithm . . . . . . . . 67
C.1. Comparison with full PCAP files . . . . . . . . . . . . . 66 B.3. Observed differences . . . . . . . . . . . . . . . . . . 67
C.2. Simple versus block coding . . . . . . . . . . . . . . . 66 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 67
C.3. Binary versus text formats . . . . . . . . . . . . . . . 67 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 70
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 67 C.2. Simple versus block coding . . . . . . . . . . . . . . . 71
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 67 C.3. Binary versus text formats . . . . . . . . . . . . . . . 71
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 68 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 71
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 68 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 72
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 72
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 73
1. Introduction 1. Introduction
There has long been a need to collect DNS queries and responses on There has long been a need for server operators to collect DNS
authoritative and recursive name servers for monitoring and analysis. queries and responses on authoritative and recursive name servers for
This data is used in a number of ways including traffic monitoring, monitoring and analysis. This data is used in a number of ways
analyzing network attacks and "day in the life" (DITL) [ditl] including traffic monitoring, analyzing network attacks and "day in
analysis. the life" (DITL) [ditl] analysis.
A wide variety of tools already exist that facilitate the collection A wide variety of tools already exist that facilitate the collection
of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap
[dnscap] and dnstap [dnstap]. However, there is no standard exchange [dnscap] and dnstap [dnstap]. However, there is no standard exchange
format for large DNS packet captures. The PCAP [pcap] or PCAP-NG format for large DNS packet captures. The PCAP [pcap] or PCAP-NG
[pcapng] formats are typically used in practice for packet captures, [pcapng] formats are typically used in practice for packet captures,
but these file formats can contain a great deal of additional but these file formats can contain a great deal of additional
information that is not directly pertinent to DNS traffic analysis information that is not directly pertinent to DNS traffic analysis
and thus unnecessarily increases the capture file size. and thus unnecessarily increases the capture file size. Additionally
these tools and formats typically have no filter mechanism to
selectively record only certain fields at capture time, requiring
post-processing for anonymization or pseudonymization of data to
protect user privacy.
There has also been work on using text based formats to describe DNS There has also been work on using text based formats to describe DNS
packets such as [I-D.daley-dnsxml], [I-D.hoffman-dns-in-json], but packets such as [I-D.daley-dnsxml], [RFC8427], but these are largely
these are largely aimed at producing convenient representations of aimed at producing convenient representations of single messages.
single messages.
Many DNS operators may receive hundreds of thousands of queries per Many DNS operators may receive hundreds of thousands of queries per
second on a single name server instance so a mechanism to minimize second on a single name server instance so a mechanism to minimize
the storage size (and therefore upload overhead) of the data the storage and transmission size (and therefore upload overhead) of
collected is highly desirable. the data collected is highly desirable.
The format described in this document, C-DNS (Compacted-DNS), The format described in this document, C-DNS (Compacted-DNS),
focusses on the problem of capturing and storing large packet capture focusses on the problem of capturing and storing large packet capture
files of DNS traffic with the following goals in mind: files of DNS traffic with the following goals in mind:
o Minimize the file size for storage and transmission. o Minimize the file size for storage and transmission.
o Minimize the overhead of producing the packet capture file and the o Minimize the overhead of producing the packet capture file and the
cost of any further (general purpose) compression of the file. cost of any further (general purpose) compression of the file.
skipping to change at page 4, line 49 skipping to change at page 5, line 8
messages, see Section 7. messages, see Section 7.
o Notes on converting C-DNS data to PCAP format, see Section 9. o Notes on converting C-DNS data to PCAP format, see Section 9.
o Some high level implementation considerations for applications o Some high level implementation considerations for applications
designed to produce C-DNS, see Section 10. designed to produce C-DNS, see Section 10.
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
document are to be interpreted as described in [RFC2119]. "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
"Packet" refers to an individual IPv4 or IPv6 packet. Typically "Packet" refers to an individual IPv4 or IPv6 packet. Typically
packets are UDP datagrams, but may also be part of a TCP data stream. packets are UDP datagrams, but may also be part of a TCP data stream.
"Message", unless otherwise qualified, refers to a DNS payload "Message", unless otherwise qualified, refers to a DNS payload
extracted from a UDP datagram or a TCP data stream. extracted from a UDP datagram or a TCP data stream.
The parts of DNS messages are named as they are in [RFC1035]. The parts of DNS messages are named as they are in [RFC1035].
Specifically, the DNS message has five sections: Header, Question, Specifically, the DNS message has five sections: Header, Question,
Answer, Authority, and Additional. Answer, Authority, and Additional.
Pairs of DNS messages are called a Query and a Response. Pairs of DNS messages are called a Query and a Response.
3. Data collection use cases 3. Data collection use cases
In an ideal world, it would be optimal to collect full packet From a purely server operator perspective, collecting full packet
captures of all packets going in or out of a name server. However, captures of all packets going in or out of a name server provides the
there are several design choices or other limitations that are common most comprehensive picture of network activity. However, there are
to many DNS installations and operators. several design choices or other limitations that are common to many
DNS installations and operators.
o DNS servers are hosted in a variety of situations: o DNS servers are hosted in a variety of situations:
* Self-hosted servers * Self-hosted servers
* Third party hosting (including multiple third parties) * Third party hosting (including multiple third parties)
* Third party hardware (including multiple third parties) * Third party hardware (including multiple third parties)
o Data is collected under different conditions: o Data is collected under different conditions:
skipping to change at page 7, line 45 skipping to change at page 8, line 5
* The packet has a well-formed 12 byte DNS Header with a * The packet has a well-formed 12 byte DNS Header with a
recognised OPCODE. recognised OPCODE.
* The section counts are consistent with the section contents. * The section counts are consistent with the section contents.
* All of the resource records can be fully parsed. * All of the resource records can be fully parsed.
2. All top level fields in each Q/R data item will be optional. 2. All top level fields in each Q/R data item will be optional.
* Rationale: Different users will have different requirements * Rationale: Different operators will have different
for data to be available for analysis. Users with minimal requirements for data to be available for analysis. Operators
requirements should not have to pay the cost of recording full with minimal requirements should not have to pay the cost of
data, though this will limit the ability to perform certain recording full data, though this will limit the ability to
kinds of data analysis and also to reconstruct packet perform certain kinds of data analysis and also to reconstruct
captures. For example, omitting the resource records from a packet captures. For example, omitting the resource records
Response will reduce the C-DNS file size; in principle from a Response will reduce the C-DNS file size; in principle
responses can be synthesized if there is enough context. responses can be synthesized if there is enough context.
Operators may have different policies for collecting user data
and can choose to omit or anonymize certain fields at capture
time e.g. client address.
3. Multiple Q/R data items will be collected into blocks in the 3. Multiple Q/R data items will be collected into blocks in the
format. Common data in a block will be abstracted and referenced format. Common data in a block will be abstracted and referenced
from individual Q/R data items by indexing. The maximum number from individual Q/R data items by indexing. The maximum number
of Q/R data items in a block will be configurable. of Q/R data items in a block will be configurable.
* Rationale: This blocking and indexing provides a significant * Rationale: This blocking and indexing provides a significant
reduction in the volume of file data generated. Although this reduction in the volume of file data generated. Although this
introduces complexity, it provides compression of the data introduces complexity, it provides compression of the data
that makes use of knowledge of the DNS message structure. that makes use of knowledge of the DNS message structure.
skipping to change at page 9, line 12 skipping to change at page 9, line 24
The choice of CBOR was made taking a number of factors into account. The choice of CBOR was made taking a number of factors into account.
o CBOR is a binary representation, and thus is economical in storage o CBOR is a binary representation, and thus is economical in storage
space. space.
o Other binary representations were investigated, and whilst all had o Other binary representations were investigated, and whilst all had
attractive features, none had a significant advantage over CBOR. attractive features, none had a significant advantage over CBOR.
See Appendix C for some discussion of this. See Appendix C for some discussion of this.
o CBOR is an IETF standard and familiar to IETF participants. It is o CBOR is an IETF specification and familiar to IETF participants.
based on the now-common ideas of lists and objects, and thus It is based on the now-common ideas of lists and objects, and thus
requires very little familiarization for those in the wider requires very little familiarization for those in the wider
industry. industry.
o CBOR is a simple format, and can easily be implemented from o CBOR is a simple format, and can easily be implemented from
scratch if necessary. More complex formats require library scratch if necessary. More complex formats require library
support which may present problems on unusual platforms. support which may present problems on unusual platforms.
o CBOR can also be easily converted to text formats such as JSON o CBOR can also be easily converted to text formats such as JSON
([RFC8259]) for debugging and other human inspection requirements. ([RFC8259]) for debugging and other human inspection requirements.
skipping to change at page 11, line 22 skipping to change at page 11, line 22
| Client port | | Client port |
+-------------------------+ +------------------+ +-------------------------+ +------------------+
| Transaction ID | +------>| Name/RDATA array |<------+ | Transaction ID | +------>| Name/RDATA array |<------+
+-------------------------+ | +------------------+ | +-------------------------+ | +------------------+ |
| Query signature |--+ | | | Query signature |--+ | |
+-------------------------+ | | +-----------------+ | +-------------------------+ | | +-----------------+ |
| Client hoplimit (q) | +--)------>| Query Signature | | | Client hoplimit (q) | +--)------>| Query Signature | |
+-------------------------+ | +-----------------+------+ | +-------------------------+ | +-----------------+------+ |
| Response delay (r) | | | Server address | | | Response delay (r) | | | Server address | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +------------------------+ |
| Query name (q) |--+--+ | Server port | | | Query name |--+--+ | Server port | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +------------------------+ |
| Query size (q) | | | Transport flags | | | Query size (q) | | | Transport flags | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +------------------------+ |
| Response size (r) | | | QR type | | | Response size (r) | | | QR type | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +------------------------+ |
| Response processing (r) | | | QR signature flags | | | Response processing (r) | | | QR signature flags | |
| +-----------------------+ | +------------------------+ | | +-----------------------+ | +------------------------+ |
| | Bailiwick index |--+ | Query OPCODE (q) | | | | Bailiwick index |--+ | Query OPCODE (q) | |
| +-----------------------+ +------------------------+ | | +-----------------------+ +------------------------+ |
| | Flags | | QR DNS flags | | | | Flags | | QR DNS flags | |
skipping to change at page 13, line 7 skipping to change at page 13, line 7
items is in turn followed by the Address/Event Counts data items (an items is in turn followed by the Address/Event Counts data items (an
array of per-client counts of particular IP events) and then array of per-client counts of particular IP events) and then
Malformed Message data items (an array of malformed messages that Malformed Message data items (an array of malformed messages that
stored in the Block). stored in the Block).
The exact nature of the DNS data will affect what block size is the The exact nature of the DNS data will affect what block size is the
best fit, however sample data for a root server indicated that block best fit, however sample data for a root server indicated that block
sizes up to 10,000 Q/R data items give good results. See sizes up to 10,000 Q/R data items give good results. See
Appendix C.6 for more details. Appendix C.6 for more details.
This design exploits data commonality and block based storage to
minimise the C-DNS file size. As a result C-DNS cannot be streamed
below the level of a block.
6.1. Block Parameters 6.1. Block Parameters
The details of the Block Parameters items are not shown in the The details of the Block Parameters items are not shown in the
diagrams but are discussed here for context. diagrams but are discussed here for context.
An array of Block Parameters items is stored in the File Preamble An array of Block Parameters items is stored in the File Preamble
(with a minimum of one item at index 0); a Block Parameters item (with a minimum of one item at index 0); a Block Parameters item
consists of a collection of Storage and Collection Parameters that consists of a collection of Storage and Collection Parameters that
applies to any given Block. An array is used in order to support use applies to any given Block. An array is used in order to support use
cases such as wanting to merge C-DNS files from different sources. cases such as wanting to merge C-DNS files from different sources.
skipping to change at page 13, line 35 skipping to change at page 13, line 39
contains information about the specific data fields stored in the contains information about the specific data fields stored in the
C-DNS file. C-DNS file.
These parameters include: These parameters include:
o The sub-second timing resolution used by the data. o The sub-second timing resolution used by the data.
o Information (hints) on which optional data are omitted. See o Information (hints) on which optional data are omitted. See
Section 6.2.1. Section 6.2.1.
o Recorded OPCODES and RR types. See Section 6.2.2. o Recorded OPCODES [opcodes] and RR types [rrtypes]. See
Section 6.2.2.
o Flags indicating, for example, whether the data is sampled or o Flags indicating, for example, whether the data is sampled or
anonymised. See Section 6.2.3. anonymized. See Section 6.2.3 and Section 15.
o Client and server IPv4 and IPv6 address prefixes. See o Client and server IPv4 and IPv6 address prefixes. See
Section 6.2.4 Section 6.2.4
6.2.1. Optional data items 6.2.1. Optional data items
To enable implementations to store data to their precise requirements To enable implementations to store data to their precise requirements
in as space-efficient manner as possible, all fields in the following in as space-efficient manner as possible, all fields in the following
arrays are optional: arrays are optional:
skipping to change at page 14, line 29 skipping to change at page 14, line 40
2. How can it determine if a data item is not present because it 2. How can it determine if a data item is not present because it
was: was:
* explicitly not recorded or * explicitly not recorded or
* the data item was not available/present. * the data item was not available/present.
For example, capturing C-DNS data from within a nameserver For example, capturing C-DNS data from within a nameserver
implementation makes it unlikely that the Client Hoplimit can be implementation makes it unlikely that the Client Hoplimit can be
recorded. Or, if there is no query ARCount recorded and no query OPT recorded. Or, if there is no query ARCount recorded and no query OPT
RDATA recorded, is that because no query contained an OPT RR, or RDATA [RFC6891] recorded, is that because no query contained an OPT
because that data was not stored? RR, or because that data was not stored?
The Storage Parameters therefore also contains a Storage Hints item The Storage Parameters therefore also contains a Storage Hints item
which specifies which items the encoder of the file omits from the which specifies which items the encoder of the file omits from the
stored data. An implementation decoding that file can then use these stored data and will therefore never be present. (This approach is
to quickly determine whether the input data is rich enough for its taken because a flag that indicated which items were included for
needs. collection would not guarantee that the item was present, only that
it might be.) An implementation decoding that file can then use
these to quickly determine whether the input data is rich enough for
its needs.
6.2.2. Optional RRs and OPCODEs 6.2.2. Optional RRs and OPCODEs
Also included in the Storage Parameters are explicit arrays listing Also included in the Storage Parameters are explicit arrays listing
the RR types and the OPCODEs to be recorded. These remove any the RR types and the OPCODEs to be recorded. These remove any
ambiguity over whether messages containing particular OPCODEs or RR ambiguity over whether messages containing particular OPCODEs or are
types are not present because they did not occur, or because the not present because they did not occur, or because the implementation
implementation is not configured to record them. is not configured to record them.
In the case of OPCODEs, for a message to be fully parsable, the In the case of OPCODEs, for a message to be fully parsable, the
OPCODE must be known to the collecting implementation. Any message OPCODE must be known to the collecting implementation. Any message
with an OPCODE unknown to the collecting implementation cannot be with an OPCODE unknown to the collecting implementation cannot be
validated as correctly formed, and so must be treated as malformed. validated as correctly formed, and so must be treated as malformed.
Messages with OPCODES known to the recording application but not Messages with OPCODES known to the recording application but not
listed in the Storage Parameters are discarded (regardless of whether listed in the Storage Parameters are discarded by the recording
they are malformed or not). application during C-DNS capture (regardless of whether they are
malformed or not).
In the case of RR records, each record in a message must be fully In the case of RR records, each record in a message must be fully
parsable, including parsing the record RDATA, as otherwise the parsable, including parsing the record RDATA, as otherwise the
message cannot be validated as correctly formed. Any RR record with message cannot be validated as correctly formed. Any RR record with
an RR type not known to the collecting implementation cannot be an RR type not known to the collecting implementation cannot be
validated as correctly formed, and so must be treated as malformed. validated as correctly formed, and so must be treated as malformed.
Once a message is correctly parsed, an implementation is free to Once a message is correctly parsed, an implementation is free to
record only a subset of the RR records present. record only a subset of the RR records present.
6.2.3. Storage flags 6.2.3. Storage flags
The Storage Parameters contains flags that can be used to indicate The Storage Parameters contains flags that can be used to indicate
if: if:
o the data is anonymised, o the data is anonymized,
o the data is produced from sample data, or o the data is produced from sample data, or
o names in the data have been normalised (converted to uniform o names in the data have been normalized (converted to uniform
case). case).
The Storage Parameters also contains optional fields holding details The Storage Parameters also contains optional fields holding details
of the sampling method used and the anonymisation method used. It is of the sampling method used and the anonymization method used. It is
RECOMMENDED these fields contain URIs pointing to resources RECOMMENDED these fields contain URIs [RFC3986] pointing to resources
describing the methods used. describing the methods used. See Section 15 for further discussion
of anonymization and normalization.
6.2.4. IP Address storage 6.2.4. IP Address storage
The format contains fields to indicate if only IP prefixes were The format can store either full IP addresses or just IP prefixes,
stored. If IP address prefixes are given, only the prefix bits of the Storage Parameters contains fields to indicate if only IP
addresses are stored. For example, if a client IPv4 prefix of 16 is prefixes were stored.
specified, a client address of 192.0.2.1 will be stored as 0xc000
(192.0), reducing address storage space requirements. If the IP address prefixes are absent, then full addresses are
stored. In this case the IP version can be directly inferred from
the stored address length and the fields "qr-transport-flags" in
QueryResponseSignature and "mm-transport-flags" in
MalformedMessageData (which contain the IP version bit) are optional.
If IP address prefixes are given, only the prefix bits of addresses
are stored. In this case the fields "qr-transport-flags" in
QueryResponseSignature and "mm-transport-flags" in
MalformedMessageData MUST be present, so that the IP version can be
determined. See Section 7.5.3.2 and Section 7.5.3.5.
As an example of storing only IP prefixes, if a client IPv6 prefix of
48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334
will be stored as 0x20010db885a3, reducing address storage space
requirements. Similarly, if a client IPv4 prefix of 16 is specified,
a client address of 192.0.2.1 will be stored as 0xc000 (192.0).
7. C-DNS format detailed description 7. C-DNS format detailed description
The CDDL definition for the C-DNS format is given in Appendix A. The CDDL definition for the C-DNS format is given in Appendix A.
7.1. Map quantities and indexes 7.1. Map quantities and indexes
All map keys are integers with values specified in the CDDL. String All map keys are integers with values specified in the CDDL. String
keys would significantly bloat the file size. keys would significantly bloat the file size.
skipping to change at page 16, line 33 skipping to change at page 17, line 22
n)) != 0". n)) != 0".
For the sake of readability, all type and field names defined in the For the sake of readability, all type and field names defined in the
CDDL definition are shown in double quotes. Type names are by CDDL definition are shown in double quotes. Type names are by
convention camel case (e.g. "BlockTable"), field names are lower- convention camel case (e.g. "BlockTable"), field names are lower-
case with hyphens (e.g. "block-tables"). case with hyphens (e.g. "block-tables").
For the sake of brevity, the following conventions are used in the For the sake of brevity, the following conventions are used in the
tables: tables:
o The column O marks whether items in a map are optional. o The column M marks whether items in a map are mandatory.
* O - Optional. The item may be omitted. * X - Mandatory items.
* M - Mandatory. The item must be present. * C - Conditionally mandatory item. Such items are usually
optional but may be mandatory in some configurations.
* If the column is empty, the item is optional.
o The column T gives the CBOR data type of the item. o The column T gives the CBOR data type of the item.
* U - Unsigned integer * U - Unsigned integer
* I - Signed integer * I - Signed integer (i.e. CBOR unsigned or negative integer)
* B - Byte string * B - Boolean
* S - Byte string
* T - Text string * T - Text string
* M - Map * M - Map
* A - Array * A - Array
In the case of maps and arrays, more information on the type of each In the case of maps and arrays, more information on the type of each
value, include the CDDL definition name if applicable, is given in value, include the CDDL definition name if applicable, is given in
the description. the description.
7.3. "File" 7.3. "File"
A C-DNS file has an outer structure "File", a map that contains the A C-DNS file has an outer structure "File", a map that contains the
following: following:
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
| file-type-id | M | T | String "C-DNS" identifying the file type. | | file-type-id | X | T | String "C-DNS" identifying the file type. |
| | | | | | | | | |
| file-preamble | M | M | Version and parameter information for the | | file-preamble | X | M | Version and parameter information for the |
| | | | whole file. Map of type "FilePreamble", | | | | | whole file. Map of type "FilePreamble", |
| | | | see Section 7.4. | | | | | see Section 7.4. |
| | | | | | | | | |
| file-blocks | M | A | Array of items of type "Block", see | | file-blocks | X | A | Array of items of type "Block", see |
| | | | Section 7.5. The array may be empty if | | | | | Section 7.5. The array may be empty if |
| | | | the file contains no data. | | | | | the file contains no data. |
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
7.4. "FilePreamble" 7.4. "FilePreamble"
Information about data in the file. A map containing the following: Information about data in the file. A map containing the following:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| major-format-version | M | U | Unsigned integer '1'. The major | | major-format-version | X | U | Unsigned integer '1'. The major |
| | | | version of format used in file. | | | | | version of format used in file. |
| | | | See Section 8. |
| | | | | | | | | |
| minor-format-version | M | U | Unsigned integer '0'. The minor | | minor-format-version | X | U | Unsigned integer '0'. The minor |
| | | | version of format used in file. | | | | | version of format used in file. |
| | | | See Section 8. |
| | | | | | | | | |
| private-version | O | U | Version indicator available for | | private-version | | U | Version indicator available for |
| | | | private use by implementations. | | | | | private use by implementations. |
| | | | | | | | | |
| block-parameters | M | A | Array of items of type | | block-parameters | X | A | Array of items of type |
| | | | "BlockParameters", see Section | | | | | "BlockParameters", see Section |
| | | | 7.4.1. The array must contain at | | | | | 7.4.1. The array must contain at |
| | | | least one entry. (The "block- | | | | | least one entry. (The "block- |
| | | | parameters-index" item in each | | | | | parameters-index" item in each |
| | | | "BlockPreamble" indicates which | | | | | "BlockPreamble" indicates which |
| | | | array entry applies to that | | | | | array entry applies to that |
| | | | "Block".) | | | | | "Block".) |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
7.4.1. "BlockParameters" 7.4.1. "BlockParameters"
Parameters relating to data storage and collection which apply to one Parameters relating to data storage and collection which apply to one
or more items of type "Block". A map containing the following: or more items of type "Block". A map containing the following:
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
| storage-parameters | M | M | Parameters relating to data | | storage-parameters | X | M | Parameters relating to data |
| | | | storage in a "Block" item. Map | | | | | storage in a "Block" item. Map |
| | | | of type "StorageParameters", see | | | | | of type "StorageParameters", see |
| | | | Section 7.4.1.1. | | | | | Section 7.4.1.1. |
| | | | | | | | | |
| collection-parameters | O | M | Parameters relating to collection | | collection-parameters | | M | Parameters relating to collection |
| | | | of the data in a "Block" item. | | | | | of the data in a "Block" item. |
| | | | Map of type | | | | | Map of type |
| | | | "CollectionParameters", see | | | | | "CollectionParameters", see |
| | | | Section 7.4.2. | | | | | Section 7.4.2. |
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
7.4.1.1. "StorageParameters" 7.4.1.1. "StorageParameters"
Parameters relating to how data is stored in the items of type Parameters relating to how data is stored in the items of type
"Block". A map containing the following: "Block". A map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| ticks-per-second | M | U | Sub-second timing is recorded in | | ticks-per-second | X | U | Sub-second timing is recorded in |
| | | | ticks. This specifies the number of | | | | | ticks. This specifies the number of |
| | | | ticks in a second. | | | | | ticks in a second. |
| | | | | | | | | |
| max-block-items | M | U | The maximum number of items stored in | | max-block-items | X | U | The maximum number of items stored in |
| | | | any of the arrays in a "Block" item | | | | | any of the arrays in a "Block" item |
| | | | (Q/R items, address event counts or | | | | | (Q/R items, address event counts or |
| | | | malformed messages). An indication to | | | | | malformed messages). An indication to |
| | | | a decoder of the resources needed to | | | | | a decoder of the resources needed to |
| | | | process the file. | | | | | process the file. |
| | | | | | | | | |
| storage-hints | M | M | Collection of hints as to which fields | | storage-hints | X | M | Collection of hints as to which fields |
| | | | are omitted in the arrays that have | | | | | are omitted in the arrays that have |
| | | | optional fields. Map of type | | | | | optional fields. Map of type |
| | | | "StorageHints", see Section 7.4.1.1.1. | | | | | "StorageHints", see Section 7.4.1.1.1. |
| | | | | | | | | |
| opcodes | M | A | Array of OPCODES (unsigned integers) | | opcodes | X | A | Array of OPCODES [opcodes] (unsigned |
| | | | recorded by the collection | | | | | integers, each in the range 0 to 15 |
| | | | inclusive) recorded by the collection |
| | | | implementation. See Section 6.2.2. | | | | | implementation. See Section 6.2.2. |
| | | | | | | | | |
| rr-types | M | A | Array of RR types (unsigned integers) | | rr-types | X | A | Array of RR types [rrtypes] (unsigned |
| | | | recorded by the collection | | | | | integers, each in the range 0 to 65535 |
| | | | inclusive) recorded by the collection |
| | | | implementation. See Section 6.2.2. | | | | | implementation. See Section 6.2.2. |
| | | | | | | | | |
| storage-flags | O | U | Bit flags indicating attributes of | | storage-flags | | U | Bit flags indicating attributes of |
| | | | stored data. | | | | | stored data. |
| | | | Bit 0. 1 if the data has been | | | | | Bit 0. 1 if the data has been |
| | | | anonymised. | | | | | anonymized. |
| | | | Bit 1. 1 if the data is sampled data. | | | | | Bit 1. 1 if the data is sampled data. |
| | | | Bit 2. 1 if the names have been | | | | | Bit 2. 1 if the names have been |
| | | | normalised (converted to uniform | | | | | normalized (converted to uniform |
| | | | case). | | | | | case). |
| | | | | | | | | |
| client-address | O | U | IPv4 client address prefix length. If | | client-address | | U | IPv4 client address prefix length, in |
| -prefix-ipv4 | | | specified, only the address prefix | | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| client-address | O | U | IPv6 client address prefix length. If | | client-address | | U | IPv6 client address prefix length, in |
| -prefix-ipv6 | | | specified, only the address prefix | | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| server-address | O | U | IPv4 server address prefix length. If | | server-address | | U | IPv4 server address prefix length, in |
| -prefix-ipv4 | | | specified, only the address prefix | | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| server-address | O | U | IPv6 server address prefix length. If | | server-address | | U | IPv6 server address prefix length, in |
| -prefix-ipv6 | | | specified, only the address prefix | | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| sampling-method | O | T | Information on the sampling method | | sampling-method | | T | Information on the sampling method |
| | | | used. See Section 6.2.3. | | | | | used. See Section 6.2.3. |
| | | | | | | | | |
| anonymisation | O | T | Information on the anonymisation | | anonymization | | T | Information on the anonymization |
| -method | | | method used. See Section 6.2.3. | | -method | | | method used. See Section 6.2.3. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.4.1.1.1. "StorageHints" 7.4.1.1.1. "StorageHints"
An indicator of which fields the collecting implementation omits in An indicator of which fields the collecting implementation omits in
the arrays with optional fields. A map containing the following: the maps with optional fields. A map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| query-response | M | U | Hints indicating which "QueryResponse" | | query-response | X | U | Hints indicating which "QueryResponse" |
| -hints | | | fields are omitted, see section | | -hints | | | fields are candidates for capture or |
| | | | Section 7.6. If the field is omitted | | | | | omitted, see section Section 7.6. If a |
| | | | the bit is unset. | | | | | bit is unset, the field is omitted |
| | | | from the capture. |
| | | | Bit 0. time-offset | | | | | Bit 0. time-offset |
| | | | Bit 1. client-address-index | | | | | Bit 1. client-address-index |
| | | | Bit 2. client-port | | | | | Bit 2. client-port |
| | | | Bit 3. transaction-id | | | | | Bit 3. transaction-id |
| | | | Bit 4. qr-signature-index | | | | | Bit 4. qr-signature-index |
| | | | Bit 5. client-hoplimit | | | | | Bit 5. client-hoplimit |
| | | | Bit 6. response-delay | | | | | Bit 6. response-delay |
| | | | Bit 7. query-name-index | | | | | Bit 7. query-name-index |
| | | | Bit 8. query-size | | | | | Bit 8. query-size |
| | | | Bit 9. response-size | | | | | Bit 9. response-size |
| | | | Bit 10. response-processing-data | | | | | Bit 10. response-processing-data |
| | | | Bit 11. query-question-sections | | | | | Bit 11. query-question-sections |
| | | | Bit 12. query-answer-sections | | | | | Bit 12. query-answer-sections |
| | | | Bit 13. query-authority-sections | | | | | Bit 13. query-authority-sections |
| | | | Bit 14. query-additional-sections | | | | | Bit 14. query-additional-sections |
| | | | Bit 15. response-answer-sections | | | | | Bit 15. response-answer-sections |
| | | | Bit 16. response-authority-sections | | | | | Bit 16. response-authority-sections |
| | | | Bit 17. response-additional-sections | | | | | Bit 17. response-additional-sections |
| | | | | | | | | |
| query-response | M | U | Hints indicating which | | query-response | X | U | Hints indicating which |
| -signature-hints | | | "QueryResponseSignature" fields are | | -signature-hints | | | "QueryResponseSignature" fields are |
| | | | omitted, see section Section 7.5.3.2. | | | | | candidates for capture or omitted, see |
| | | | If the field is omitted the bit is | | | | | section Section 7.5.3.2. If a bit is |
| | | | unset. | | | | | unset, the field is omitted from the |
| | | | capture. |
| | | | Bit 0. server-address | | | | | Bit 0. server-address |
| | | | Bit 1. server-port | | | | | Bit 1. server-port |
| | | | Bit 2. qr-transport-flags | | | | | Bit 2. qr-transport-flags |
| | | | Bit 3. qr-type | | | | | Bit 3. qr-type |
| | | | Bit 4. qr-sig-flags | | | | | Bit 4. qr-sig-flags |
| | | | Bit 5. query-opcode | | | | | Bit 5. query-opcode |
| | | | Bit 6. dns-flags | | | | | Bit 6. dns-flags |
| | | | Bit 7. query-rcode | | | | | Bit 7. query-rcode |
| | | | Bit 8. query-class-type | | | | | Bit 8. query-class-type |
| | | | Bit 9. query-qdcount | | | | | Bit 9. query-qdcount |
| | | | Bit 10. query-ancount | | | | | Bit 10. query-ancount |
| | | | Bit 11. query-nscount | | | | | Bit 11. query-nscount |
| | | | Bit 12. query-arcount | | | | | Bit 12. query-arcount |
| | | | Bit 13. query-edns-version | | | | | Bit 13. query-edns-version |
| | | | Bit 14. query-udp-size | | | | | Bit 14. query-udp-size |
| | | | Bit 15. query-opt-rdata | | | | | Bit 15. query-opt-rdata |
| | | | Bit 16. response-rcode | | | | | Bit 16. response-rcode |
| | | | | | | | | |
| rr-hints | M | U | Hints indicating which optional "RR" | | rr-hints | X | U | Hints indicating which optional "RR" |
| | | | fields are omitted, see Section | | | | | fields are candidates for capture or |
| | | | 7.5.3.4. If the field is omitted the | | | | | omitted, see Section 7.5.3.4. If a bit |
| | | | bit is unset. | | | | | is unset, the field is omitted from |
| | | | the capture. |
| | | | Bit 0. ttl | | | | | Bit 0. ttl |
| | | | Bit 1. rdata-index | | | | | Bit 1. rdata-index |
| other-data-hints | M | U | Hints indicating which other data | | other-data-hints | X | U | Hints indicating which other data |
| | | | types are are omitted. If the data | | | | | types are omitted. If a bit is unset, |
| | | | type is are omitted the bit is unset. | | | | | the the data type is omitted from the |
| | | | capture. |
| | | | Bit 0. malformed-messages | | | | | Bit 0. malformed-messages |
| | | | Bit 1. address-event-counts | | | | | Bit 1. address-event-counts |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.4.2. "CollectionParameters" 7.4.2. "CollectionParameters"
Parameters relating to how data in the file was collected. Parameters providing information to how data in the file was
collected (applicable for some, but not all collection environments).
The values are informational only and serve as hints to downstream
analysers as to the configuration of a collecting implementation.
They can provide context when interpreting what data is present/
absent from the capture but cannot necessarily be validated against
the data captured.
These parameters have no default. If they do not appear, nothing can These parameters have no default. If they do not appear, nothing can
be inferred about their value. be inferred about their value.
A map containing the following items: A map containing the following items:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| query-timeout | O | U | To be matched with a query, a response | | query-timeout | | U | To be matched with a query, a response |
| | | | must arrive within this number of | | | | | must arrive within this number of |
| | | | seconds. | | | | | seconds. |
| | | | | | | | | |
| skew-timeout | O | U | The network stack may report a | | skew-timeout | | U | The network stack may report a |
| | | | response before the corresponding | | | | | response before the corresponding |
| | | | query. A response is not considered to | | | | | query. A response is not considered to |
| | | | be missing a query until after this | | | | | be missing a query until after this |
| | | | many micro-seconds. | | | | | many micro-seconds. |
| | | | | | | | | |
| snaplen | O | U | Collect up to this many bytes per | | snaplen | | U | Collect up to this many bytes per |
| | | | packet. | | | | | packet. |
| | | | | | | | | |
| promisc | O | U | 1 if promiscuous mode was enabled on | | promisc | | B | "true" if promiscuous mode |
| | | | the interface, 0 otherwise. | | | | | [pcap-options] was enabled on the |
| | | | interface, "false" otherwise. |
| | | | | | | | | |
| interfaces | O | A | Array of identifiers (of type text | | interfaces | | A | Array of identifiers (of type text |
| | | | string) of the interfaces used for | | | | | string) of the interfaces used for |
| | | | collection. | | | | | collection. |
| | | | | | | | | |
| server-addresses | O | A | Array of server collection IP | | server-addresses | | A | Array of server collection IP |
| | | | addresses (of type byte string). Hint | | | | | addresses (of type byte string). Hint |
| | | | for downstream analysers; does not | | | | | for downstream analysers; does not |
| | | | affect collection. | | | | | affect collection. |
| | | | | | | | | |
| vlan-ids | O | A | Array of identifiers (of type unsigned | | vlan-ids | | A | Array of identifiers (of type unsigned |
| | | | integer) of VLANs selected for | | | | | integer, each in the range 1 to 4094 |
| | | | collection. | | | | | inclusive) of VLANs [IEEE802.1Q] |
| | | | selected for collection. VLAN IDs are |
| | | | unique only within an administrative |
| | | | domain. |
| | | | | | | | | |
| filter | O | T | "tcpdump" [pcap] style filter for | | filter | | T | "tcpdump" [pcap-filter] style filter |
| | | | input. | | | | | for input. |
| | | | | | | | | |
| generator-id | O | T | String identifying the collection | | generator-id | | T | Implementation specific human-readable |
| | | | string identifying the collection |
| | | | method. | | | | | method. |
| | | | | | | | | |
| host-id | O | T | String identifying the collecting | | host-id | | T | String identifying the collecting |
| | | | host. Empty if converting an existing | | | | | host. Empty if converting an existing |
| | | | packet capture file. | | | | | packet capture file. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.5. "Block" 7.5. "Block"
Container for data with common collection and and storage parameters. Container for data with common collection and storage parameters. A
A map containing the following: map containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| block-preamble | M | M | Overall information for the "Block" | | block-preamble | X | M | Overall information for the "Block" |
| | | | item. Map of type "BlockPreamble", | | | | | item. Map of type "BlockPreamble", |
| | | | see Section 7.5.1. | | | | | see Section 7.5.1. |
| | | | | | | | | |
| block-statistics | O | M | Statistics about the "Block" item. | | block-statistics | | M | Statistics about the "Block" item. |
| | | | Map of type "BlockStatistics", see | | | | | Map of type "BlockStatistics", see |
| | | | Section 7.5.2. | | | | | Section 7.5.2. |
| | | | | | | | | |
| block-tables | O | M | The arrays containing data | | block-tables | | M | The arrays containing data |
| | | | referenced by individual | | | | | referenced by individual |
| | | | "QueryResponse" or | | | | | "QueryResponse" or |
| | | | "MalformedMessage" items. Map of | | | | | "MalformedMessage" items. Map of |
| | | | type "BlockTables", see Section | | | | | type "BlockTables", see Section |
| | | | 7.5.3. | | | | | 7.5.3. |
| | | | | | | | | |
| query-responses | O | A | Details of individual DNS Q/R data | | query-responses | | A | Details of individual DNS Q/R data |
| | | | items. Array of items of type | | | | | items. Array of items of type |
| | | | "QueryResponse", see Section 7.6. If | | | | | "QueryResponse", see Section 7.6. If |
| | | | present, the array must not be | | | | | present, the array must not be |
| | | | empty. | | | | | empty. |
| | | | | | | | | |
| address-event | O | A | Per client counts of ICMP messages | | address-event | | A | Per client counts of ICMP messages |
| -counts | | | and TCP resets. Array of items of | | -counts | | | and TCP resets. Array of items of |
| | | | type "AddressEventCount", see | | | | | type "AddressEventCount", see |
| | | | Section 7.7. If present, the array | | | | | Section 7.7. If present, the array |
| | | | must not be empty. | | | | | must not be empty. |
| | | | | | | | | |
| malformed-messages | O | A | Details of malformed DNS messages. | | malformed-messages | | A | Details of malformed DNS messages. |
| | | | Array of items of type | | | | | Array of items of type |
| | | | "MalformedMessage", see Section 7.8. | | | | | "MalformedMessage", see Section 7.8. |
| | | | If present, the array must not be | | | | | If present, the array must not be |
| | | | empty. | | | | | empty. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.5.1. "BlockPreamble" 7.5.1. "BlockPreamble"
Overall information for a "Block" item. A map containing the Overall information for a "Block" item. A map containing the
following: following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| earliest-time | O | A | A timestamp (2 unsigned integers, | | earliest-time | C | A | A timestamp (2 unsigned integers, |
| | | | "Timestamp") for the earliest record | | | | | "Timestamp") for the earliest record |
| | | | in the "Block" item. The first integer | | | | | in the "Block" item. The first integer |
| | | | is the number of seconds since the | | | | | is the number of seconds since the |
| | | | Posix epoch ("time_t"). The second | | | | | POSIX epoch [posix-time] ("time_t"), |
| | | | integer is the number of ticks since | | | | | excluding leap seconds. The second |
| | | | the start of the second. This | | | | | integer is the number of ticks (see |
| | | | timestamp can only be omitted if all | | | | | Section 7.4.1.1) since the start of |
| | | | block items containing a time offset | | | | | the second. This field is mandatory |
| | | | from the start of the block also omit | | | | | unless all block items containing a |
| | | | that time offset. | | | | | time offset from the start of the |
| | | | block also omit that time offset. |
| | | | | | | | | |
| block-parameters | O | U | The index of the item in the "block- | | block-parameters | | U | The index of the item in the "block- |
| -index | | | parameters" array (in the "file- | | -index | | | parameters" array (in the "file- |
| | | | premable" item) applicable to this | | | | | premable" item) applicable to this |
| | | | block. If not present, index 0 is | | | | | block. If not present, index 0 is |
| | | | used. See Section 7.4.1. | | | | | used. See Section 7.4.1. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.5.2. "BlockStatistics" 7.5.2. "BlockStatistics"
Basic statistical information about a "Block" item. A map containing Basic statistical information about a "Block" item. A map containing
the following: the following:
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
| processed-messages | O | U | Total number of DNS messages | | processed-messages | | U | Total number of DNS messages |
| | | | processed from the input traffic | | | | | processed from the input traffic |
| | | | stream during collection of data in | | | | | stream during collection of data in |
| | | | this "Block" item. | | | | | this "Block" item. |
| | | | | | | | | |
| qr-data-items | O | U | Total number of Q/R data items in | | qr-data-items | | U | Total number of Q/R data items in |
| | | | this "Block" item. | | | | | this "Block" item. |
| | | | | | | | | |
| unmatched-queries | O | U | Number of unmatched queries in this | | unmatched-queries | | U | Number of unmatched queries in this |
| | | | "Block" item. | | | | | "Block" item. |
| | | | | | | | | |
| unmatched-responses | O | U | Number of unmatched responses in | | unmatched-responses | | U | Number of unmatched responses in |
| | | | this "Block" item. | | | | | this "Block" item. |
| | | | | | | | | |
| discarded-opcode | O | U | Number of DNS messages processed | | discarded-opcode | | U | Number of DNS messages processed |
| | | | from the input traffic stream | | | | | from the input traffic stream |
| | | | during collection of data in this | | | | | during collection of data in this |
| | | | "Block" item but not recorded | | | | | "Block" item but not recorded |
| | | | because their OPCODE is not in the | | | | | because their OPCODE is not in the |
| | | | list to be collected. | | | | | list to be collected. |
| | | | | | | | | |
| malformed-items | O | U | Number of malformed messages found | | malformed-items | | U | Number of malformed messages found |
| | | | in input for this "Block" item. | | | | | in input for this "Block" item. |
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
7.5.3. "BlockTables" 7.5.3. "BlockTables"
Arrays containing data referenced by individual "QueryResponse" or Map of arrays containing data referenced by individual
"MalformedMessage" items in this "Block". Each element is an array "QueryResponse" or "MalformedMessage" items in this "Block". Each
which, if present, must not be empty. element is an array which, if present, must not be empty.
An item in the "qlist" array contains indexes to values in the "qrr" An item in the "qlist" array contains indexes to values in the "qrr"
array. Therefore, if "qlist" is present, "qrr" must also be present. array. Therefore, if "qlist" is present, "qrr" must also be present.
Similarly, if "rrlist" is present, "rr" must also be present. Similarly, if "rrlist" is present, "rr" must also be present.
The map contains the following items: The map contains the following items:
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
| ip-address | O | A | Array of IP addresses, in network | | ip-address | | A | Array of IP addresses, in network |
| | | | byte order (of type byte string). If | | | | | byte order (of type byte string). If |
| | | | client or server address prefixes are | | | | | client or server address prefixes are |
| | | | set, only the address prefix bits are | | | | | set, only the address prefix bits are |
| | | | stored. Each string is therefore up | | | | | stored. Each string is therefore up |
| | | | to 4 bytes long for an IPv4 address, | | | | | to 4 bytes long for an IPv4 address, |
| | | | or up to 16 bytes long for an IPv6 | | | | | or up to 16 bytes long for an IPv6 |
| | | | address. See Section 7.4.1.1. | | | | | address. See Section 7.4.1.1. |
| | | | | | | | | |
| classtype | O | A | Array of RR class and type | | classtype | | A | Array of RR class and type |
| | | | information. Type is "ClassType", see | | | | | information. Type is "ClassType", see |
| | | | Section 7.5.3.1. | | | | | Section 7.5.3.1. |
| | | | | | | | | |
| name-rdata | O | A | Array where each entry is the | | name-rdata | | A | Array where each entry is the |
| | | | contents of a single NAME or RDATA | | | | | contents of a single NAME or RDATA in |
| | | | (of type byte string). Note that | | | | | wire format (of type byte string). |
| | | | NAMEs, and labels within RDATA | | | | | Note that NAMEs, and labels within |
| | | | contents, are full domain names or | | | | | RDATA contents, are full domain names |
| | | | labels; no DNS style name compression | | | | | or labels; no [RFC1035] name |
| | | | is used on the individual | | | | | compression is used on the individual |
| | | | names/labels within the format. | | | | | names/labels within the format. |
| | | | | | | | | |
| qr-sig | O | A | Array Q/R data item signatures. Type | | qr-sig | | A | Array Q/R data item signatures. Type |
| | | | is "QueryResponseSignature", see | | | | | is "QueryResponseSignature", see |
| | | | Section 7.5.3.2. | | | | | Section 7.5.3.2. |
| | | | | | | | | |
| qlist | O | A | Array of type "QuestionList". A | | qlist | | A | Array of type "QuestionList". A |
| | | | "QuestionList" is an array of | | | | | "QuestionList" is an array of |
| | | | unsigned integers, indexes to | | | | | unsigned integers, indexes to |
| | | | "Question" items in the "qrr" array. | | | | | "Question" items in the "qrr" array. |
| | | | | | | | | |
| qrr | O | A | Array of type "Question". Each entry | | qrr | | A | Array of type "Question". Each entry |
| | | | is the contents of a single question, | | | | | is the contents of a single question, |
| | | | where a question is the second or | | | | | where a question is the second or |
| | | | subsequent question in a query. See | | | | | subsequent question in a query. See |
| | | | Section 7.5.3.3. | | | | | Section 7.5.3.3. |
| | | | | | | | | |
| rrlist | O | A | Array of type "RRList". An "RRList" | | rrlist | | A | Array of type "RRList". An "RRList" |
| | | | is an array of unsigned integers, | | | | | is an array of unsigned integers, |
| | | | indexes to "RR" items in the "rr" | | | | | indexes to "RR" items in the "rr" |
| | | | array. | | | | | array. |
| | | | | | | | | |
| rr | O | A | Array of type "RR". Each entry is the | | rr | | A | Array of type "RR". Each entry is the |
| | | | contents of a single RR. See Section | | | | | contents of a single RR. See Section |
| | | | 7.5.3.4. | | | | | 7.5.3.4. |
| | | | | | | | | |
| malformed-message | O | A | Array of the contents of malformed | | malformed-message | | A | Array of the contents of malformed |
| -data | | | messages. Array of type | | -data | | | messages. Array of type |
| | | | "MalformedMessageData", see Section | | | | | "MalformedMessageData", see Section |
| | | | 7.5.3.5. | | | | | 7.5.3.5. |
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
7.5.3.1. "ClassType" 7.5.3.1. "ClassType"
RR class and type information. A map containing the following: RR class and type information. A map containing the following:
+-------+---+---+--------------+ +-------+---+---+--------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+-------+---+---+--------------+ +-------+---+---+--------------------------+
| type | M | U | TYPE value. | | type | X | U | TYPE value [rrtypes]. |
| | | | | | | | | |
| class | M | U | CLASS value. | | class | X | U | CLASS value [rrclasses]. |
+-------+---+---+--------------+ +-------+---+---+--------------------------+
7.5.3.2. "QueryResponseSignature" 7.5.3.2. "QueryResponseSignature"
Elements of a Q/R data item that are often common between multiple Elements of a Q/R data item that are often common between multiple
individual Q/R data items. A map containing the following: individual Q/R data items. A map containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| server-address | O | U | The index in the item in the "ip- | | server-address | | U | The index in the item in the "ip- |
| -index | | | address" array of the server IP | | -index | | | address" array of the server IP |
| | | | address. See Section 7.5.3. | | | | | address. See Section 7.5.3. |
| | | | | | | | | |
| server-port | O | U | The server port. | | server-port | | U | The server port. |
| | | | | | | | | |
| qr-transport-flags | O | U | Bit flags describing the transport | | qr-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the query. | | | | | used to service the query. Same |
| | | | definition as "mm-transport-flags" |
| | | | in Section 7.5.3.5, with an |
| | | | additional indicator for trailing |
| | | | bytes, see Appendix A. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if | | | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6 | | | | | IPv6. See Section 6.2.4. |
| | | | Bit 1-4. Transport. 4 bit unsigned | | | | | Bit 1-4. Transport. 4 bit unsigned |
| | | | value where 0 = UDP, 1 = TCP, 2 = | | | | | value where 0 = UDP, 1 = TCP, 2 = |
| | | | TLS, 3 = DTLS. Values 4-15 are | | | | | TLS, 3 = DTLS [RFC7858], 4 = DoH |
| | | | reserved for future use. | | | | | [RFC8484]. Values 5-15 are reserved |
| | | | for future use. |
| | | | Bit 5. 1 if trailing bytes in query | | | | | Bit 5. 1 if trailing bytes in query |
| | | | packet. See Section 11.2. | | | | | packet. See Section 11.2. |
| | | | | | | | | |
| qr-type | O | U | Type of Query/Response transaction. | | qr-type | | U | Type of Query/Response transaction. |
| | | | 0 = Stub. A query from a stub | | | | | 0 = Stub. A query from a stub |
| | | | resolver. | | | | | resolver. |
| | | | 1 = Client. An incoming query to a | | | | | 1 = Client. An incoming query to a |
| | | | recursive resolver. | | | | | recursive resolver. |
| | | | 2 = Resolver. A query sent from a | | | | | 2 = Resolver. A query sent from a |
| | | | recursive resolver to an authorative | | | | | recursive resolver to an authorative |
| | | | resolver. | | | | | resolver. |
| | | | 3 = Authorative. A query to an | | | | | 3 = Authorative. A query to an |
| | | | authorative resolver. | | | | | authorative resolver. |
| | | | 4 = Forwarder. A query sent from a | | | | | 4 = Forwarder. A query sent from a |
| | | | recursive resolver to an upstream | | | | | recursive resolver to an upstream |
| | | | recursive resolver. | | | | | recursive resolver. |
| | | | 5 = Tool. A query sent to a server | | | | | 5 = Tool. A query sent to a server |
| | | | by a server tool. | | | | | by a server tool. |
| | | | | | | | | |
| qr-sig-flags | O | U | Bit flags explicitly indicating | | qr-sig-flags | | U | Bit flags explicitly indicating |
| | | | attributes of the message pair | | | | | attributes of the message pair |
| | | | represented by this Q/R data item | | | | | represented by this Q/R data item |
| | | | (not all attributes may be recorded | | | | | (not all attributes may be recorded |
| | | | or deducible). | | | | | or deducible). |
| | | | Bit 0. 1 if a Query was present. | | | | | Bit 0. 1 if a Query was present. |
| | | | Bit 1. 1 if a Response was present. | | | | | Bit 1. 1 if a Response was present. |
| | | | Bit 2. 1 if a Query was present and | | | | | Bit 2. 1 if a Query was present and |
| | | | it had an OPT Resource Record. | | | | | it had an OPT Resource Record. |
| | | | Bit 3. 1 if a Response was present | | | | | Bit 3. 1 if a Response was present |
| | | | and it had an OPT Resource Record. | | | | | and it had an OPT Resource Record. |
| | | | Bit 4. 1 if a Query was present but | | | | | Bit 4. 1 if a Query was present but |
| | | | had no Question. | | | | | had no Question. |
| | | | Bit 5. 1 if a Response was present | | | | | Bit 5. 1 if a Response was present |
| | | | but had no Question (only one query- | | | | | but had no Question (only one query- |
| | | | name-index is stored per Q/R item). | | | | | name-index is stored per Q/R item). |
| | | | | | | | | |
| query-opcode | O | U | Query OPCODE. | | query-opcode | | U | Query OPCODE. |
| | | | | | | | | |
| qr-dns-flags | O | U | Bit flags with values from the Query | | qr-dns-flags | | U | Bit flags with values from the Query |
| | | | and Response DNS flags. Flag values | | | | | and Response DNS flags. Flag values |
| | | | are 0 if the Query or Response is | | | | | are 0 if the Query or Response is |
| | | | not present. | | | | | not present. |
| | | | Bit 0. Query Checking Disabled (CD). | | | | | Bit 0. Query Checking Disabled (CD). |
| | | | Bit 1. Query Authenticated Data | | | | | Bit 1. Query Authenticated Data |
| | | | (AD). | | | | | (AD). |
| | | | Bit 2. Query reserved (Z). | | | | | Bit 2. Query reserved (Z). |
| | | | Bit 3. Query Recursion Available | | | | | Bit 3. Query Recursion Available |
| | | | (RA). | | | | | (RA). |
| | | | Bit 4. Query Recursion Desired (RD). | | | | | Bit 4. Query Recursion Desired (RD). |
skipping to change at page 29, line 8 skipping to change at page 30, line 13
| | | | (AD). | | | | | (AD). |
| | | | Bit 10. Response reserved (Z). | | | | | Bit 10. Response reserved (Z). |
| | | | Bit 11. Response Recursion Available | | | | | Bit 11. Response Recursion Available |
| | | | (RA). | | | | | (RA). |
| | | | Bit 12. Response Recursion Desired | | | | | Bit 12. Response Recursion Desired |
| | | | (RD). | | | | | (RD). |
| | | | Bit 13. Response TrunCation (TC). | | | | | Bit 13. Response TrunCation (TC). |
| | | | Bit 14. Response Authoritative | | | | | Bit 14. Response Authoritative |
| | | | Answer (AA). | | | | | Answer (AA). |
| | | | | | | | | |
| query-rcode | O | U | Query RCODE. If the Query contains | | query-rcode | | U | Query RCODE. If the Query contains |
| | | | OPT, this value incorporates any | | | | | OPT [RFC6891], this value |
| | | | EXTENDED_RCODE_VALUE. | | | | | incorporates any |
| | | | EXTENDED_RCODE_VALUE [rcodes]. |
| | | | | | | | | |
| query-classtype | O | U | The index to the item in the the | | query-classtype | | U | The index to the item in the the |
| -index | | | "classtype" array of the CLASS and | | -index | | | "classtype" array of the CLASS and |
| | | | TYPE of the first Question. See | | | | | TYPE of the first Question. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
| | | | | | | | | |
| query-qd-count | O | U | The QDCOUNT in the Query, or | | query-qd-count | | U | The QDCOUNT in the Query, or |
| | | | Response if no Query present. | | | | | Response if no Query present. |
| | | | | | | | | |
| query-an-count | O | U | Query ANCOUNT. | | query-an-count | | U | Query ANCOUNT. |
| | | | | | | | | |
| query-ns-count | O | U | Query NSCOUNT. | | query-ns-count | | U | Query NSCOUNT. |
| | | | | | | | | |
| query-ar-count | O | U | Query ARCOUNT. | | query-ar-count | | U | Query ARCOUNT. |
| | | | | | | | | |
| edns-version | O | U | The Query EDNS version. | | edns-version | | U | The Query EDNS version. |
| | | | | | | | | |
| udp-buf-size | O | U | The Query EDNS sender's UDP payload | | udp-buf-size | | U | The Query EDNS sender's UDP payload |
| | | | size. | | | | | size. |
| | | | | | | | | |
| opt-rdata-index | O | U | The index in the "name-rdata" array | | opt-rdata-index | | U | The index in the "name-rdata" array |
| | | | of the OPT RDATA. See Section 7.5.3. | | | | | of the OPT RDATA. See Section 7.5.3. |
| | | | | | | | | |
| response-rcode | O | U | Response RCODE. If the Response | | response-rcode | | U | Response RCODE. If the Response |
| | | | contains OPT, this value | | | | | contains OPT [RFC6891], this value |
| | | | incorporates any | | | | | incorporates any |
| | | | EXTENDED_RCODE_VALUE. | | | | | EXTENDED_RCODE_VALUE [rcodes]. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.5.3.3. "Question" 7.5.3.3. "Question"
Details on individual Questions in a Question section. A map Details on individual Questions in a Question section. A map
containing the following: containing the following:
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| name-index | M | U | The index in the "name-rdata" array of | | name-index | X | U | The index in the "name-rdata" array of |
| | | | the QNAME. See Section 7.5.3. | | | | | the QNAME. See Section 7.5.3. |
| | | | | | | | | |
| classtype-index | M | U | The index in the "classtype" array of | | classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the Question. See | | | | | the CLASS and TYPE of the Question. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
7.5.3.4. "RR" 7.5.3.4. "RR"
Details on individual Resource Records in RR sections. A map Details on individual Resource Records in RR sections. A map
containing the following: containing the following:
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| name-index | M | U | The index in the "name-rdata" array of | | name-index | X | U | The index in the "name-rdata" array of |
| | | | the NAME. See Section 7.5.3. | | | | | the NAME. See Section 7.5.3. |
| | | | | | | | | |
| classtype-index | M | U | The index in the "classtype" array of | | classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the RR. See | | | | | the CLASS and TYPE of the RR. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
| | | | | | | | | |
| ttl | O | U | The RR Time to Live. | | ttl | | U | The RR Time to Live. |
| | | | | | | | | |
| rdata-index | O | U | The index in the "name-rdata" array of | | rdata-index | | U | The index in the "name-rdata" array of |
| | | | the RR RDATA. See Section 7.5.3. | | | | | the RR RDATA. See Section 7.5.3. |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
7.5.3.5. "MalformedMessageData" 7.5.3.5. "MalformedMessageData"
Details on malformed message items in this "Block" item. A map Details on malformed message items in this "Block" item. A map
containing the following: containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| server-address | O | U | The index in the "ip-address" array | | server-address | | U | The index in the "ip-address" array |
| -index | | | of the server IP address. See | | -index | | | of the server IP address. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
| | | | | | | | | |
| server-port | O | U | The server port. | | server-port | | U | The server port. |
| | | | | | | | | |
| mm-transport-flags | O | U | Bit flags describing the transport | | mm-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the query. Bit 0 is | | | | | used to service the query, see |
| | | | the least significant bit. | | | | | Section 6.2.4. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if | | | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6 | | | | | IPv6 |
| | | | Bit 1-4. Transport. 4 bit unsigned | | | | | Bit 1-4. Transport. 4 bit unsigned |
| | | | value where 0 = UDP, 1 = TCP, 2 = | | | | | value where 0 = UDP, 1 = TCP, 2 = |
| | | | TLS, 3 = DTLS. Values 4-15 are | | | | | TLS, 3 = DTLS [RFC7858], 4 = DoH |
| | | | reserved for future use. | | | | | [RFC8484]. Values 5-15 are reserved |
| | | | for future use. |
| | | | | | | | | |
| mm-payload | O | B | The payload (raw bytes) of the DNS | | mm-payload | | S | The payload (raw bytes) of the DNS |
| | | | message. | | | | | message. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.6. "QueryResponse" 7.6. "QueryResponse"
Details on individual Q/R data items. Details on individual Q/R data items.
Note that there is no requirement that the elements of the "query- Note that there is no requirement that the elements of the "query-
responses" array are presented in strict chronological order. responses" array are presented in strict chronological order.
A map containing the following items: A map containing the following items:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| time-offset | O | U | Q/R timestamp as an offset in | | time-offset | | U | Q/R timestamp as an offset in |
| | | | ticks from "earliest-time". The | | | | | ticks (see Section 7.4.1.1) from |
| | | | timestamp is the timestamp of the | | | | | "earliest-time". The timestamp is |
| | | | Query, or the Response if there is | | | | | the timestamp of the Query, or the |
| | | | no Query. | | | | | Response if there is no Query. |
| | | | | | | | | |
| client-address-index | O | U | The index in the "ip-address" | | client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. | | | | | array of the client IP address. |
| | | | See Section 7.5.3. | | | | | See Section 7.5.3. |
| | | | | | | | | |
| client-port | O | U | The client port. | | client-port | | U | The client port. |
| | | | | | | | | |
| transaction-id | O | U | DNS transaction identifier. | | transaction-id | | U | DNS transaction identifier. |
| | | | | | | | | |
| qr-signature-index | O | U | The index in the "qr-sig" array of | | qr-signature-index | | U | The index in the "qr-sig" array of |
| | | | the "QueryResponseSignature" item. | | | | | the "QueryResponseSignature" item. |
| | | | See Section 7.5.3. | | | | | See Section 7.5.3. |
| | | | | | | | | |
| client-hoplimit | O | U | The IPv4 TTL or IPv6 Hoplimit from | | client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from |
| | | | the Query packet. | | | | | the Query packet. |
| | | | | | | | | |
| response-delay | O | I | The time difference between Query | | response-delay | | I | The time difference between Query |
| | | | and Response, in ticks. Only | | | | | and Response, in ticks (see |
| | | | present if there is a query and a | | | | | Section 7.4.1.1). Only present if |
| | | | response. The delay can be | | | | | there is a query and a response. |
| | | | negative if the network | | | | | The delay can be negative if the |
| | | | stack/capture library returns | | | | | network stack/capture library |
| | | | packets out of order. | | | | | returns packets out of order. |
| | | | | | | | | |
| query-name-index | O | U | The index in the "name-rdata" | | query-name-index | | U | The index in the "name-rdata" |
| | | | array of the item containing the | | | | | array of the item containing the |
| | | | QNAME for the first Question. See | | | | | QNAME for the first Question. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
| | | | | | | | | |
| query-size | O | U | DNS query message size (see | | query-size | | U | DNS query message size (see |
| | | | below). | | | | | below). |
| | | | | | | | | |
| response-size | O | U | DNS query message size (see | | response-size | | U | DNS response message size (see |
| | | | below). | | | | | below). |
| | | | | | | | | |
| response-processing | O | M | Data on response processing. Map | | response-processing | | M | Data on response processing. Map |
| -data | | | of type "ResponseProcessingData", | | -data | | | of type "ResponseProcessingData", |
| | | | see Section 7.6.1. | | | | | see Section 7.6.1. |
| | | | | | | | | |
| query-extended | O | M | Extended Query data. Map of type | | query-extended | | M | Extended Query data. Map of type |
| | | | "QueryResponseExtended", see | | | | | "QueryResponseExtended", see |
| | | | Section 7.6.2. | | | | | Section 7.6.2. |
| | | | | | | | | |
| response-extended | O | M | Extended Response data. Map of | | response-extended | | M | Extended Response data. Map of |
| | | | type "QueryResponseExtended", see | | | | | type "QueryResponseExtended", see |
| | | | Section 7.6.2. | | | | | Section 7.6.2. |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
The "query-size" and "response-size" fields hold the DNS message The "query-size" and "response-size" fields hold the DNS message
size. For UDP this is the size of the UDP payload that contained the size. For UDP this is the size of the UDP payload that contained the
DNS message. For TCP it is the size of the DNS message as specified DNS message. For TCP it is the size of the DNS message as specified
in the two-byte message length header. Trailing bytes in UDP queries in the two-byte message length header. Trailing bytes in UDP queries
are routinely observed in traffic to authoritative servers and this are routinely observed in traffic to authoritative servers and this
value allows a calculation of how many trailing bytes were present. value allows a calculation of how many trailing bytes were present.
7.6.1. "ResponseProcessingData" 7.6.1. "ResponseProcessingData"
Information on the server processing that produced the response. A Information on the server processing that produced the response. A
map containing the following: map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| bailiwick-index | O | U | The index in the "name-rdata" array of | | bailiwick-index | | U | The index in the "name-rdata" array of |
| | | | the owner name for the response | | | | | the owner name for the response |
| | | | bailiwick. See Section 7.5.3. | | | | | bailiwick. See Section 7.5.3. |
| | | | | | | | | |
| processing-flags | O | U | Flags relating to response processing. | | processing-flags | | U | Flags relating to response processing. |
| | | | Bit 0. 1 if the response came from | | | | | Bit 0. 1 if the response came from |
| | | | cache. | | | | | cache. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.6.2. "QueryResponseExtended" 7.6.2. "QueryResponseExtended"
Extended data on the Q/R data item. Extended data on the Q/R data item.
Each item in the map is present only if collection of the relevant Each item in the map is present only if collection of the relevant
details is configured. details is configured.
A map containing the following items: A map containing the following items:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| question-index | O | U | The index in the "qlist" array of the | | question-index | | U | The index in the "qlist" array of the |
| | | | entry listing any second and | | | | | entry listing any second and |
| | | | subsequent Questions in the Question | | | | | subsequent Questions in the Question |
| | | | section for the Query or Response. See | | | | | section for the Query or Response. See |
| | | | Section 7.5.3. | | | | | Section 7.5.3. |
| | | | | | | | | |
| answer-index | O | U | The index in the "rrlist" array of the | | answer-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Answer Resource | | | | | entry listing the Answer Resource |
| | | | Record sections for the Query or | | | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. | | | | | Response. See Section 7.5.3. |
| | | | | | | | | |
| authority-index | O | U | The index in the "rrlist" array of the | | authority-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Authority Resource | | | | | entry listing the Authority Resource |
| | | | Record sections for the Query or | | | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. | | | | | Response. See Section 7.5.3. |
| | | | | | | | | |
| additional-index | O | U | The index in the "rrlist" array of the | | additional-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Additional Resource | | | | | entry listing the Additional Resource |
| | | | Record sections for the Query or | | | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. Note that | | | | | Response. See Section 7.5.3. Note that |
| | | | Query OPT RR data can be optionally | | | | | Query OPT RR data can be optionally |
| | | | stored in the QuerySignature. | | | | | stored in the QuerySignature. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.7. "AddressEventCount" 7.7. "AddressEventCount"
Counts of various IP related events relating to traffic with Counts of various IP related events relating to traffic with
individual client addresses. A map containing the following: individual client addresses. A map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| ae-type | M | U | The type of event. The following | | ae-type | X | U | The type of event. The following |
| | | | events types are currently defined: | | | | | events types are currently defined: |
| | | | 0. TCP reset. | | | | | 0. TCP reset. |
| | | | 1. ICMP time exceeded. | | | | | 1. ICMP time exceeded. |
| | | | 2. ICMP destination unreachable. | | | | | 2. ICMP destination unreachable. |
| | | | 3. ICMPv6 time exceeded. | | | | | 3. ICMPv6 time exceeded. |
| | | | 4. ICMPv6 destination unreachable. | | | | | 4. ICMPv6 destination unreachable. |
| | | | 5. ICMPv6 packet too big. | | | | | 5. ICMPv6 packet too big. |
| | | | | | | | | |
| ae-code | O | U | A code relating to the event. | | ae-code | | U | A code relating to the event. For ICMP |
| | | | or ICMPv6 events, this MUST be the |
| | | | ICMP [RFC0792] or ICMPv6 [RFC4443] |
| | | | code. For other events the contents |
| | | | are undefined. |
| | | | | | | | | |
| ae-address-index | M | U | The index in the "ip-address" array of | | ae-address-index | X | U | The index in the "ip-address" array of |
| | | | the client address. See Section 7.5.3. | | | | | the client address. See Section 7.5.3. |
| | | | | | | | | |
| ae-count | M | U | The number of occurrences of this | | ae-count | X | U | The number of occurrences of this |
| | | | event during the block collection | | | | | event during the block collection |
| | | | period. | | | | | period. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.8. "MalformedMessage" 7.8. "MalformedMessage"
Details of malformed messages. A map containing the following: Details of malformed messages. A map containing the following:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | O | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| time-offset | O | U | Message timestamp as an offset in | | time-offset | | U | Message timestamp as an offset in |
| | | | ticks from "earliest-time". | | | | | ticks (see Section 7.4.1.1) from |
| | | | "earliest-time". |
| | | | | | | | | |
| client-address-index | O | U | The index in the "ip-address" | | client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. | | | | | array of the client IP address. |
| | | | See Section 7.5.3. | | | | | See Section 7.5.3. |
| | | | | | | | | |
| client-port | O | U | The client port. | | client-port | | U | The client port. |
| | | | | | | | | |
| message-data-index | O | U | The index in the "malformed- | | message-data-index | | U | The index in the "malformed- |
| | | | message-data" array of the message | | | | | message-data" array of the message |
| | | | data for this message. See Section | | | | | data for this message. See Section |
| | | | 7.5.3. | | | | | 7.5.3. |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
8. Versioning 8. Versioning
The C-DNS file preamble includes a file format version; a major and The C-DNS file preamble includes a file format version; a major and
minor version number are required fields. The document defines minor version number are required fields. The document defines
version 1.0 of the C-DNS specification. This section describes the version 1.0 of the C-DNS specification. This section describes the
skipping to change at page 37, line 9 skipping to change at page 38, line 9
Some transport information is not captured in the C-DNS format. For Some transport information is not captured in the C-DNS format. For
example, the following aspects of the original packet stream cannot example, the following aspects of the original packet stream cannot
be re-constructed from the C-DNS format: be re-constructed from the C-DNS format:
o IP fragmentation o IP fragmentation
o TCP stream information: o TCP stream information:
* Multiple DNS messages may have been sent in a single TCP * Multiple DNS messages may have been sent in a single TCP
segment segment
* A DNS payload may have be split across multiple TCP segments * A DNS payload may have been split across multiple TCP segments
* Multiple DNS messages may have be sent on a single TCP session * Multiple DNS messages may have been sent on a single TCP
session
o Malformed DNS messages if the wire format is not recorded o Malformed DNS messages if the wire format is not recorded
o Any Non-DNS messages that were in the original packet stream e.g. o Any Non-DNS messages that were in the original packet stream e.g.
ICMP ICMP
Simple assumptions can be made on the reconstruction: fragmented and Simple assumptions can be made on the reconstruction: fragmented and
DNS-over-TCP messages can be reconstructed into single packets and a DNS-over-TCP messages can be reconstructed into single packets and a
single TCP session can be constructed for each TCP packet. single TCP session can be constructed for each TCP packet.
Additionally, if malformed messages and Non-DNS packets are captured Additionally, if malformed messages and Non-DNS packets are captured
separately, they can be merged with packet captures reconstructed separately, they can be merged with packet captures reconstructed
from C-DNS to produce a more complete packet stream. from C-DNS to produce a more complete packet stream.
9.1. Name compression 9.1. Name compression
All the names stored in the C-DNS format are full domain names; no All the names stored in the C-DNS format are full domain names; no
DNS style name compression is used on the individual names within the [RFC1035] name compression is used on the individual names within the
format. Therefore when reconstructing a packet, name compression format. Therefore when reconstructing a packet, name compression
must be used in order to reproduce the on the wire representation of must be used in order to reproduce the on the wire representation of
the packet. the packet.
[RFC1035] name compression works by substituting trailing sections of [RFC1035] name compression works by substituting trailing sections of
a name with a reference back to the occurrence of those sections a name with a reference back to the occurrence of those sections
earlier in the message. Not all name server software uses the same earlier in the message. Not all name server software uses the same
algorithm when compressing domain names within the responses. Some algorithm when compressing domain names within the responses. Some
attempt maximum recompression at the expense of runtime resources, attempt maximum recompression at the expense of runtime resources,
others use heuristics to balance compression and speed and others use others use heuristics to balance compression and speed and others use
skipping to change at page 38, line 49 skipping to change at page 39, line 50
implementation. implementation.
It should be noted that packet capture libraries do not necessarily It should be noted that packet capture libraries do not necessarily
provide packets in strict chronological order. This can, for provide packets in strict chronological order. This can, for
example, arise on multi-core platforms where packets arriving at a example, arise on multi-core platforms where packets arriving at a
network device are processed by different cores. On systems where network device are processed by different cores. On systems where
this behaviour has been observed, the timestamps associated with each this behaviour has been observed, the timestamps associated with each
packet are consistent; queries always have a timestamp prior to the packet are consistent; queries always have a timestamp prior to the
response timestamp. However, the order in which these packets appear response timestamp. However, the order in which these packets appear
in the packet capture stream is not necessarily strictly in the packet capture stream is not necessarily strictly
choronological; a response can appear in the capture stream before chronological; a response can appear in the capture stream before the
the query that provoked the response. For this discussion, this non- query that provoked the response. For this discussion, this non-
chronological delivery is termed "skew". chronological delivery is termed "skew".
In the presence of skew, a response packets can arrive for matching In the presence of skew, a response packets can arrive for matching
before the corresponding query. To avoid generating false instances before the corresponding query. To avoid generating false instances
of responses without a matching query, and queries without a matching of responses without a matching query, and queries without a matching
response, the matching algorithm must take account of the possibility response, the matching algorithm must take account of the possibility
of skew. of skew.
10.1. Matching algorithm 10.1. Matching algorithm
skipping to change at page 43, line 48 skipping to change at page 44, line 48
Whilst this document makes no specific recommendations with respect Whilst this document makes no specific recommendations with respect
to Canonical CBOR (see Section 3.9 of [RFC7049]) the following to Canonical CBOR (see Section 3.9 of [RFC7049]) the following
guidance may be of use to implementors. guidance may be of use to implementors.
Adherence to the first two rules given in Section 3.9 of [RFC7049] Adherence to the first two rules given in Section 3.9 of [RFC7049]
will minimise file sizes. will minimise file sizes.
Adherence to the last two rules given in Section 3.9 of [RFC7049] for Adherence to the last two rules given in Section 3.9 of [RFC7049] for
all maps and arrays would unacceptably constrain implementations, for all maps and arrays would unacceptably constrain implementations, for
example, in the use case of real-time data collection in constrained example, in the use case of real-time data collection in constrained
environments. environments where outputting block tables after query/response data
and allowing indefinite length maps and arrays could reduce memory
requirements.
11.1. Optional data 11.1. Optional data
When decoding C-DNS data some of the items required for a particular When decoding C-DNS data some of the items required for a particular
function that the consumer wishes to perform may be missing. function that the consumer wishes to perform may be missing.
Consumers should consider providing configurable default values to be Consumers should consider providing configurable default values to be
used in place of the missing values in their output. used in place of the missing values in their output.
11.2. Trailing bytes 11.2. Trailing bytes
skipping to change at page 44, line 45 skipping to change at page 45, line 45
subsequent messages in the packet being considered malformed. subsequent messages in the packet being considered malformed.
This will not generate a trailing bytes record for the processed This will not generate a trailing bytes record for the processed
well-formed message. well-formed message.
11.3. Limiting collection of RDATA 11.3. Limiting collection of RDATA
Implementations should consider providing a configurable maximum Implementations should consider providing a configurable maximum
RDATA size for capture, for example, to avoid memory issues when RDATA size for capture, for example, to avoid memory issues when
confronted with large XFR records. confronted with large XFR records.
11.4. Timestamps
The preamble to each block includes a timestamp of the earliest
record in the block. As described in Section 7.5.1, the timestamp is
an array of 2 unsigned integers. The first is a POSIX "time_t"
[posix-time]. Consumers of C-DNS should be aware of this as it
excludes leap seconds and therefore may cause minor anomalies in the
data e.g. when calculating query throughput.
12. Implementation status 12. Implementation status
[Note to RFC Editor: please remove this section and reference to [Note to RFC Editor: please remove this section and reference to
[RFC7942] prior to publication.] [RFC7942] prior to publication.]
This section records the status of known implementations of the This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942]. Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to The description of implementations in this section is intended to
assist the IETF in its decision processes in progressing drafts to assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation RFCs. Please note that the listing of any individual implementation
here does not imply endorsement by the IETF. Furthermore, no effort here does not imply endorsement by the IETF. Furthermore, no effort
has been spent to verify the information presented here that was has been spent to verify the information presented here that was
supplied by IETF contributors. This is not intended as, and must not supplied by IETF contributors. This is not intended as, and must not
be construed to be, a catalog of available implementations or their be construed to be, a catalog of available implementations or their
features. Readers are advised to note that other implementations may features. Readers are advised to note that other implementations may
exist. exist.
skipping to change at page 46, line 11 skipping to change at page 47, line 17
13. IANA considerations 13. IANA considerations
None None
14. Security considerations 14. Security considerations
Any control interface MUST perform authentication and encryption. Any control interface MUST perform authentication and encryption.
Any data upload MUST be authenticated and encrypted. Any data upload MUST be authenticated and encrypted.
15. Acknowledgements 15. Privacy considerations
Storage of DNS traffic by operators in PCAP and other formats is a
long standing and widespread practice. Section 2.5 of
[I-D.bortzmeyer-dprive-rfc7626-bis] is an analysis of the risks to
Internet users of the storage of DNS traffic data in servers
(recursive resolvers, authoritative and rogue servers).
Section 5.2 of [I-D.dickinson-dprive-bcp-op] describes mitigations
for those risks for data stored on recursive resolvers (but which
could by extension apply to authoritative servers). These include
data handling practices and methods for data minimization, IP address
pseudonymization and anonymization. Appendix B of that document
presents an analysis of 7 published anonymization processes. In
addition, RSSAC have recently published RSSAC04: [5] "
Recommendations on Anonymization Processes for Source IP Addresses
Submitted for Future Analysis".
The above analyses consider full data capture (e.g using PCAP) as a
baseline for privacy considerations and therefore this format
specification introduces no new user privacy issues beyond those of
full data capture (which are quite severe). It does provides
mechanisms to selectively record only certain fields at the time of
data capture to improve user privacy and to explicitly indicate that
data is sampled and or anonymized. It also provide flags to indicate
if data normalization has been performed; data normalization
increases user privacy by reducing the potential for fingerprinting
individuals, however, a trade-off is potentially reducing the
capacity to identify attack traffic via query name signatures.
Operators should carefully consider their operational requirements
and privacy policies and SHOULD capture at source the minimum user
data required to meet their needs.
16. Acknowledgements
The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for
many useful discussions on binary formats, compression and packet many useful discussions on binary formats, compression and packet
matching. Also Jan Vcelak and Wouter Wijngaards for discussions on matching. Also Jan Vcelak and Wouter Wijngaards for discussions on
name compression and Paul Hoffman for a detailed review of the name compression and Paul Hoffman for a detailed review of the
document and the C-DNS CDDL. document and the C-DNS CDDL.
Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson, Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson,
Stephane Bortzmeyer and many other members of DNSOP for review. Stephane Bortzmeyer and many other members of DNSOP for review.
Also, Miek Gieben for mmark [5] Also, Miek Gieben for mmark [6]
16. Changelog 17. Changelog
draft-ietf-dnsop-dns-capture-format-09
o Editorial changes arising from IESG review
o *-transport-flags and may be mandatory in some configurations
o Mark fields that are conditionally mandatory
o Change `promisc' flag CDDL data type to boolean
o Add ranges to configuration quantities where appropriate
draft-ietf-dnsop-dns-capture-format-08 draft-ietf-dnsop-dns-capture-format-08
o Convert diagrams to ASCII o Convert diagrams to ASCII
o Describe versioning o Describe versioning
o Fix unused group warning in CDDL o Fix unused group warning in CDDL
draft-ietf-dnsop-dns-capture-format-07 draft-ietf-dnsop-dns-capture-format-07
skipping to change at page 46, line 47 skipping to change at page 49, line 4
o Make RR RDATA optional o Make RR RDATA optional
o Update matching diagram and explain skew o Update matching diagram and explain skew
o Add count of discarded messages to block statistics o Add count of discarded messages to block statistics
o Editorial clarifications and improvements o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-06 draft-ietf-dnsop-dns-capture-format-06
o Correct BlockParameters type to map o Correct BlockParameters type to map
o Make RR ttl optional o Make RR ttl optional
o Add storage flag indicating name normalisation o Add storage flag indicating name normalization
o Add storage parameter fields for sampling and anonymisation
o Add storage parameter fields for sampling and anonymization
methods methods
o Editorial clarifications and improvements o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-05 draft-ietf-dnsop-dns-capture-format-05
o Make all data items in Q/R, QuerySignature and Malformed Message o Make all data items in Q/R, QuerySignature and Malformed Message
arrays optional arrays optional
o Re-structure the FilePreamble and ConfigurationParameters into o Re-structure the FilePreamble and ConfigurationParameters into
BlockParameters BlockParameters
o BlockParameters has separate Storage and Collection Parameters o BlockParameters has separate Storage and Collection Parameters
o Storage Parameters includes information on what optional fields o Storage Parameters includes information on what optional fields
are present, and flags specifying anonymisation or sampling are present, and flags specifying anonymization or sampling
o Addresses can now be stored as prefixes. o Addresses can now be stored as prefixes.
o Switch to using a variable sub-second timing granularity o Switch to using a variable sub-second timing granularity
o Add response bailiwick and query response type o Add response bailiwick and query response type
o Add specifics of how to record malformed messages o Add specifics of how to record malformed messages
o Add implementation guidance o Add implementation guidance
skipping to change at page 49, line 4 skipping to change at page 51, line 11
messages within C-DNS and what this means for packet matching. messages within C-DNS and what this means for packet matching.
This may influence which fields are optional in the rest of the This may influence which fields are optional in the rest of the
representation. representation.
o Added section on design goals to Introduction o Added section on design goals to Introduction
o Added a TODO: Can Class be optimised? Should a class of IN be o Added a TODO: Can Class be optimised? Should a class of IN be
inferred if not present? inferred if not present?
draft-dickinson-dnsop-dns-capture-format-00 draft-dickinson-dnsop-dns-capture-format-00
o Initial commit o Initial commit
17. References 18. References
17.1. Normative References 18.1. Normative References
[I-D.ietf-cbor-cddl]
Birkholz, H., Vigano, C., and C. Bormann, "Concise data
definition language (CDDL): a notational convention to
express CBOR and JSON data structures", draft-ietf-cbor-
cddl-06 (work in progress), November 2018.
[pcap-filter]
tcpdump.org, "Manpage of PCAP-FILTER", 2017,
<http://www.tcpdump.org/manpages/pcap-filter.7.html>.
[pcap-options]
tcpdump.org, "Manpage of PCAP", 2018,
<http://www.tcpdump.org/manpages/pcap.3pcap.html>.
[posix-time]
The Open Group, "Section 4.16, Base Definitions, Standard
for Information Technology - Portable Operating System
Interface (POSIX(R)) Base Specifications, Issue 7", IEEE
Standard 1003.1 2017 Edition,
DOI 10.1109/IEEESTD.2018.8277153, 2017.
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, DOI 10.17487/RFC0792, September 1981,
<https://www.rfc-editor.org/info/rfc792>.
[RFC1035] Mockapetris, P., "Domain names - implementation and [RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>. November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, <https://www.rfc- DOI 10.17487/RFC2119, March 1997, <https://www.rfc-
editor.org/info/rfc2119>. editor.org/info/rfc2119>.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifier (URI): Generic Syntax", STD 66,
RFC 3986, DOI 10.17487/RFC3986, January 2005,
<https://www.rfc-editor.org/info/rfc3986>.
[RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet
Control Message Protocol (ICMPv6) for the Internet
Protocol Version 6 (IPv6) Specification", STD 89,
RFC 4443, DOI 10.17487/RFC4443, March 2006,
<https://www.rfc-editor.org/info/rfc4443>.
[RFC6891] Damas, J., Graff, M., and P. Vixie, "Extension Mechanisms
for DNS (EDNS(0))", STD 75, RFC 6891,
DOI 10.17487/RFC6891, April 2013, <https://www.rfc-
editor.org/info/rfc6891>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <https://www.rfc-editor.org/info/rfc7049>. October 2013, <https://www.rfc-editor.org/info/rfc7049>.
17.2. Informative References [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D.,
and P. Hoffman, "Specification for DNS over Transport
Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May
2016, <https://www.rfc-editor.org/info/rfc7858>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS
(DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018,
<https://www.rfc-editor.org/info/rfc8484>.
18.2. Informative References
[ditl] DNS-OARC, "DITL", 2016, <https://www.dns- [ditl] DNS-OARC, "DITL", 2016, <https://www.dns-
oarc.net/oarc/data/ditl>. oarc.net/oarc/data/ditl>.
[dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/ [dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
dnscap>. dnscap>.
[dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>.
[dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
<https://www.dns-oarc.net/tools/dsc>. <https://www.dns-oarc.net/tools/dsc>.
[I-D.bortzmeyer-dprive-rfc7626-bis]
Bortzmeyer, S. and S. Dickinson, "DNS Privacy
Considerations", draft-bortzmeyer-dprive-rfc7626-bis-01
(work in progress), July 2018.
[I-D.daley-dnsxml] [I-D.daley-dnsxml]
Daley, J., Morris, S., and J. Dickinson, "dnsxml - A Daley, J., Morris, S., and J. Dickinson, "dnsxml - A
standard XML representation of DNS data", draft-daley- standard XML representation of DNS data", draft-daley-
dnsxml-00 (work in progress), July 2013. dnsxml-00 (work in progress), July 2013.
[I-D.hoffman-dns-in-json] [I-D.dickinson-dprive-bcp-op]
Hoffman, P., "Representing DNS Messages in JSON", draft- Dickinson, S., Overeinder, B., Rijswijk-Deij, R., and A.
hoffman-dns-in-json-16 (work in progress), May 2018. Mankin, "Recommendations for DNS Privacy Service
Operators", draft-dickinson-dprive-bcp-op-01 (work in
progress), July 2018.
[I-D.ietf-cbor-cddl] [IEEE802.1Q]
Birkholz, H., Vigano, C., and C. Bormann, "Concise data IEEE, "IEEE Standard for Local and metropolitan area
definition language (CDDL): a notational convention to networks -- Bridges and Bridged Networks",
express CBOR data structures", draft-ietf-cbor-cddl-03 DOI 10.1109/IEEESTD.2014.6991462, 2014.
(work in progress), July 2018.
[opcodes] IANA, "DNS OpCodes", 2018,
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-5>.
[packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
2014, <https://github.com/dotse/PacketQ>. 2014, <https://github.com/dotse/PacketQ>.
[pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.
[pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
Harris, "pcap-ng", 2016, <https://github.com/pcapng/ Harris, "pcap-ng", 2016, <https://github.com/pcapng/
pcapng>. pcapng>.
[rcodes] IANA, "DNS RCODEs", 2018,
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-6>.
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205, Code: The Implementation Status Section", BCP 205,
RFC 7942, DOI 10.17487/RFC7942, July 2016, RFC 7942, DOI 10.17487/RFC7942, July 2016,
<https://www.rfc-editor.org/info/rfc7942>. <https://www.rfc-editor.org/info/rfc7942>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259, Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017, <https://www.rfc- DOI 10.17487/RFC8259, December 2017, <https://www.rfc-
editor.org/info/rfc8259>. editor.org/info/rfc8259>.
17.3. URIs [RFC8427] Hoffman, P., "Representing DNS Messages in JSON",
RFC 8427, DOI 10.17487/RFC8427, July 2018,
<https://www.rfc-editor.org/info/rfc8427>.
[rrclasses]
IANA, "DNS CLASSes", 2018,
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-2>.
[rrtypes] IANA, "Resource Record (RR) TYPEs", 2018,
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-4>.
18.3. URIs
[1] https://github.com/dns-stats/compactor/wiki [1] https://github.com/dns-stats/compactor/wiki
[2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users [2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
[3] https://www.sinodun.com/2017/06/compressing-pcap-files/ [3] https://www.sinodun.com/2017/06/compressing-pcap-files/
[4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu- [4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu-
trusty-packet-capture-woes/ trusty-packet-capture-woes/
[5] https://github.com/miekg/mmark [5] https://www.icann.org/en/system/files/files/rssac-
040-07aug18-en.pdf
[6] https://www.nlnetlabs.nl/projects/nsd/ [6] https://github.com/miekg/mmark
[7] https://www.knot-dns.cz/ [7] https://www.nlnetlabs.nl/projects/nsd/
[8] https://avro.apache.org/ [8] https://www.knot-dns.cz/
[9] https://developers.google.com/protocol-buffers/ [9] https://avro.apache.org/
[10] http://cbor.io [10] https://developers.google.com/protocol-buffers/
[11] https://github.com/kubo/snzip [11] http://cbor.io
[12] http://google.github.io/snappy/ [12] https://github.com/kubo/snzip
[13] http://lz4.github.io/lz4/ [13] http://google.github.io/snappy/
[14] http://www.gzip.org/ [14] http://lz4.github.io/lz4/
[15] http://facebook.github.io/zstd/ [15] http://www.gzip.org/
[16] http://tukaani.org/xz/ [16] http://facebook.github.io/zstd/
[17] https://github.com/dns-stats/draft-dns-capture- [17] http://tukaani.org/xz/
format/blob/master/file-size-versus-block-size.png
[18] https://github.com/dns-stats/draft-dns-capture- [18] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.png
[19] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.svg format/blob/master/file-size-versus-block-size.svg
Appendix A. CDDL Appendix A. CDDL
This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for
C-DNS. C-DNS.
CDDL does not permit a range of allowed values to be specified for a CDDL does not permit a range of allowed values to be specified for a
bitfield. Where necessary, those values are given as a CDDL group, bitfield. Where necessary, those values are given as a CDDL group,
but the group definition is commented out to prevent CDDL tooling but the group definition is commented out to prevent CDDL tooling
from warning that the group is unused. from warning that the group is unused.
; CDDL specification of the file format for C-DNS, ; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS messages and ; which describes a collection of DNS messages and
; traffic meta-data. ; traffic meta-data.
; ;
; The overall structure of a file. ; The overall structure of a file.
; ;
File = [ File = [
file-type-id : tstr .regexp "C-DNS", file-type-id : "C-DNS",
file-preamble : FilePreamble, file-preamble : FilePreamble,
file-blocks : [* Block], file-blocks : [* Block],
] ]
; ;
; The file preamble. ; The file preamble.
; ;
FilePreamble = { FilePreamble = {
major-format-version => uint .eq 1, major-format-version => 1,
minor-format-version => uint .eq 0, minor-format-version => 0,
? private-version => uint, ? private-version => uint,
block-parameters => [+ BlockParameters], block-parameters => [+ BlockParameters],
} }
major-format-version = 0 major-format-version = 0
minor-format-version = 1 minor-format-version = 1
private-version = 2 private-version = 2
block-parameters = 3 block-parameters = 3
BlockParameters = {
storage-parameters => StorageParameters,
? collection-parameters => CollectionParameters,
}
storage-parameters = 0
collection-parameters = 1
StorageParameters = { BlockParameters = {
ticks-per-second => uint, storage-parameters => StorageParameters,
max-block-items => uint, ? collection-parameters => CollectionParameters,
storage-hints => StorageHints,
opcodes => [+ uint],
rr-types => [+ uint],
? storage-flags => StorageFlags,
? client-address-prefix-ipv4 => uint,
? client-address-prefix-ipv6 => uint,
? server-address-prefix-ipv4 => uint,
? server-address-prefix-ipv6 => uint,
? sampling-method => tstr,
? anonymisation-method => tstr,
}
ticks-per-second = 0
max-block-items = 1
storage-hints = 2
opcodes = 3
rr-types = 4
storage-flags = 5
client-address-prefix-ipv4 = 6
client-address-prefix-ipv6 = 7
server-address-prefix-ipv4 = 8
server-address-prefix-ipv6 = 9
sampling-method = 10
anonymisation-method = 11
; A hint indicates if the collection method will output the }
; item or will ignore the item if present. storage-parameters = 0
StorageHints = { collection-parameters = 1
query-response-hints => QueryResponseHints,
query-response-signature-hints => QueryResponseSignatureHints,
rr-hints => RRHints,
other-data-hints => OtherDataHints,
}
query-response-hints = 0
query-response-signature-hints = 1
rr-hints = 2
other-data-hints = 3
QueryResponseHintValues = &( IPv6PrefixLength = 1..128
time-offset : 0, IPv4PrefixLength = 1..32
client-address-index : 1, OpcodeRange = 0..15
client-port : 2, RRTypeRange = 0..65535
transaction-id : 3,
qr-signature-index : 4,
client-hoplimit : 5,
response-delay : 6,
query-name-index : 7,
query-size : 8,
response-size : 9,
response-processing-data : 10,
query-question-sections : 11, ; Second & subsequent questions
query-answer-sections : 12,
query-authority-sections : 13,
query-additional-sections : 14,
response-answer-sections : 15,
response-authority-sections : 16,
response-additional-sections : 17,
)
QueryResponseHints = uint .bits QueryResponseHintValues
QueryResponseSignatureHintValues = &( StorageParameters = {
server-address : 0, ticks-per-second => uint,
server-port : 1, max-block-items => uint,
qr-transport-flags : 2, storage-hints => StorageHints,
qr-type : 3, opcodes => [+ OpcodeRange],
qr-sig-flags : 4, rr-types => [+ RRTypeRange],
query-opcode : 5, ? storage-flags => StorageFlags,
dns-flags : 6, ? client-address-prefix-ipv4 => IPv4PrefixLength,
query-rcode : 7, ? client-address-prefix-ipv6 => IPv6PrefixLength,
query-class-type : 8, ? server-address-prefix-ipv4 => IPv4PrefixLength,
query-qdcount : 9, ? server-address-prefix-ipv6 => IPv6PrefixLength,
query-ancount : 10, ? sampling-method => tstr,
query-arcount : 11, ? anonymisation-method => tstr,
query-nscount : 12, }
query-edns-version : 13, ticks-per-second = 0
query-udp-size : 14, max-block-items = 1
query-opt-rdata : 15, storage-hints = 2
response-rcode : 16, opcodes = 3
) rr-types = 4
QueryResponseSignatureHints = uint .bits QueryResponseSignatureHintValues storage-flags = 5
client-address-prefix-ipv4 = 6
client-address-prefix-ipv6 = 7
server-address-prefix-ipv4 = 8
server-address-prefix-ipv6 = 9
sampling-method = 10
anonymisation-method = 11
RRHintValues = &( ; A hint indicates if the collection method will output the
ttl : 0, ; item or will ignore the item if present.
rdata-index : 1, StorageHints = {
) query-response-hints => QueryResponseHints,
RRHints = uint .bits RRHintValues query-response-signature-hints =>
OtherDataHintValues = &( QueryResponseSignatureHints,
malformed-messages : 0, rr-hints => RRHints,
address-event-counts : 1, other-data-hints => OtherDataHints,
) }
OtherDataHints = uint .bits OtherDataHintValues query-response-hints = 0
query-response-signature-hints = 1
rr-hints = 2
other-data-hints = 3
StorageFlagValues = &( QueryResponseHintValues = &(
anonymised-data : 0, time-offset : 0,
sampled-data : 1, client-address-index : 1,
normalised-names : 2, client-port : 2,
) transaction-id : 3,
StorageFlags = uint .bits StorageFlagValues qr-signature-index : 4,
client-hoplimit : 5,
response-delay : 6,
query-name-index : 7,
query-size : 8,
response-size : 9,
response-processing-data : 10,
query-question-sections : 11, ; Second & subsequent
; questions
query-answer-sections : 12,
query-authority-sections : 13,
query-additional-sections : 14,
response-answer-sections : 15,
response-authority-sections : 16,
response-additional-sections : 17,
)
QueryResponseHints = uint .bits QueryResponseHintValues
CollectionParameters = { QueryResponseSignatureHintValues = &(
? query-timeout => uint, server-address : 0,
? skew-timeout => uint, server-port : 1,
? snaplen => uint, qr-transport-flags : 2,
? promisc => uint, qr-type : 3,
? interfaces => [+ tstr], qr-sig-flags : 4,
? server-addresses => [+ IPAddress], ; Hint for later analysis query-opcode : 5,
? vlan-ids => [+ uint], dns-flags : 6,
? filter => tstr, query-rcode : 7,
? generator-id => tstr, query-class-type : 8,
? host-id => tstr, query-qdcount : 9,
} query-ancount : 10,
query-timeout = 0 query-arcount : 11,
skew-timeout = 1 query-nscount : 12,
snaplen = 2 query-edns-version : 13,
promisc = 3 query-udp-size : 14,
interfaces = 4 query-opt-rdata : 15,
server-addresses = 5 response-rcode : 16,
vlan-ids = 6 )
filter = 7 QueryResponseSignatureHints =
generator-id = 8 uint .bits QueryResponseSignatureHintValues
host-id = 9
; RRHintValues = &(
; Data in the file is stored in Blocks. ttl : 0,
; rdata-index : 1,
Block = { )
block-preamble => BlockPreamble, RRHints = uint .bits RRHintValues
? block-statistics => BlockStatistics, ; Much of this could be derived
? block-tables => BlockTables,
? query-responses => [+ QueryResponse],
? address-event-counts => [+ AddressEventCount],
? malformed-messages => [+ MalformedMessage],
}
block-preamble = 0
block-statistics = 1
block-tables = 2
query-responses = 3
address-event-counts = 4
malformed-messages = 5
; OtherDataHintValues = &(
; The (mandatory) preamble to a block. malformed-messages : 0,
; address-event-counts : 1,
BlockPreamble = { )
? earliest-time => Timestamp, OtherDataHints = uint .bits OtherDataHintValues
? block-parameters-index => uint .default 0,
}
earliest-time = 0
block-parameters-index = 1
; Ticks are subsecond intervals. The number of ticks in a second is file/block StorageFlagValues = &(
; metadata. Signed and unsigned tick types are defined. anonymised-data : 0,
ticks = int sampled-data : 1,
uticks = uint normalised-names : 2,
)
StorageFlags = uint .bits StorageFlagValues
Timestamp = [ ; Hints for later analysis.
timestamp-secs : uint, VLANIdRange = 1..4094
timestamp-uticks : uticks,
]
; CollectionParameters = {
; Statistics about the block contents. ? query-timeout => uint,
; ? skew-timeout => uint,
BlockStatistics = { ? snaplen => uint,
? processed-messages => uint, ? promisc => bool,
? qr-data-items => uint, ? interfaces => [+ tstr],
? unmatched-queries => uint, ? server-addresses => [+ IPAddress],
? unmatched-responses => uint, ? vlan-ids => [+ VLANIdRange],
? discarded-opcode => uint, ? filter => tstr,
? malformed-items => uint, ? generator-id => tstr,
} ? host-id => tstr,
processed-messages = 0 }
qr-data-items = 1 query-timeout = 0
unmatched-queries = 2 skew-timeout = 1
unmatched-responses = 3 snaplen = 2
discarded-opcode = 4 promisc = 3
malformed-items = 5 interfaces = 4
server-addresses = 5
vlan-ids = 6
filter = 7
generator-id = 8
host-id = 9
; ;
; Tables of common data referenced from records in a block. ; Data in the file is stored in Blocks.
; ;
BlockTables = { Block = {
? ip-address => [+ IPAddress], block-preamble => BlockPreamble,
? classtype => [+ ClassType], ? block-statistics => BlockStatistics, ; Much of this
? name-rdata => [+ bstr], ; Holds both Name RDATA and RDATA ; could be derived
? qr-sig => [+ QueryResponseSignature], ? block-tables => BlockTables,
? QuestionTables, ? query-responses => [+ QueryResponse],
? RRTables, ? address-event-counts => [+ AddressEventCount],
? malformed-message-data => [+ MalformedMessageData], ? malformed-messages => [+ MalformedMessage],
} }
ip-address = 0 block-preamble = 0
classtype = 1 block-statistics = 1
name-rdata = 2 block-tables = 2
qr-sig = 3 query-responses = 3
qlist = 4 address-event-counts = 4
qrr = 5 malformed-messages = 5
rrlist = 6
rr = 7
malformed-message-data = 8
IPv4Address = bstr .size 4 ;
IPv6Address = bstr .size 16 ; The (mandatory) preamble to a block.
IPAddress = IPv4Address / IPv6Address ;
BlockPreamble = {
? earliest-time => Timestamp,
? block-parameters-index => uint .default 0,
}
earliest-time = 0
block-parameters-index = 1
ClassType = { ; Ticks are subsecond intervals. The number of ticks in a second is
type => uint, ; file/block metadata. Signed and unsigned tick types are defined.
class => uint, ticks = int
} uticks = uint
type = 0
class = 1
QueryResponseSignature = { Timestamp = [
? server-address-index => uint, timestamp-secs : uint,
? server-port => uint, timestamp-uticks : uticks,
? qr-transport-flags => QueryResponseTransportFlags, ]
? qr-type => QueryResponseType,
? qr-sig-flags => QueryResponseFlags,
? query-opcode => uint,
? qr-dns-flags => DNSFlags,
? query-rcode => uint,
? query-classtype-index => uint,
? query-qd-count => uint,
? query-an-count => uint,
? query-ns-count => uint,
? query-ar-count => uint,
? edns-version => uint,
? udp-buf-size => uint,
? opt-rdata-index => uint,
? response-rcode => uint,
}
server-address-index = 0
server-port = 1
qr-transport-flags = 2
qr-type = 3
qr-sig-flags = 4
query-opcode = 5
qr-dns-flags = 6
query-rcode = 7
query-classtype-index = 8
query-qd-count = 9
query-an-count = 10
query-ns-count = 12
query-ar-count = 12
edns-version = 13
udp-buf-size = 14
opt-rdata-index = 15
response-rcode = 16
; Transport gives the values that may appear in bits 1..4 of ;
; TransportFlags. There is currently no way to express this in ; Statistics about the block contents.
; CDDL, so Transport is unused. To avoid confusion when used ;
; with CDDL tools, it is commented out. BlockStatistics = {
; ? processed-messages => uint,
; Transport = &( ? qr-data-items => uint,
; udp : 0, ? unmatched-queries => uint,
; tcp : 1, ? unmatched-responses => uint,
; tls : 2, ? discarded-opcode => uint,
; dtls : 3, ? malformed-items => uint,
; ) }
processed-messages = 0
qr-data-items = 1
unmatched-queries = 2
unmatched-responses = 3
discarded-opcode = 4
malformed-items = 5
TransportFlagValues = &( ;
ip-version : 0, ; 0=IPv4, 1=IPv6 ; Tables of common data referenced from records in a block.
) / (1..4) ;
TransportFlags = uint .bits TransportFlagValues BlockTables = {
? ip-address => [+ IPAddress],
? classtype => [+ ClassType],
? name-rdata => [+ bstr], ; Holds both Names
; and RDATA
? qr-sig => [+ QueryResponseSignature],
? QuestionTables,
? RRTables,
? malformed-message-data => [+ MalformedMessageData],
}
ip-address = 0
classtype = 1
name-rdata = 2
qr-sig = 3
qlist = 4
qrr = 5
rrlist = 6
rr = 7
malformed-message-data = 8
QueryResponseTransportFlagValues = &( IPv4Address = bstr .size 4
query-trailingdata : 5, IPv6Address = bstr .size 16
) / TransportFlagValues IPAddress = IPv4Address / IPv6Address
QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues
QueryResponseType = &( ClassType = {
stub : 0, type => uint,
client : 1, class => uint,
resolver : 2, }
auth : 3, type = 0
forwarder : 4, class = 1
tool : 5,
)
QueryResponseFlagValues = &(
has-query : 0,
has-reponse : 1,
query-has-opt : 2,
response-has-opt : 3,
query-has-no-question : 4,
response-has-no-question: 5,
)
QueryResponseFlags = uint .bits QueryResponseFlagValues
DNSFlagValues = &( QueryResponseSignature = {
query-cd : 0, ? server-address-index => uint,
query-ad : 1, ? server-port => uint,
query-z : 2, ? qr-transport-flags => QueryResponseTransportFlags,
query-ra : 3, ? qr-type => QueryResponseType,
query-rd : 4, ? qr-sig-flags => QueryResponseFlags,
query-tc : 5, ? query-opcode => uint,
query-aa : 6, ? qr-dns-flags => DNSFlags,
query-do : 7, ? query-rcode => uint,
response-cd: 8, ? query-classtype-index => uint,
response-ad: 9, ? query-qd-count => uint,
response-z : 10, ? query-an-count => uint,
response-ra: 11, ? query-ns-count => uint,
response-rd: 12, ? query-ar-count => uint,
response-tc: 13, ? edns-version => uint,
response-aa: 14, ? udp-buf-size => uint,
) ? opt-rdata-index => uint,
DNSFlags = uint .bits DNSFlagValues ? response-rcode => uint,
}
server-address-index = 0
server-port = 1
qr-transport-flags = 2
qr-type = 3
qr-sig-flags = 4
query-opcode = 5
qr-dns-flags = 6
query-rcode = 7
query-classtype-index = 8
query-qd-count = 9
query-an-count = 10
query-ns-count = 12
query-ar-count = 12
edns-version = 13
udp-buf-size = 14
opt-rdata-index = 15
response-rcode = 16
QuestionTables = ( ; Transport gives the values that may appear in bits 1..4 of
qlist => [+ QuestionList], ; TransportFlags. There is currently no way to express this in
qrr => [+ Question] ; CDDL, so Transport is unused. To avoid confusion when used
) ; with CDDL tools, it is commented out.
;
; Transport = &(
; udp : 0,
; tcp : 1,
; tls : 2,
; dtls : 3,
; doh : 4,
; )
QuestionList = [+ uint] ; Index of Question TransportFlagValues = &(
ip-version : 0, ; 0=IPv4, 1=IPv6
) / (1..4)
TransportFlags = uint .bits TransportFlagValues
Question = { ; Second and subsequent questions QueryResponseTransportFlagValues = &(
name-index => uint, ; Index to a name in the name-rdata table query-trailingdata : 5,
classtype-index => uint,
}
name-index = 0
classtype-index = 1
RRTables = ( ) / TransportFlagValues
rrlist => [+ RRList], QueryResponseTransportFlags =
rr => [+ RR] uint .bits QueryResponseTransportFlagValues
)
RRList = [+ uint] ; Index of RR
RR = { QueryResponseType = &(
name-index => uint, ; Index to a name in the name-rdata table stub : 0,
classtype-index => uint, client : 1,
? ttl => uint, resolver : 2,
? rdata-index => uint, ; Index to RDATA in the name-rdata table auth : 3,
} forwarder : 4,
; Other map key values already defined above. tool : 5,
ttl = 2 )
rdata-index = 3
MalformedMessageData = { QueryResponseFlagValues = &(
? server-address-index => uint, has-query : 0,
? server-port => uint, has-reponse : 1,
? mm-transport-flags => TransportFlags, query-has-opt : 2,
? mm-payload => bstr, response-has-opt : 3,
} query-has-no-question : 4,
; Other map key values already defined above. response-has-no-question: 5,
mm-transport-flags = 2 )
mm-payload = 3 QueryResponseFlags = uint .bits QueryResponseFlagValues
; DNSFlagValues = &(
; A single query/response pair. query-cd : 0,
; query-ad : 1,
QueryResponse = { query-z : 2,
? time-offset => uticks, ; Time offset from start of block query-ra : 3,
? client-address-index => uint, query-rd : 4,
? client-port => uint, query-tc : 5,
? transaction-id => uint, query-aa : 6,
? qr-signature-index => uint, query-do : 7,
? client-hoplimit => uint, response-cd: 8,
? response-delay => ticks, response-ad: 9,
? query-name-index => uint, response-z : 10,
? query-size => uint, ; DNS size of query response-ra: 11,
? response-size => uint, ; DNS size of response response-rd: 12,
? response-processing-data => ResponseProcessingData, response-tc: 13,
? query-extended => QueryResponseExtended, response-aa: 14,
? response-extended => QueryResponseExtended, )
} DNSFlags = uint .bits DNSFlagValues
time-offset = 0
client-address-index = 1
client-port = 2
transaction-id = 3
qr-signature-index = 4
client-hoplimit = 5
response-delay = 6
query-name-index = 7
query-size = 8
response-size = 9
response-processing-data = 10
query-extended = 11
response-extended = 12
ResponseProcessingData = { QuestionTables = (
? bailiwick-index => uint, qlist => [+ QuestionList],
? processing-flags => ResponseProcessingFlags, qrr => [+ Question]
} )
bailiwick-index = 0
processing-flags = 1
ResponseProcessingFlagValues = &( QuestionList = [+ uint] ; Index of Question
from-cache : 0, Question = { ; Second and subsequent questions
) name-index => uint, ; Index to a name in the
ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues ; name-rdata table
classtype-index => uint,
}
name-index = 0
classtype-index = 1
QueryResponseExtended = { RRTables = (
? question-index => uint, ; Index of QuestionList rrlist => [+ RRList],
? answer-index => uint, ; Index of RRList rr => [+ RR]
? authority-index => uint, )
? additional-index => uint,
}
question-index = 0
answer-index = 1
authority-index = 2
additional-index = 3
; RRList = [+ uint] ; Index of RR
; Address event data.
;
AddressEventCount = {
ae-type => &AddressEventType,
? ae-code => uint,
ae-address-index => uint,
ae-count => uint,
}
ae-type = 0
ae-code = 1
ae-address-index = 2
ae-count = 3
AddressEventType = ( RR = {
tcp-reset : 0, name-index => uint, ; Index to a name in the
icmp-time-exceeded : 1, ; name-rdata table
icmp-dest-unreachable : 2, classtype-index => uint,
icmpv6-time-exceeded : 3, ? ttl => uint,
icmpv6-dest-unreachable: 4, ? rdata-index => uint, ; Index to RDATA in the
icmpv6-packet-too-big : 5, ; name-rdata table
) }
; Other map key values already defined above.
ttl = 2
rdata-index = 3
; MalformedMessageData = {
; Malformed messages. ? server-address-index => uint,
; ? server-port => uint,
MalformedMessage = { ? mm-transport-flags => TransportFlags,
? time-offset => uticks, ; Time offset from start of block ? mm-payload => bstr,
? client-address-index => uint, }
? client-port => uint, ; Other map key values already defined above.
? message-data-index => uint, mm-transport-flags = 2
} mm-payload = 3
; Other map key values already defined above.
message-data-index = 3 ;
; A single query/response pair.
;
QueryResponse = {
? time-offset => uticks, ; Time offset from
; start of block
? client-address-index => uint,
? client-port => uint,
? transaction-id => uint,
? qr-signature-index => uint,
? client-hoplimit => uint,
? response-delay => ticks,
? query-name-index => uint,
? query-size => uint, ; DNS size of query
? response-size => uint, ; DNS size of response
? response-processing-data => ResponseProcessingData,
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
time-offset = 0
client-address-index = 1
client-port = 2
transaction-id = 3
qr-signature-index = 4
client-hoplimit = 5
response-delay = 6
query-name-index = 7
query-size = 8
response-size = 9
response-processing-data = 10
query-extended = 11
response-extended = 12
ResponseProcessingData = {
? bailiwick-index => uint,
? processing-flags => ResponseProcessingFlags,
}
bailiwick-index = 0
processing-flags = 1
ResponseProcessingFlagValues = &(
from-cache : 0,
)
ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues
QueryResponseExtended = {
? question-index => uint, ; Index of QuestionList
? answer-index => uint, ; Index of RRList
? authority-index => uint,
? additional-index => uint,
}
question-index = 0
answer-index = 1
authority-index = 2
additional-index = 3
;
; Address event data.
;
AddressEventCount = {
ae-type => &AddressEventType,
? ae-code => uint,
ae-address-index => uint,
ae-count => uint,
}
ae-type = 0
ae-code = 1
ae-address-index = 2
ae-count = 3
AddressEventType = (
tcp-reset : 0,
icmp-time-exceeded : 1,
icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5,
)
;
; Malformed messages.
;
MalformedMessage = {
? time-offset => uticks, ; Time offset from
; start of block
? client-address-index => uint,
? client-port => uint,
? message-data-index => uint,
}
; Other map key values already defined above.
message-data-index = 3
Appendix B. DNS Name compression example Appendix B. DNS Name compression example
The basic algorithm, which follows the guidance in [RFC1035], is The basic algorithm, which follows the guidance in [RFC1035], is
simply to collect each name, and the offset in the packet at which it simply to collect each name, and the offset in the packet at which it
starts, during packet construction. As each name is added, it is starts, during packet construction. As each name is added, it is
offered to each of the collected names in order of collection, offered to each of the collected names in order of collection,
starting from the first name. If labels at the end of the name can starting from the first name. If labels at the end of the name can
be replaced with a reference back to part (or all) of the earlier be replaced with a reference back to part (or all) of the earlier
name, and if the uncompressed part of the name is shorter than any name, and if the uncompressed part of the name is shorter than any
compression already found, the earlier name is noted as the compression already found, the earlier name is noted as the
compression target for the name. compression target for the name.
The following tables illustrate the process. In an example packet, The following tables illustrate the process. In an example packet,
the first name is example.com. the first name is foo.example.
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| 1 | example.com | | | | 1 | foo.example | | |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
The next name added is bar.com. This is matched against example.com. The next name added is bar.example. This is matched against
The com part of this can be used as a compression target, with the foo.example. The example part of this can be used as a compression
remaining uncompressed part of the name being bar. target, with the remaining uncompressed part of the name being bar.
+---+-------------+--------------+--------------------+ +---+-------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+-----------------------+
| 1 | example.com | | | | 1 | foo.example | | |
| 2 | bar.com | bar | 1 + offset to com | | 2 | bar.example | bar | 1 + offset to example |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+-----------------------+
The third name added is www.bar.com. This is first matched against The third name added is www.bar.example. This is first matched
example.com, and as before this is recorded as a compression target, against foo.example, and as before this is recorded as a compression
with the remaining uncompressed part of the name being www.bar. It target, with the remaining uncompressed part of the name being
is then matched against the second name, which again can be a www.bar. It is then matched against the second name, which again can
compression target. Because the remaining uncompressed part of the be a compression target. Because the remaining uncompressed part of
name is www, this is an improved compression, and so it is adopted. the name is www, this is an improved compression, and so it is
adopted.
+---+-------------+--------------+--------------------+ +---+-----------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+ +---+-----------------+--------------+-----------------------+
| 1 | example.com | | | | 1 | foo.example | | |
| 2 | bar.com | bar | 1 + offset to com | | 2 | bar.example | bar | 1 + offset to example |
| 3 | www.bar.com | www | 2 | | 3 | www.bar.example | www | 2 |
+---+-------------+--------------+--------------------+ +---+-----------------+--------------+-----------------------+
As an optimization, if a name is already perfectly compressed (in As an optimization, if a name is already perfectly compressed (in
other words, the uncompressed part of the name is empty), then no other words, the uncompressed part of the name is empty), then no
further names will be considered for compression. further names will be considered for compression.
B.1. NSD compression algorithm B.1. NSD compression algorithm
Using the above basic algorithm the packet lengths of responses Using the above basic algorithm the packet lengths of responses
generated by NSD [6] can be matched almost exactly. At the time of generated by NSD [7] can be matched almost exactly. At the time of
writing, a tiny number (<.01%) of the reconstructed packets had writing, a tiny number (<.01%) of the reconstructed packets had
incorrect lengths. incorrect lengths.
B.2. Knot Authoritative compression algorithm B.2. Knot Authoritative compression algorithm
The Knot Authoritative [7] name server uses different compression The Knot Authoritative [8] name server uses different compression
behavior, which is the result of internal optimization designed to behavior, which is the result of internal optimization designed to
balance runtime speed with compression size gains. In brief, and balance runtime speed with compression size gains. In brief, and
omitting complications, Knot Authoritative will only consider the omitting complications, Knot Authoritative will only consider the
QNAME and names in the immediately preceding RR section in an RRSET QNAME and names in the immediately preceding RR section in an RRSET
as compression targets. as compression targets.
A set of smart heuristics as described below can be implemented to A set of smart heuristics as described below can be implemented to
mimic this and while not perfect it produces output nearly, but not mimic this and while not perfect it produces output nearly, but not
quite, as good a match as with NSD. The heuristics are: quite, as good a match as with NSD. The heuristics are:
skipping to change at page 63, line 27 skipping to change at page 67, line 50
In sample traffic collected on a root name server around 2-4% of In sample traffic collected on a root name server around 2-4% of
responses generated by Knot had different packet lengths to those responses generated by Knot had different packet lengths to those
produced by NSD. produced by NSD.
Appendix C. Comparison of Binary Formats Appendix C. Comparison of Binary Formats
Several binary serialisation formats were considered, and for Several binary serialisation formats were considered, and for
completeness were also compared to JSON. completeness were also compared to JSON.
o Apache Avro [8]. Data is stored according to a pre-defined o Apache Avro [9]. Data is stored according to a pre-defined
schema. The schema itself is always included in the data file. schema. The schema itself is always included in the data file.
Data can therefore be stored untagged, for a smaller serialisation Data can therefore be stored untagged, for a smaller serialisation
size, and be written and read by an Avro library. size, and be written and read by an Avro library.
* At the time of writing, Avro libraries are available for C, * At the time of writing, Avro libraries are available for C,
C++, C#, Java, Python, Ruby and PHP. Optionally tools are C++, C#, Java, Python, Ruby and PHP. Optionally tools are
available for C++, Java and C# to generate code for encoding available for C++, Java and C# to generate code for encoding
and decoding. and decoding.
o Google Protocol Buffers [9]. Data is stored according to a pre- o Google Protocol Buffers [10]. Data is stored according to a pre-
defined schema. The schema is used by a generator to generate defined schema. The schema is used by a generator to generate
code for encoding and decoding the data. Data can therefore be code for encoding and decoding the data. Data can therefore be
stored untagged, for a smaller serialisation size. The schema is stored untagged, for a smaller serialisation size. The schema is
not stored with the data, so unlike Avro cannot be read with a not stored with the data, so unlike Avro cannot be read with a
generic library. generic library.
* Code must be generated for a particular data schema to to read * Code must be generated for a particular data schema to read and
and write data using that schema. At the time of writing, the write data using that schema. At the time of writing, the
Google code generator can currently generate code for encoding Google code generator can currently generate code for encoding
and decoding a schema for C++, Go, Java, Python, Ruby, C#, and decoding a schema for C++, Go, Java, Python, Ruby, C#,
Objective-C, Javascript and PHP. Objective-C, Javascript and PHP.
o CBOR [10]. Defined in [RFC7049], this serialisation format is o CBOR [11]. Defined in [RFC7049], this serialisation format is
comparable to JSON but with a binary representation. It does not comparable to JSON but with a binary representation. It does not
use a pre-defined schema, so data is always stored tagged. use a pre-defined schema, so data is always stored tagged.
However, CBOR data schemas can be described using CDDL However, CBOR data schemas can be described using CDDL
[I-D.ietf-cbor-cddl] and tools exist to verify data files conform [I-D.ietf-cbor-cddl] and tools exist to verify data files conform
to the schema. to the schema.
* CBOR is a simple format, and simple to implement. At the time * CBOR is a simple format, and simple to implement. At the time
of writing, the CBOR website lists implementations for 16 of writing, the CBOR website lists implementations for 16
languages. languages.
Avro and Protocol Buffers both allow storage of untagged data, but Avro and Protocol Buffers both allow storage of untagged data, but
because they rely on the data schema for this, their implementation because they rely on the data schema for this, their implementation
skipping to change at page 64, line 28 skipping to change at page 69, line 5
A test program was written which reads input from a PCAP file and A test program was written which reads input from a PCAP file and
writes output using one of two basic structures; either a simple writes output using one of two basic structures; either a simple
structure, where each query/response pair is represented in a single structure, where each query/response pair is represented in a single
record entry, or the C-DNS block structure. record entry, or the C-DNS block structure.
The resulting output files were then compressed using a variety of The resulting output files were then compressed using a variety of
common general-purpose lossless compression tools to explore the common general-purpose lossless compression tools to explore the
compressibility of the formats. The compression tools employed were: compressibility of the formats. The compression tools employed were:
o snzip [11]. A command line compression tool based on the Google o snzip [12]. A command line compression tool based on the Google
Snappy [12] library. Snappy [13] library.
o lz4 [13]. The command line compression tool from the reference C o lz4 [14]. The command line compression tool from the reference C
LZ4 implementation. LZ4 implementation.
o gzip [14]. The ubiquitous GNU zip tool. o gzip [15]. The ubiquitous GNU zip tool.
o zstd [15]. Compression using the Zstandard algorithm. o zstd [16]. Compression using the Zstandard algorithm.
o xz [16]. A popular compression tool noted for high compression. o xz [17]. A popular compression tool noted for high compression.
In all cases the compression tools were run using their default In all cases the compression tools were run using their default
settings. settings.
Note that this draft does not mandate the use of compression, nor any Note that this draft does not mandate the use of compression, nor any
particular compression scheme, but it anticipates that in practice particular compression scheme, but it anticipates that in practice
output data will be subject to general-purpose compression, and so output data will be subject to general-purpose compression, and so
this should be taken into consideration. this should be taken into consideration.
"test.pcap", a 662Mb capture of sample data from a root instance was "test.pcap", a 662Mb capture of sample data from a root instance was
skipping to change at page 68, line 38 skipping to change at page 73, line 8
| 320000 | 54.41 | 799.20 | 13.97 | | 320000 | 54.41 | 799.20 | 13.97 |
+------------+-----------+--------+-----------+ +------------+-----------+--------+-----------+
Increasing block size, therefore, tends to increase maximum RSS a Increasing block size, therefore, tends to increase maximum RSS a
little, with no significant effect (if anything a small reduction) on little, with no significant effect (if anything a small reduction) on
CPU consumption. CPU consumption.
The following figure plots the effect of increasing block size on The following figure plots the effect of increasing block size on
output file size for different compressions. output file size for different compressions.
Figure showing effect of block size on file size (PNG) [17] Figure showing effect of block size on file size (PNG) [18]
Figure showing effect of block size on file size (SVG) [18] Figure showing effect of block size on file size (SVG) [19]
From the above, there is obviously scope for tuning the default block From the above, there is obviously scope for tuning the default block
size to the compression being employed, traffic characteristics, size to the compression being employed, traffic characteristics,
frequency of output file rollover etc. Using a strong compression, frequency of output file rollover etc. Using a strong compression
block sizes over 10,000 query/response pairs would seem to offer scheme, block sizes over 10,000 query/response pairs would seem to
limited improvements. offer limited improvements.
Authors' Addresses Authors' Addresses
John Dickinson John Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom United Kingdom
Email: jad@sinodun.com Email: jad@sinodun.com
Jim Hague Jim Hague
 End of changes. 277 change blocks. 
842 lines changed or deleted 1088 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/