draft-ietf-dnsop-dns-capture-format-10.txt   rfc8618.txt 
dnsop J. Dickinson Internet Engineering Task Force (IETF) J. Dickinson
Internet-Draft J. Hague Request for Comments: 8618 J. Hague
Intended status: Standards Track S. Dickinson Category: Standards Track S. Dickinson
Expires: June 15, 2019 Sinodun IT ISSN: 2070-1721 Sinodun IT
T. Manderson T. Manderson
J. Bond
ICANN ICANN
December 12, 2018 J. Bond
Wikimedia Foundation, Inc.
September 2019
C-DNS: A DNS Packet Capture Format Compacted-DNS (C-DNS): A Format for DNS Packet Capture
draft-ietf-dnsop-dns-capture-format-10
Abstract Abstract
This document describes a data representation for collections of DNS This document describes a data representation for collections of DNS
messages. The format is designed for efficient storage and messages. The format is designed for efficient storage and
transmission of large packet captures of DNS traffic; it attempts to transmission of large packet captures of DNS traffic; it attempts to
minimize the size of such packet capture files but retain the full minimize the size of such packet capture files but retain the full
DNS message contents along with the most useful transport metadata. DNS message contents along with the most useful transport metadata.
It is intended to assist with the development of DNS traffic It is intended to assist with the development of DNS traffic-
monitoring applications. monitoring applications.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This is an Internet Standards Track document.
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.
This Internet-Draft will expire on June 15, 2019. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8618.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Data collection use cases . . . . . . . . . . . . . . . . . . 5 3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 5
4. Design considerations . . . . . . . . . . . . . . . . . . . . 7 4. Design Considerations . . . . . . . . . . . . . . . . . . . . 8
5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 9 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 10
6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9 6. C-DNS Format Conceptual Overview . . . . . . . . . . . . . . 10
6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 14
6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 14
6.2.1. Optional data items . . . . . . . . . . . . . . . . . 14 6.2.1. Optional Data Items . . . . . . . . . . . . . . . . . 15
6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 15 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 16
6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15 6.2.3. Storage Flags . . . . . . . . . . . . . . . . . . . . 17
6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 16 6.2.4. IP Address Storage . . . . . . . . . . . . . . . . . 17
7. C-DNS format detailed description . . . . . . . . . . . . . . 16 7. C-DNS Format Detailed Description . . . . . . . . . . . . . . 18
7.1. Map quantities and indexes . . . . . . . . . . . . . . . 16 7.1. Map Quantities and Indexes . . . . . . . . . . . . . . . 18
7.2. Tabular representation . . . . . . . . . . . . . . . . . 17 7.2. Tabular Representation . . . . . . . . . . . . . . . . . 18
7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 18 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 18 7.3.1. "FilePreamble" . . . . . . . . . . . . . . . . . . . 20
7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 19 7.3.1.1. "BlockParameters" . . . . . . . . . . . . . . . . 20
7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 22 7.3.1.1.1. "StorageParameters" . . . . . . . . . . . . . 21
7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 24 7.3.1.1.1.1. "StorageHints" . . . . . . . . . . . . . 22
7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 24 7.3.1.1.2. "CollectionParameters" . . . . . . . . . . . 24
7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 25 7.3.2. "Block" . . . . . . . . . . . . . . . . . . . . . . . 25
7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 26 7.3.2.1. "BlockPreamble" . . . . . . . . . . . . . . . . . 26
7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 32 7.3.2.2. "BlockStatistics" . . . . . . . . . . . . . . . . 27
7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 34 7.3.2.3. "BlockTables" . . . . . . . . . . . . . . . . . . 28
7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 34 7.3.2.3.1. "ClassType" . . . . . . . . . . . . . . . . . 29
7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 35 7.3.2.3.2. "QueryResponseSignature" . . . . . . . . . . 30
7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 36 7.3.2.3.3. "Question" . . . . . . . . . . . . . . . . . 33
8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 37 7.3.2.3.4. "RR" . . . . . . . . . . . . . . . . . . . . 34
9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 37 7.3.2.3.5. "MalformedMessageData" . . . . . . . . . . . 34
9.1. Name compression . . . . . . . . . . . . . . . . . . . . 38
10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 39 7.3.2.4. "QueryResponse" . . . . . . . . . . . . . . . . . 35
10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 40 7.3.2.4.1. "ResponseProcessingData" . . . . . . . . . . 36
10.2. Message identifiers . . . . . . . . . . . . . . . . . . 42 7.3.2.4.2. "QueryResponseExtended" . . . . . . . . . . . 37
10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 42 7.3.2.5. "AddressEventCount" . . . . . . . . . . . . . . . 38
10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 43 7.3.2.6. "MalformedMessage" . . . . . . . . . . . . . . . 39
10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 43 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 39
10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 43 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 40
10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 43 9.1. Name Compression . . . . . . . . . . . . . . . . . . . . 42
10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 44 10. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 42
10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 44 10.1. Matching Algorithm . . . . . . . . . . . . . . . . . . . 43
10.8. Post processing . . . . . . . . . . . . . . . . . . . . 44 10.2. Message Identifiers . . . . . . . . . . . . . . . . . . 45
11. Implementation guidance . . . . . . . . . . . . . . . . . . . 44 10.2.1. Primary ID (Required) . . . . . . . . . . . . . . . 45
11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 45 10.2.2. Secondary ID (Optional) . . . . . . . . . . . . . . 46
11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 45 10.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 46
11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 45 10.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 46
11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 45 10.5. Algorithm Limitations . . . . . . . . . . . . . . . . . 47
12. Implementation status . . . . . . . . . . . . . . . . . . . . 46 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 47
12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 46 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 47
13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 47 10.8. Post-Processing . . . . . . . . . . . . . . . . . . . . 47
13.1. Transport types . . . . . . . . . . . . . . . . . . . . 47 11. Implementation Guidance . . . . . . . . . . . . . . . . . . . 47
13.2. Data storage flags . . . . . . . . . . . . . . . . . . . 48 11.1. Optional Data . . . . . . . . . . . . . . . . . . . . . 48
13.3. Response processing flags . . . . . . . . . . . . . . . 48 11.2. Trailing Bytes . . . . . . . . . . . . . . . . . . . . . 48
13.4. AddressEvent types . . . . . . . . . . . . . . . . . . . 49 11.3. Limiting Collection of RDATA . . . . . . . . . . . . . . 49
14. Security considerations . . . . . . . . . . . . . . . . . . . 49 11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 49
15. Privacy considerations . . . . . . . . . . . . . . . . . . . 50 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 49
16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 50 12.1. Transport Types . . . . . . . . . . . . . . . . . . . . 49
17. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 51 12.2. Data Storage Flags . . . . . . . . . . . . . . . . . . . 50
18. References . . . . . . . . . . . . . . . . . . . . . . . . . 54 12.3. Response-Processing Flags . . . . . . . . . . . . . . . 51
18.1. Normative References . . . . . . . . . . . . . . . . . . 54 12.4. AddressEvent Types . . . . . . . . . . . . . . . . . . . 51
18.2. Informative References . . . . . . . . . . . . . . . . . 55 13. Security Considerations . . . . . . . . . . . . . . . . . . . 52
18.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 57 14. Privacy Considerations . . . . . . . . . . . . . . . . . . . 52
15. References . . . . . . . . . . . . . . . . . . . . . . . . . 53
15.1. Normative References . . . . . . . . . . . . . . . . . . 53
15.2. Informative References . . . . . . . . . . . . . . . . . 55
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 58 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 58
Appendix B. DNS Name compression example . . . . . . . . . . . . 68 Appendix B. DNS Name Compression Example . . . . . . . . . . . . 69
B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 69 B.1. NSD Compression Algorithm . . . . . . . . . . . . . . . . 70
B.2. Knot Authoritative compression algorithm . . . . . . . . 70 B.2. Knot Authoritative Compression Algorithm . . . . . . . . 70
B.3. Observed differences . . . . . . . . . . . . . . . . . . 70 B.3. Observed Differences . . . . . . . . . . . . . . . . . . 71
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 70 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 71
C.1. Comparison with full PCAP files . . . . . . . . . . . . . 73 C.1. Comparison with Full PCAP Files . . . . . . . . . . . . . 74
C.2. Simple versus block coding . . . . . . . . . . . . . . . 74 C.2. Simple versus Block Coding . . . . . . . . . . . . . . . 74
C.3. Binary versus text formats . . . . . . . . . . . . . . . 74 C.3. Binary versus Text Formats . . . . . . . . . . . . . . . 75
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 74 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 75
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 75 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 75
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 75 C.6. Block Size Choice . . . . . . . . . . . . . . . . . . . . 76
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 76
Appendix D. Data Fields for Traffic Regeneration . . . . . . . . 77
D.1. Recommended Fields for Traffic Regeneration . . . . . . . 77
D.2. Issues with Small Data Captures . . . . . . . . . . . . . 77
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 78
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 79
1. Introduction 1. Introduction
There has long been a need for server operators to collect DNS There has long been a need for server operators to collect DNS
queries and responses on authoritative and recursive name servers for Queries and Responses on authoritative and recursive name servers for
monitoring and analysis. This data is used in a number of ways monitoring and analysis. This data is used in a number of ways,
including traffic monitoring, analyzing network attacks and "day in including traffic monitoring, analyzing network attacks, and "day in
the life" (DITL) [ditl] analysis. the life" (DITL) [ditl] analysis.
A wide variety of tools already exist that facilitate the collection A wide variety of tools already exist that facilitate the collection
of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap of DNS traffic data, such as the DNS Statistics Collector (DSC)
[dnscap] and dnstap [dnstap]. However, there is no standard exchange [dsc], packetq [packetq], dnscap [dnscap], and dnstap [dnstap].
format for large DNS packet captures. The PCAP [pcap] or PCAP-NG However, there is no standard exchange format for large DNS packet
[pcapng] formats are typically used in practice for packet captures, captures. The PCAP ("packet capture") [pcap] format or the PCAP Next
but these file formats can contain a great deal of additional Generation (PCAP-NG) [pcapng] format is typically used in practice
information that is not directly pertinent to DNS traffic analysis for packet captures, but these file formats can contain a great deal
and thus unnecessarily increases the capture file size. Additionally of additional information that is not directly pertinent to DNS
these tools and formats typically have no filter mechanism to traffic analysis and thus unnecessarily increases the capture file
selectively record only certain fields at capture time, requiring size. Additionally, these tools and formats typically have no filter
post-processing for anonymization or pseudonymization of data to mechanism to selectively record only certain fields at capture time,
protect user privacy. requiring post-processing for anonymization or pseudonymization of
data to protect user privacy.
There has also been work on using text based formats to describe DNS There has also been work on using text-based formats to describe DNS
packets such as [I-D.daley-dnsxml], [RFC8427], but these are largely packets (for example, see [dnsxml] and [RFC8427]), but this work is
aimed at producing convenient representations of single messages. largely aimed at producing convenient representations of single
messages.
Many DNS operators may receive hundreds of thousands of queries per Many DNS operators may receive hundreds of thousands of Queries per
second on a single name server instance so a mechanism to minimize second on a single name server instance, so a mechanism to minimize
the storage and transmission size (and therefore upload overhead) of the storage and transmission size (and therefore upload overhead) of
the data collected is highly desirable. the data collected is highly desirable.
The format described in this document, C-DNS (Compacted-DNS), The format described in this document, C-DNS (Compacted-DNS), focuses
focusses on the problem of capturing and storing large packet capture on the problem of capturing and storing large packet capture files of
files of DNS traffic with the following goals in mind: DNS traffic with the following goals in mind:
o Minimize the file size for storage and transmission. o Minimize the file size for storage and transmission.
o Minimize the overhead of producing the packet capture file and the o Minimize the overhead of producing the packet capture file and the
cost of any further (general purpose) compression of the file. cost of any further (general-purpose) compression of the file.
This document contains: This document contains:
o A discussion of some common use cases in which DNS data is o A discussion of some common use cases in which DNS data is
collected, see Section 3. collected; see Section 3.
o A discussion of the major design considerations in developing an o A discussion of the major design considerations in developing an
efficient data representation for collections of DNS messages, see efficient data representation for collections of DNS messages; see
Section 4. Section 4.
o A description of why CBOR [RFC7049] was chosen for this format, o A description of why the Concise Binary Object Representation
see Section 5. (CBOR) [RFC7049] was chosen for this format; see Section 5.
o A conceptual overview of the C-DNS format, see Section 6. o A conceptual overview of the C-DNS format; see Section 6.
o The definition of the C-DNS format for the collection of DNS o The definition of the C-DNS format for the collection of DNS
messages, see Section 7. messages; see Section 7.
o Notes on converting C-DNS data to PCAP format, see Section 9. o Notes on converting C-DNS data to PCAP format; see Section 9.
o Some high level implementation considerations for applications o Some high-level implementation considerations for applications
designed to produce C-DNS, see Section 10. designed to produce C-DNS; see Section 10.
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in
14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
"Packet" refers to an individual IPv4 or IPv6 packet. Typically "Packet" refers to an individual IPv4 or IPv6 packet. Typically,
packets are UDP datagrams, but may also be part of a TCP data stream. packets are UDP datagrams, but such packets may also be part of a TCP
"Message", unless otherwise qualified, refers to a DNS payload data stream. "Message", unless otherwise qualified, refers to a DNS
extracted from a UDP datagram or a TCP data stream. payload extracted from a UDP datagram or a TCP data stream.
The parts of DNS messages are named as they are in [RFC1035]. The parts of DNS messages are named as they are in [RFC1035].
Specifically, the DNS message has five sections: Header, Question, Specifically, the DNS message has five sections: Header, Question,
Answer, Authority, and Additional. Answer, Authority, and Additional.
Pairs of DNS messages are called a Query and a Response. 3. Data Collection Use Cases
3. Data collection use cases
From a purely server operator perspective, collecting full packet From a purely server operator perspective, collecting full packet
captures of all packets going in or out of a name server provides the captures of all packets going into or out of a name server provides
most comprehensive picture of network activity. However, there are the most comprehensive picture of network activity. However, there
several design choices or other limitations that are common to many are several design choices or other limitations that are common to
DNS installations and operators. many DNS installations and operators.
o DNS servers are hosted in a variety of situations: o DNS servers are hosted in a variety of situations:
* Self-hosted servers * Self-hosted servers
* Third party hosting (including multiple third parties) * Third-party hosting (including multiple third parties)
* Third party hardware (including multiple third parties) * Third-party hardware (including multiple third parties)
o Data is collected under different conditions: o Data is collected under different conditions:
* On well-provisioned servers running in a steady state * On well-provisioned servers running in a steady state
* On heavily loaded servers * On heavily loaded servers
* On virtualized servers * On virtualized servers
* On servers that are under DoS attack * On servers that are under DoS attack
skipping to change at page 6, line 27 skipping to change at page 6, line 46
o The capabilities of data collection (and upload) networks vary: o The capabilities of data collection (and upload) networks vary:
* Out-of-band networks with the same capacity as the in-band * Out-of-band networks with the same capacity as the in-band
network network
* Out-of-band networks with less capacity than the in-band * Out-of-band networks with less capacity than the in-band
network network
* Everything being on the in-band network * Everything being on the in-band network
Thus, there is a wide range of use cases from very limited data Thus, there is a wide range of use cases, from very limited data
collection environments (third party hardware, servers that are under collection environments (third-party hardware, servers that are under
attack, packet capture on the name server itself and no out-of-band attack, packet capture on the name server itself and no out-of-band
network) to "limitless" environments (self hosted, well provisioned network) to "limitless" environments (self-hosted, well-provisioned
servers, using a network tap or port mirroring with an out-of-band servers, using a network tap or port mirroring with out-of-band
networks with the same capacity as the in-band network). In the networks with the same capacity as the in-band network). In the
former, it is infeasible to reliably collect full packet captures, former case, it is infeasible to reliably collect full packet
especially if the server is under attack. In the latter case, captures, especially if the server is under attack. In the latter
collection of full packet captures may be reasonable. case, collection of full packet captures may be reasonable.
As a result of these restrictions, the C-DNS data format is designed As a result of these restrictions, the C-DNS data format is designed
with the most limited use case in mind such that: with the most limited use case in mind, such that:
o data collection will occur on the same hardware as the name server o Data collection will occur on the same hardware as the name server
itself itself
o collected data will be stored on the same hardware as the name o Collected data will be stored on the same hardware as the name
server itself, at least temporarily server itself, at least temporarily
o collected data being returned to some central analysis system will o Collected data being returned to some central analysis system will
use the same network interface as the DNS queries and responses use the same network interface as the DNS Queries and Responses
o There can be multiple third-party servers involved
o there can be multiple third party servers involved
Because of these considerations, a major factor in the design of the Because of these considerations, a major factor in the design of the
format is minimal storage size of the capture files. format is minimal storage size of the capture files.
Another significant consideration for any application that records Another significant consideration for any application that records
DNS traffic is that the running of the name server software and the DNS traffic is that the running of the name server software and the
transmission of DNS queries and responses are the most important jobs transmission of DNS Queries and Responses are the most important jobs
of a name server; capturing data is not. Any data collection system of a name server; capturing data is not. Any data collection system
co-located with the name server needs to be intelligent enough to co-located with the name server needs to be intelligent enough to
carefully manage its CPU, disk, memory and network utilization. This carefully manage its CPU, disk, memory, and network utilization.
leads to designing a format that requires a relatively low overhead This leads to designing a format that requires a relatively low
to produce and minimizes the requirement for further potentially overhead to produce and minimizes the requirement for further
costly compression. potentially costly compression.
However, it is also essential that interoperability with less However, it is also essential that interoperability with less
restricted infrastructure is maintained. In particular, it is highly restricted infrastructure is maintained. In particular, it is highly
desirable that the collection format should facilitate the re- desirable that the collection format should facilitate the
creation of common formats (such as PCAP) that are as close to the re-creation of common formats (such as PCAP) that are as close to the
original as is realistic given the restrictions above. original as is realistic, given the restrictions above.
4. Design considerations 4. Design Considerations
This section presents some of the major design considerations used in This section presents some of the major design considerations used in
the development of the C-DNS format. the development of the C-DNS format.
1. The basic unit of data is a combined DNS Query and the associated 1. The basic unit of data is a combined DNS Query and the associated
Response (a "Q/R data item"). The same structure will be used Response (a "Query/Response (Q/R) data item"). The same
for unmatched Queries and Responses. Queries without Responses structure will be used for unmatched Queries and Responses.
will be captured omitting the response data. Responses without Queries without Responses will be captured omitting the Response
queries will be captured omitting the Query data (but using the data. Responses without Queries will be captured omitting the
Question section from the response, if present, as an identifying Query data (but using the Question section from the Response, if
QNAME). present, as an identifying QNAME).
* Rationale: A Query and Response represents the basic level of * Rationale: A Query and the associated Response represent the
a client's interaction with the server. Also, combining the basic level of a client's interaction with the server. Also,
Query and Response into one item often reduces storage combining the Query and Response into one item often reduces
requirements due to commonality in the data of the two storage requirements due to commonality in the data of the two
messages. messages.
In the context of generating a C-DNS file it is assumed that only In the context of generating a C-DNS file, it is assumed that
those DNS payloads which can be parsed to produce a well-formed only those DNS payloads that can be parsed to produce a
DNS message are stored in the C-DNS format and that all other well-formed DNS message are stored in the structured Query/
messages will be (optionally) recorded as malformed messages. Response data items of the C-DNS format and that all other
Parsing a well-formed message means as a minimum: messages will (optionally) be recorded as separate malformed
messages. Parsing a well-formed message means, at a minimum, the
following:
* The packet has a well-formed 12 byte DNS Header with a * The packet has a well-formed 12-byte DNS Header with a
recognised OPCODE. recognized OPCODE.
* The section counts are consistent with the section contents. * The section counts are consistent with the section contents.
* All of the resource records can be fully parsed. * All of the Resource Records (RRs) can be fully parsed.
2. All top level fields in each Q/R data item will be optional. 2. All top-level fields in each Query/Response data item will be
optional.
* Rationale: Different operators will have different * Rationale: Different operators will have different
requirements for data to be available for analysis. Operators requirements for data to be available for analysis. Operators
with minimal requirements should not have to pay the cost of with minimal requirements should not have to pay the cost of
recording full data, though this will limit the ability to recording full data, though this will limit the ability to
perform certain kinds of data analysis and also to reconstruct perform certain kinds of data analysis and also to reconstruct
packet captures. For example, omitting the resource records packet captures. For example, omitting the RRs from a
from a Response will reduce the C-DNS file size; in principle Response will reduce the C-DNS file size; in principle,
responses can be synthesized if there is enough context. Responses can be synthesized if there is enough context.
Operators may have different policies for collecting user data Operators may have different policies for collecting user data
and can choose to omit or anonymize certain fields at capture and can choose to omit or anonymize certain fields at capture
time e.g. client address. time, e.g., client address.
3. Multiple Q/R data items will be collected into blocks in the 3. Multiple Query/Response data items will be collected into blocks
format. Common data in a block will be abstracted and referenced in the format. Common data in a block will be abstracted and
from individual Q/R data items by indexing. The maximum number referenced from individual Query/Response data items by indexing.
of Q/R data items in a block will be configurable. The maximum number of Query/Response data items in a block will
be configurable.
* Rationale: This blocking and indexing provides a significant * Rationale: This blocking and indexing action provides a
reduction in the volume of file data generated. Although this significant reduction in the volume of file data generated.
introduces complexity, it provides compression of the data Although this introduces complexity, it provides compression
that makes use of knowledge of the DNS message structure. of the data that makes use of knowledge of the DNS message
structure.
* It is anticipated that the files produced can be subject to * It is anticipated that the files produced can be subject to
further compression using general purpose compression tools. further compression using general-purpose compression tools.
Measurements show that blocking significantly reduces the CPU Measurements show that blocking significantly reduces the CPU
required to perform such strong compression. See required to perform such strong compression. See
Appendix C.2. Appendix C.2.
* Examples of commonality between DNS messages are that in most * Examples of commonality between DNS messages are that in most
cases the QUESTION RR is the same in the query and response, cases the QUESTION RR is the same in the Query and Response
and that there is a finite set of query signatures (based on a and that there is a finite set of Query "signatures" (based on
subset of attributes). For many authoritative servers there a subset of attributes). For many authoritative servers,
is very likely to be a finite set of responses that are there is very likely to be a finite set of Responses that are
generated, of which a large number are NXDOMAIN. generated, of which a large number are NXDOMAIN.
4. Traffic metadata can optionally be included in each block. 4. Traffic metadata can optionally be included in each block.
Specifically, counts of some types of non-DNS packets (e.g. Specifically, counts of some types of non-DNS packets (e.g.,
ICMP, TCP resets) sent to the server may be of interest. ICMP, TCP resets) sent to the server may be of interest.
5. The wire format content of malformed DNS messages may optionally 5. The wire-format content of malformed DNS messages may optionally
be recorded. be recorded.
* Rationale: Any structured capture format that does not capture * Rationale: Any structured capture format that does not capture
the DNS payload byte for byte will be limited to some extent the DNS payload byte for byte will be limited to some extent
in that it cannot represent malformed DNS messages. Only in that it cannot represent malformed DNS messages. Only
those messages that can be fully parsed and transformed into those messages that can be fully parsed and transformed into
the structured format can be fully represented. Note, the structured format can be fully represented. Note,
however, this can result in rather misleading statistics. For however, that this can result in rather misleading statistics.
example, a malformed query which cannot be represented in the For example, a malformed Query that cannot be represented in
C-DNS format will lead to the (well formed) DNS responses with the C-DNS format will lead to the (well-formed) DNS Response
error code FORMERR appearing as 'unmatched'. Therefore it can with error code FORMERR appearing as "unmatched". Therefore,
greatly aid downstream analysis to have the wire format of the it can greatly aid downstream analysis to have the wire format
malformed DNS messages available directly in the C-DNS file. of the malformed DNS messages available directly in the
C-DNS file.
5. Choice of CBOR 5. Choice of CBOR
This document presents a detailed format description using CBOR, the This document presents a detailed format description for C-DNS. The
Concise Binary Object Representation defined in [RFC7049]. format uses CBOR [RFC7049].
The choice of CBOR was made taking a number of factors into account. The choice of CBOR was made taking a number of factors into account.
o CBOR is a binary representation, and thus is economical in storage o CBOR is a binary representation and thus is economical in storage
space. space.
o Other binary representations were investigated, and whilst all had o Other binary representations were investigated, and whilst all had
attractive features, none had a significant advantage over CBOR. attractive features, none had a significant advantage over CBOR.
See Appendix C for some discussion of this. See Appendix C for some discussion of this.
o CBOR is an IETF specification and familiar to IETF participants. o CBOR is an IETF specification and is familiar to IETF
It is based on the now-common ideas of lists and objects, and thus participants. It is based on the now-common ideas of lists and
requires very little familiarization for those in the wider objects and thus requires very little familiarization for those in
industry. the wider industry.
o CBOR is a simple format, and can easily be implemented from o CBOR is a simple format and can easily be implemented from scratch
scratch if necessary. More complex formats require library if necessary. Formats that are more complex require library
support which may present problems on unusual platforms. support, which may present problems on unusual platforms.
o CBOR can also be easily converted to text formats such as JSON o CBOR can also be easily converted to text formats such as JSON
([RFC8259]) for debugging and other human inspection requirements. [RFC8259] for debugging and other human inspection requirements.
o CBOR data schemas can be described using CDDL o CBOR data schemas can be described using the Concise Data
[I-D.ietf-cbor-cddl]. Definition Language (CDDL) [RFC8610].
6. C-DNS format conceptual overview 6. C-DNS Format Conceptual Overview
The following figures show purely schematic representations of the The following figures show purely schematic representations of the
C-DNS format to convey the high-level structure of the C-DNS format. C-DNS format to convey the high-level structure of the C-DNS format.
Section 7 provides a detailed discussion of the CBOR representation Section 7 provides a detailed discussion of the CBOR representation
and individual elements. and individual elements.
Figure 1 shows the C-DNS format at the top level including the file Figure 1 shows the C-DNS format at the top level, including the file
header and data blocks. The Query/Response data items, Address/Event header and data blocks. The Query/Response data items, Address/Event
Count data items and Malformed Message data items link to various Count data items, and Malformed Message data items link to various
Block tables. Block Tables.
+-------+ +-------+
+ C-DNS | + C-DNS |
+-------+--------------------------+ +-------+--------------------------+
| File type identifier | | File Type Identifier |
+----------------------------------+ +----------------------------------+
| File preamble | | File Preamble |
| +--------------------------------+ | +--------------------------------+
| | Format version info | | | Format Version |
| +--------------------------------+ | +--------------------------------+
| | Block parameters | | | Block Parameters |
+-+--------------------------------+ +-+--------------------------------+
| Block | | Block |
| +--------------------------------+ | +--------------------------------+
| | Block preamble | | | Block Preamble |
| +--------------------------------+ | +--------------------------------+
| | Block statistics | | | Block Statistics |
| +--------------------------------+ | +--------------------------------+
| | Block tables | | | Block Tables |
| +--------------------------------+ | +--------------------------------+
| | Query/Response data items | | | Query/Response data items |
| +--------------------------------+ | +--------------------------------+
| | Address/Event Count data items | | | Address/Event Count data items |
| +--------------------------------+ | +--------------------------------+
| | Malformed Message data items | | | Malformed Message data items |
+-+--------------------------------+ +-+--------------------------------+
| Block | | Block |
| +--------------------------------+ | +--------------------------------+
| | Block preamble | | | Block Preamble |
| +--------------------------------+ | +--------------------------------+
| | Block statistics | | | Block Statistics |
| +--------------------------------+ | +--------------------------------+
| | Block tables | | | Block Tables |
| +--------------------------------+ | +--------------------------------+
| | Query/Response data items | | | Query/Response data items |
| +--------------------------------+ | +--------------------------------+
| | Address/Event Count data items | | | Address/Event Count data items |
| +--------------------------------+ | +--------------------------------+
| | Malformed Message data items | | | Malformed Message data items |
+-+--------------------------------+ +-+--------------------------------+
| Further Blocks... | | Further Blocks... |
+----------------------------------+ +----------------------------------+
Figure 1: The C-DNS format. Figure 1: The C-DNS Format
Figure 2 shows some more detailed relationships within each block, Figure 2 shows some more-detailed relationships within each Block,
specifically those between the Query/Response data item and the specifically those between the Query/Response data item and the
relevant Block tables. relevant Block Tables. Some fields have been omitted for clarity.
+----------------+ +----------------+
| Query/Response | | Query/Response |
+-------------------------+ +-------------------------+
| Time offset | | Time Offset |
+-------------------------+ +------------------+ +-------------------------+ +------------------+
| Client address |------------>| IP address array | | Client Address |---------+->| IP Address array |
+-------------------------+ +------------------+ +-------------------------+ | +------------------+
| Client port | | Client Port | |
+-------------------------+ +------------------+ +-------------------------+ | +------------------+
| Transaction ID | +------>| Name/RDATA array |<------+ | Transaction ID | +---)->| Name/RDATA array |<--------+
+-------------------------+ | +------------------+ | +-------------------------+ | | +------------------+ |
| Query signature |--+ | | | Query Signature |--+ | | |
+-------------------------+ | | +-----------------+ | +-------------------------+ | | | +-----------------+ |
| Client hoplimit (q) | +--)------>| Query Signature | | | Client Hoplimit (q) | +--)---)->| Query Signature | |
+-------------------------+ | +-----------------+------+ | +-------------------------+ | | +-----------------+-------+ |
| Response delay (r) | | | Server address | | | Response Delay (r) | | +--| Server Address | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +-------------------------+ |
| Query name |--+--+ | Server port | | | Query Name |--+--+ | Server Port | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +-------------------------+ |
| Query size (q) | | | Transport flags | | | Query Size (q) | | | Transport Flags | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +-------------------------+ |
| Response size (r) | | | QR type | | | Response Size (r) | | | QR Type | |
+-------------------------+ | +------------------------+ | +-------------------------+ | +-------------------------+ |
| Response processing (r) | | | QR signature flags | | | Response Processing (r) | | | QR Signature Flags | |
| +-----------------------+ | +------------------------+ | | +-----------------------+ | +-------------------------+ |
| | Bailiwick index |--+ | Query OPCODE (q) | | | | Bailiwick |--+ | Query OPCODE (q) | |
| +-----------------------+ +------------------------+ | | +-----------------------+ +-------------------------+ |
| | Flags | | QR DNS flags | | | | Flags | | QR DNS Flags | |
+-+-----------------------+ +------------------------+ | +-+-----------------------+ +-------------------------+ |
| Extra query info (q) | | Query RCODE (q) | | | Extra Query Info (q) | | Query RCODE (q) | |
| +-----------------------+ +------------------------+ | | +-----------------------+ +-------------------------+ |
| | Question |--+---+ +--+-Query Class/Type (q) | | | | Question |--+---+ +--+-Query Class/Type (q) | |
| +-----------------------+ | | +------------------------+ | | +-----------------------+ | | +-------------------------+ |
| | Answer |--+ | | | Query QD count (q) | | | | Answer |--+ | | | Query QDCOUNT (q) | |
| +-----------------------+ | | | +------------------------+ | | +-----------------------+ | | | +-------------------------+ |
| | Authority |--+ | | | Query AN count (q) | | | | Authority |--+ | | | Query ANCOUNT (q) | |
| +-----------------------+ | | | +------------------------+ | | +-----------------------+ | | | +-------------------------+ |
| | Additional |--+ | | | Query NS count (q) | | | | Additional |--+ | | | Query NSCOUNT (q) | |
+-+-----------------------+ | | | +------------------------+ | +-+-----------------------+ | | | +-------------------------+ |
| Extra response info (r) | |-+ | | | Query EDNS version (q) | | | Extra Response Info (r) | |-+ | | | Query ARCOUNT (q) | |
| +-----------------------+ | | | | +------------------------+ | | +-----------------------+ | | | | +-------------------------+ |
| | Answer |--+ | | | | EDNS UDP size (q) | | | | Answer |--+ | | | | Query EDNS version (q) | |
| +-----------------------+ | | | | +------------------------+ | | +-----------------------+ | | | | +-------------------------+ |
| | Authority |--+ | | | | Query Opt RDATA (q) | | | | Authority |--+ | | | | Query EDNS UDP Size (q) | |
| +-----------------------+ | | | | +------------------------+ | | +-----------------------+ | | | | +-------------------------+ |
| | Additional |--+ | | | | Response RCODE (r) | | | | Additional |--+ | | | | Query OPT RDATA (q) |--+
+-+-----------------------+ | | | +------------------------+ | +-+-----------------------+ | | | +-------------------------+ |
| | | | | | | | Response RCODE (r) | |
| | | | | | | +-------------------------+ |
+ -----------------------------+ | +----------+ | + -----------------------------+ | +----------+ |
| | | | | | | |
| + -----------------------------+ | | | + -----------------------------+ | |
| | +---------------+ +----------+ | | | | +---------------+ +----------+ | |
| +->| Question list |->| Question | | | | +->| Question List |->| Question | | |
| | array | | array | | | | | array | | array | | |
| +---------------+ +----------+--+ | | | +---------------+ +----------+--+ | |
| | Name |--+------)------------------+ | | Name |--+-----)--------------------+
| +-------------+ | | +------------+ | +-------------+ | | +------------+
| | Class/type |--)---+--+->| Class/Type | | | Class/Type |--)---+-+->| Class/Type |
| +-------------+ | | | array | | +-------------+ | | | array |
| | | +------------+--+ | | | +------------+--+
| | | | Class | | | | | CLASS |
| +---------------+ +----------+ | | +---------------+ | +---------------+ +----------+ | | +---------------+
+--->| RR list array |->| RR array | | | | Type | +--->| RR List array |->| RR array | | | | TYPE |
+---------+-----+ +----------+--+ | | +---------------+ +---------+-----+ +----------+--+ | | +---------------+
| Name |--+ | | Name |--+ |
+-------------+ | +-------------+ |
| Class/type |------+ | Class/Type |------+
+-------------+ +-------------+
Figure 2: The Query/Response data item and subsidiary tables. Figure 2: The Query/Response Data Item and Subsidiary Tables
In Figure 2 data items annotated (q) are only present when a query/ In Figure 2, data items annotated (q) are only present when a
response has a query, and those annotated (r) are only present when a Query/Response has a Query, and those annotated (r) are only present
query/response response is present. when a Query/Response Response is present.
A C-DNS file begins with a file header containing a File Type A C-DNS file begins with a file header containing a File Type
Identifier and a File Preamble. The File Preamble contains Identifier and a File Preamble. The File Preamble contains
information on the file Format Version and an array of Block information on the file Format Version and an array of Block
Parameters items (the contents of which include Collection and Parameters items (the contents of which include Collection and
Storage Parameters used for one or more blocks). Storage Parameters used for one or more Blocks).
The file header is followed by a series of data Blocks. The file header is followed by a series of Blocks.
A Block consists of a Block Preamble item, some Block Statistics for A Block consists of a Block Preamble item, some Block Statistics for
the traffic stored within the Block and then various arrays of common the traffic stored within the Block, and then various arrays of
data collectively called the Block Tables. This is then followed by common data collectively called the Block Tables. This is then
an array of the Query/Response data items detailing the queries and followed by an array of the Query/Response data items detailing the
responses stored within the Block. The array of Query/Response data Queries and Responses stored within the Block. The array of
items is in turn followed by the Address/Event Counts data items (an Query/Response data items is in turn followed by the Address/Event
array of per-client counts of particular IP events) and then Count data items (an array of per-client counts of particular IP
Malformed Message data items (an array of malformed messages that events) and then Malformed Message data items (an array of malformed
stored in the Block). messages that are stored in the Block).
The exact nature of the DNS data will affect what block size is the The exact nature of the DNS data will affect what Block size is the
best fit, however sample data for a root server indicated that block best fit; however, sample data for a root server indicated that Block
sizes up to 10,000 Q/R data items give good results. See sizes up to 10,000 Query/Response data items give good results. See
Appendix C.6 for more details. Appendix C.6 for more details.
This design exploits data commonality and block based storage to This design exploits data commonality and block-based storage to
minimise the C-DNS file size. As a result C-DNS cannot be streamed minimize the C-DNS file size. As a result, C-DNS cannot be streamed
below the level of a block. below the level of a Block.
6.1. Block Parameters 6.1. Block Parameters
The details of the Block Parameters items are not shown in the The details of the Block Parameters items are not shown in the
diagrams but are discussed here for context. diagrams but are discussed here for context.
An array of Block Parameters items is stored in the File Preamble An array of Block Parameters items is stored in the File Preamble
(with a minimum of one item at index 0); a Block Parameters item (with a minimum of one item at index 0); a Block Parameters item
consists of a collection of Storage and Collection Parameters that consists of a collection of Storage and Collection Parameters that
applies to any given Block. An array is used in order to support use applies to any given Block. An array is used in order to support use
cases such as wanting to merge C-DNS files from different sources. cases such as wanting to merge C-DNS files from different sources.
The Block Preamble item then contains an optional index for the Block The Block Preamble item then contains an optional index for the Block
Parameters item that applies for that Block; if not present the index Parameters item that applies for that Block; if not present, the
defaults to 0. Hence, in effect, a global Block Parameters item is index defaults to 0. Hence, in effect, a global Block Parameters
defined which can then be overridden per Block. item is defined that can then be overridden per Block.
6.2. Storage Parameters 6.2. Storage Parameters
The Block Parameters item includes a Storage Parameters item - this The Block Parameters item includes a Storage Parameters item -- this
contains information about the specific data fields stored in the contains information about the specific data fields stored in the
C-DNS file. C-DNS file.
These parameters include: These parameters include:
o The sub-second timing resolution used by the data. o The sub-second timing resolution used by the data.
o Information (hints) on which optional data are omitted. See o Information (hints) on which optional data are omitted. See
Section 6.2.1. Section 6.2.1.
o Recorded OPCODES [opcodes] and RR types [rrtypes]. See o Recorded OPCODES [opcodes] and RR TYPEs [rrtypes]. See
Section 6.2.2. Section 6.2.2.
o Flags indicating, for example, whether the data is sampled or o Flags indicating, for example, whether the data is sampled or
anonymized. See Section 6.2.3 and Section 15. anonymized. See Sections 6.2.3 and 14.
o Client and server IPv4 and IPv6 address prefixes. See o Client and server IPv4 and IPv6 address prefixes. See
Section 6.2.4 Section 6.2.4.
6.2.1. Optional data items 6.2.1. Optional Data Items
To enable implementations to store data to their precise requirements To enable implementations to store data to their precise requirements
in as space-efficient manner as possible, all fields in the following in as space-efficient a manner as possible, all fields in the
arrays are optional: following arrays are optional:
o Query/Response o Query/Response
o Query Signature o Query Signature
o Malformed messages o Malformed Messages
In other words, an implementation can choose to omit any data item In other words, an implementation can choose to omit any data item
that is not required for its use case. In addition, implementations that is not required for its use case (whilst observing the
may be configured to not record all RRs, or only record messages with restrictions relating to IP address storage described in
certain OPCODES. Section 6.2.4). In addition, implementations may be configured to
not record all RRs or to only record messages with certain OPCODES.
This does, however, mean that a consumer of a C-DNS file faces two This does, however, mean that a consumer of a C-DNS file faces two
problems: problems:
1. How can it quickly determine if a file definitely does not 1. How can it quickly determine if a file definitely does not
contain the data items it requires to complete a particular task contain the data items it requires to complete a particular task
(e.g. reconstructing query traffic or performing a specific piece (e.g., reconstructing DNS traffic or performing a specific piece
of data analysis)? of data analysis)?
2. How can it determine if a data item is not present because it 2. How can it determine whether a data item is not present because
was: it was (1) explicitly not recorded or (2) not available/present?
* explicitly not recorded or
* the data item was not available/present.
For example, capturing C-DNS data from within a nameserver For example, capturing C-DNS data from within a name server
implementation makes it unlikely that the Client Hoplimit can be implementation makes it unlikely that the Client Hoplimit can be
recorded. Or, if there is no query ARCount recorded and no query OPT recorded. Or, if there is no Query ARCOUNT recorded and no Query OPT
RDATA [RFC6891] recorded, is that because no query contained an OPT RDATA [RFC6891] recorded, is that because no Query contained an OPT
RR, or because that data was not stored? RR, or because that data was not stored?
The Storage Parameters therefore also contains a Storage Hints item The Storage Parameters item therefore also contains a Storage Hints
which specifies which items the encoder of the file omits from the item, which specifies which items the encoder of the file omits from
stored data and will therefore never be present. (This approach is the stored data and will therefore never be present. (This approach
taken because a flag that indicated which items were included for is taken because a flag that indicated which items were included for
collection would not guarantee that the item was present, only that collection would not guarantee that the item was present -- only that
it might be.) An implementation decoding that file can then use it might be.) An implementation decoding that file can then use
these to quickly determine whether the input data is rich enough for these flags to quickly determine whether the input data is not rich
its needs. enough for its needs.
One scenario where this may be particularly important is the case of
regenerating traffic. It is possible to collect such a small set of
data items that an implementation decoding the file cannot determine
if a given Query/Response data item was generated from just a Query,
just a Response, or a Query/Response pair. This makes it impossible
to reconstruct DNS traffic even if sensible defaults are provided for
the missing data items. This is discussed in more detail in
Section 9.
6.2.2. Optional RRs and OPCODEs 6.2.2. Optional RRs and OPCODEs
Also included in the Storage Parameters are explicit arrays listing Also included in the Storage Parameters item are explicit arrays
the RR types and the OPCODEs to be recorded. These remove any listing the RR TYPEs and the OPCODEs to be recorded. These arrays
ambiguity over whether messages containing particular OPCODEs or are remove any ambiguity over whether, for example, messages containing
not present because they did not occur, or because the implementation particular OPCODEs are not present because (1) certain OPCODEs did
is not configured to record them. not occur or (2) the implementation is not configured to record them.
In the case of OPCODEs, for a message to be fully parsable, the In the case of OPCODEs, for a message to be fully parsable, the
OPCODE must be known to the collecting implementation. Any message OPCODE must be known to the collecting implementation. Any message
with an OPCODE unknown to the collecting implementation cannot be with an OPCODE unknown to the collecting implementation cannot be
validated as correctly formed, and so must be treated as malformed. validated as correctly formed and so must be treated as malformed.
Messages with OPCODES known to the recording application but not Messages with OPCODES known to the recording application but not
listed in the Storage Parameters are discarded by the recording listed in the Storage Parameters item are discarded by the recording
application during C-DNS capture (regardless of whether they are application during C-DNS capture (regardless of whether they are
malformed or not). malformed or not).
In the case of RR records, each record in a message must be fully In the case of RRs, each record in a message must be fully parsable,
parsable, including parsing the record RDATA, as otherwise the including parsing the record RDATA, as otherwise the message cannot
message cannot be validated as correctly formed. Any RR record with be validated as correctly formed. Any RR with an RR TYPE not known
an RR type not known to the collecting implementation cannot be to the collecting implementation cannot be validated as correctly
validated as correctly formed, and so must be treated as malformed. formed and so must be treated as malformed.
Once a message is correctly parsed, an implementation is free to Once a message is correctly parsed, an implementation is free to
record only a subset of the RR records present. record only a subset of the RRs present.
6.2.3. Storage flags 6.2.3. Storage Flags
The Storage Parameters contains flags that can be used to indicate The Storage Parameters item contains flags that can be used to
if: indicate if:
o the data is anonymized, o the data is anonymized,
o the data is produced from sample data, or o the data is produced from sample data, or
o names in the data have been normalized (converted to uniform o names in the data have been normalized (converted to uniform
case). case).
The Storage Parameters also contains optional fields holding details The Storage Parameters item also contains optional fields holding
of the sampling method used and the anonymization method used. It is details of the sampling method used and the anonymization method
RECOMMENDED these fields contain URIs [RFC3986] pointing to resources used. It is RECOMMENDED that these fields contain URIs [RFC3986]
describing the methods used. See Section 15 for further discussion pointing to resources describing the methods used. See Section 14
of anonymization and normalization. for further discussion of anonymization and normalization.
6.2.4. IP Address storage 6.2.4. IP Address Storage
The format can store either full IP addresses or just IP prefixes, The format can store either full IP addresses or just IP prefixes;
the Storage Parameters contains fields to indicate if only IP the Storage Parameters item contains fields to indicate if only IP
prefixes were stored. prefixes were stored.
If the IP address prefixes are absent, then full addresses are If the IP address prefixes are absent, then full addresses are
stored. In this case the IP version can be directly inferred from stored. In this case, the IP version can be directly inferred from
the stored address length and the fields "qr-transport-flags" in the stored address length and the fields "qr-transport-flags" in
QueryResponseSignature and "mm-transport-flags" in QueryResponseSignature, "ae-transport-flags" in AddressEventCount,
MalformedMessageData (which contain the IP version bit) are optional. and "mm-transport-flags" in MalformedMessageData (which contain the
IP version bit) are optional.
If IP address prefixes are given, only the prefix bits of addresses If IP address prefixes are given, only the prefix bits of addresses
are stored. In this case the fields "qr-transport-flags" in are stored. In this case, in order to determine the IP version, the
QueryResponseSignature and "mm-transport-flags" in fields "qr-transport-flags" in QueryResponseSignature, "ae-transport-
MalformedMessageData MUST be present, so that the IP version can be flags" in AddressEventCount, and "mm-transport-flags" in
determined. See Section 7.5.3.2 and Section 7.5.3.5. MalformedMessageData MUST be present. See Sections 7.3.2.3.2 and
7.3.2.3.5.
As an example of storing only IP prefixes, if a client IPv6 prefix of As an example of storing only IP prefixes, if a client IPv6 prefix of
48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334 48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334
will be stored as 0x20010db885a3, reducing address storage space will be stored as 0x20010db885a3, reducing address storage space
requirements. Similarly, if a client IPv4 prefix of 16 is specified, requirements. Similarly, if a client IPv4 prefix of 16 is specified,
a client address of 192.0.2.1 will be stored as 0xc000 (192.0). a client address of 192.0.2.1 will be stored as 0xc000 (192.0).
7. C-DNS format detailed description 7. C-DNS Format Detailed Description
The CDDL definition for the C-DNS format is given in Appendix A. The CDDL definition for the C-DNS format is given in Appendix A.
7.1. Map quantities and indexes 7.1. Map Quantities and Indexes
All map keys are integers with values specified in the CDDL. String All map keys are integers with values specified in the CDDL. String
keys would significantly bloat the file size. keys would significantly bloat the file size.
All key values specified are positive integers under 24, so their All key values specified are positive integers under 24, so their
CBOR representation is a single byte. Positive integer values not CBOR representation is a single byte. Positive integer values not
currently used as keys in a map are reserved for use in future currently used as keys in a map are reserved for use in future
standard extensions. standard extensions.
Implementations may choose to add additional implementation-specific Implementations may choose to add additional implementation-specific
entries to any map. Negative integer map keys are reserved for these entries to any map. Negative integer map keys are reserved for these
values. Key values from -1 to -24 also have a single byte CBOR values. Key values from -1 to -24 also have a single-byte CBOR
representation, so such implementation-specific extensions are not at representation, so such implementation-specific extensions are not at
any space efficiency disadvantage. any space efficiency disadvantage.
An item described as an index is the index of the data item in the An item described as an index is the index of the data item in the
referenced array. Indexes are 0-based. referenced array. Indexes are 0-based.
7.2. Tabular representation 7.2. Tabular Representation
The following sections present the C-DNS specification in tabular The following sections present the C-DNS specification in tabular
format with a detailed description of each item. format with a detailed description of each item.
In all quantities that contain bit flags, bit 0 indicates the least In all quantities that contain bit flags, bit 0 indicates the least
significant bit, i.e. flag "n" in quantity "q" is on if "(q & (1 << significant bit, i.e., flag "n" in quantity "q" is on if
n)) != 0". "(q & (1 << n)) != 0".
For the sake of readability, all type and field names defined in the For the sake of readability, all type and field names defined in the
CDDL definition are shown in double quotes. Type names are by CDDL definition are shown in double quotes. Type names are by
convention camel case (e.g. "BlockTable"), field names are lower- convention camel case (e.g., "BlockTables"), and field names are
case with hyphens (e.g. "block-tables"). lowercase with hyphens (e.g., "block-tables").
For the sake of brevity, the following conventions are used in the For the sake of brevity, the following conventions are used in the
tables: tables:
o The column M marks whether items in a map are mandatory. o The column M marks whether items in a map are mandatory.
* X - Mandatory items. * X - Mandatory items.
* C - Conditionally mandatory item. Such items are usually * C - Conditionally mandatory items. Such items are usually
optional but may be mandatory in some configurations. optional but may be mandatory in some configurations.
* If the column is empty, the item is optional. * If the column is empty, the item is optional.
o The column T gives the CBOR data type of the item. o The column T gives the CBOR datatype of the item.
* U - Unsigned integer * U - Unsigned integer.
* I - Signed integer (i.e. CBOR unsigned or negative integer) * I - Signed integer (i.e., either a CBOR unsigned integer or a
CBOR negative integer).
* B - Boolean * B - Boolean.
* S - Byte string * S - Byte string.
* T - Text string * T - Text string.
* M - Map * M - Map.
* A - Array * A - Array.
In the case of maps and arrays, more information on the type of each In the case of maps and arrays, more information on the type of each
value, include the CDDL definition name if applicable, is given in value, including the CDDL definition name if applicable, is given in
the description. the description.
7.3. "File" 7.3. "File"
A C-DNS file has an outer structure "File", a map that contains the A C-DNS file has an outer structure "File", an array that contains
following: the following:
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
| file-type-id | X | T | String "C-DNS" identifying the file type. | | file-type-id | X | T | String "C-DNS" identifying the file type. |
| | | | | | | | | |
| file-preamble | X | M | Version and parameter information for the | | file-preamble | X | M | Version and parameter information for the |
| | | | whole file. Map of type "FilePreamble", | | | | | whole file. Map of type "FilePreamble"; |
| | | | see Section 7.4. | | | | | see Section 7.3.1. |
| | | | | | | | | |
| file-blocks | X | A | Array of items of type "Block", see | | file-blocks | X | A | Array of items of type "Block"; see |
| | | | Section 7.5. The array may be empty if | | | | | Section 7.3.2. The array may be empty if |
| | | | the file contains no data. | | | | | the file contains no data. |
+---------------+---+---+-------------------------------------------+ +---------------+---+---+-------------------------------------------+
7.4. "FilePreamble" 7.3.1. "FilePreamble"
Information about data in the file. A map containing the following: Information about data in the file. A map containing the following:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| major-format-version | X | U | Unsigned integer '1'. The major | | major-format-version | X | U | Unsigned integer "1". The major |
| | | | version of format used in file. | | | | | version of the format used in the |
| | | | See Section 8. | | | | | file. See Section 8. |
| | | | | | | | | |
| minor-format-version | X | U | Unsigned integer '0'. The minor | | minor-format-version | X | U | Unsigned integer "0". The minor |
| | | | version of format used in file. | | | | | version of the format used in the |
| | | | See Section 8. | | | | | file. See Section 8. |
| | | | | | | | | |
| private-version | | U | Version indicator available for | | private-version | | U | Version indicator available for |
| | | | private use by implementations. | | | | | private use by implementations. |
| | | | | | | | | |
| block-parameters | X | A | Array of items of type | | block-parameters | X | A | Array of items of type |
| | | | "BlockParameters", see Section | | | | | "BlockParameters". See Section |
| | | | 7.4.1. The array must contain at | | | | | 7.3.1.1. The array must contain |
| | | | least one entry. (The "block- | | | | | at least one entry. (The |
| | | | parameters-index" item in each | | | | | "block-parameters-index" item in |
| | | | "BlockPreamble" indicates which | | | | | each "BlockPreamble" indicates |
| | | | array entry applies to that | | | | | which array entry applies to that |
| | | | "Block".) | | | | | "Block".) |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
7.4.1. "BlockParameters" 7.3.1.1. "BlockParameters"
Parameters relating to data storage and collection which apply to one Parameters relating to data storage and collection that apply to one
or more items of type "Block". A map containing the following: or more items of type "Block". A map containing the following:
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
| storage-parameters | X | M | Parameters relating to data | | storage-parameters | X | M | Parameters relating to data |
| | | | storage in a "Block" item. Map | | | | | storage in a "Block" item. Map |
| | | | of type "StorageParameters", see | | | | | of type "StorageParameters"; see |
| | | | Section 7.4.1.1. | | | | | Section 7.3.1.1.1. |
| | | | | | | | | |
| collection-parameters | | M | Parameters relating to collection | | collection-parameters | | M | Parameters relating to collection |
| | | | of the data in a "Block" item. | | | | | of the data in a "Block" item. |
| | | | Map of type | | | | | Map of type |
| | | | "CollectionParameters", see | | | | | "CollectionParameters"; see |
| | | | Section 7.4.2. | | | | | Section 7.3.1.1.2. |
+-----------------------+---+---+-----------------------------------+ +-----------------------+---+---+-----------------------------------+
7.4.1.1. "StorageParameters" 7.3.1.1.1. "StorageParameters"
Parameters relating to how data is stored in the items of type Parameters relating to how data is stored in the items of type
"Block". A map containing the following: "Block". A map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| ticks-per-second | X | U | Sub-second timing is recorded in | | ticks-per-second | X | U | Sub-second timing is recorded in |
| | | | ticks. This specifies the number of | | | | | ticks. This specifies the number of |
| | | | ticks in a second. | | | | | ticks in a second. |
| | | | | | | | | |
| max-block-items | X | U | The maximum number of items stored in | | max-block-items | X | U | The maximum number of items stored in |
| | | | any of the arrays in a "Block" item | | | | | any of the arrays in a "Block" item |
| | | | (Q/R items, address event counts or | | | | | (Q/R, Address/Event Count, or |
| | | | malformed messages). An indication to | | | | | Malformed Message data items). An |
| | | | a decoder of the resources needed to | | | | | indication to a decoder of the |
| | | | process the file. | | | | | resources needed to process the file. |
| | | | | | | | | |
| storage-hints | X | M | Collection of hints as to which fields | | storage-hints | X | M | Collection of hints as to which fields |
| | | | are omitted in the arrays that have | | | | | are omitted in the arrays that have |
| | | | optional fields. Map of type | | | | | optional fields. Map of type |
| | | | "StorageHints", see Section 7.4.1.1.1. | | | | | "StorageHints". See Section |
| | | | 7.3.1.1.1.1. |
| | | | | | | | | |
| opcodes | X | A | Array of OPCODES [opcodes] (unsigned | | opcodes | X | A | Array of OPCODES [opcodes] (unsigned |
| | | | integers, each in the range 0 to 15 | | | | | integers, each in the range 0 to 15 |
| | | | inclusive) recorded by the collection | | | | | inclusive) recorded by the collecting |
| | | | implementation. See Section 6.2.2. | | | | | implementation. See Section 6.2.2. |
| | | | | | | | | |
| rr-types | X | A | Array of RR types [rrtypes] (unsigned | | rr-types | X | A | Array of RR TYPEs [rrtypes] (unsigned |
| | | | integers, each in the range 0 to 65535 | | | | | integers, each in the range 0 to 65535 |
| | | | inclusive) recorded by the collection | | | | | inclusive) recorded by the collecting |
| | | | implementation. See Section 6.2.2. | | | | | implementation. See Section 6.2.2. |
| | | | | | | | | |
| storage-flags | | U | Bit flags indicating attributes of | | storage-flags | | U | Bit flags indicating attributes of |
| | | | stored data. | | | | | stored data. |
| | | | Bit 0. 1 if the data has been | | | | | Bit 0. 1 if the data has been |
| | | | anonymized. | | | | | anonymized. |
| | | | Bit 1. 1 if the data is sampled data. | | | | | Bit 1. 1 if the data is sampled data. |
| | | | Bit 2. 1 if the names have been | | | | | Bit 2. 1 if the names have been |
| | | | normalized (converted to uniform | | | | | normalized (converted to uniform |
| | | | case). | | | | | case). |
| | | | | | | | | |
| client-address | | U | IPv4 client address prefix length, in | | client-address | | U | IPv4 client address prefix length, in |
| -prefix-ipv4 | | | the range 1 to 32 inclusive. If | | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix | | | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| client-address | | U | IPv6 client address prefix length, in | | client-address | | U | IPv6 client address prefix length, in |
| -prefix-ipv6 | | | the range 1 to 128 inclusive. If | | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix | | | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| server-address | | U | IPv4 server address prefix length, in | | server-address | | U | IPv4 server address prefix length, in |
| -prefix-ipv4 | | | the range 1 to 32 inclusive. If | | -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix | | | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| server-address | | U | IPv6 server address prefix length, in | | server-address | | U | IPv6 server address prefix length, in |
| -prefix-ipv6 | | | the range 1 to 128 inclusive. If | | -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix | | | | | specified, only the address prefix |
| | | | bits are stored. | | | | | bits are stored. |
| | | | | | | | | |
| sampling-method | | T | Information on the sampling method | | sampling-method | | T | Information on the sampling method |
| | | | used. See Section 6.2.3. | | | | | used. See Section 6.2.3. |
| | | | | | | | | |
| anonymization | | T | Information on the anonymization | | anonymization | | T | Information on the anonymization |
| -method | | | method used. See Section 6.2.3. | | -method | | | method used. See Section 6.2.3. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.4.1.1.1. "StorageHints" 7.3.1.1.1.1. "StorageHints"
An indicator of which fields the collecting implementation omits in An indicator of which fields the collecting implementation omits in
the maps with optional fields. A map containing the following: the maps with optional fields. Note that hints have a top-down
precedence. In other words, where a map contains another map, the
hint on the containing map overrides any hints in the contained map
and the contained map is omitted. A map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| query-response | X | U | Hints indicating which "QueryResponse" | | query-response | X | U | Hints indicating which "QueryResponse" |
| -hints | | | fields are candidates for capture or | | -hints | | | fields are omitted; see Section |
| | | | omitted, see section Section 7.6. If a | | | | | 7.3.2.4. If a bit is unset, the field |
| | | | bit is unset, the field is omitted | | | | | is omitted from the capture. |
| | | | from the capture. | | | | | Bit 0. time-offset |
| | | | Bit 0. time-offset | | | | | Bit 1. client-address-index |
| | | | Bit 1. client-address-index | | | | | Bit 2. client-port |
| | | | Bit 2. client-port | | | | | Bit 3. transaction-id |
| | | | Bit 3. transaction-id | | | | | Bit 4. qr-signature-index |
| | | | Bit 4. qr-signature-index | | | | | Bit 5. client-hoplimit |
| | | | Bit 5. client-hoplimit | | | | | Bit 6. response-delay |
| | | | Bit 6. response-delay | | | | | Bit 7. query-name-index |
| | | | Bit 7. query-name-index | | | | | Bit 8. query-size |
| | | | Bit 8. query-size | | | | | Bit 9. response-size |
| | | | Bit 9. response-size | | | | | Bit 10. response-processing-data |
| | | | Bit 10. response-processing-data | | | | | Bit 11. query-question-sections |
| | | | Bit 11. query-question-sections | | | | | Bit 12. query-answer-sections |
| | | | Bit 12. query-answer-sections | | | | | Bit 13. query-authority-sections |
| | | | Bit 13. query-authority-sections | | | | | Bit 14. query-additional-sections |
| | | | Bit 14. query-additional-sections | | | | | Bit 15. response-answer-sections |
| | | | Bit 15. response-answer-sections | | | | | Bit 16. response-authority-sections |
| | | | Bit 16. response-authority-sections | | | | | Bit 17. response-additional-sections |
| | | | Bit 17. response-additional-sections |
| | | | | | | | | |
| query-response | X | U | Hints indicating which | | query-response | X | U | Hints indicating which |
| -signature-hints | | | "QueryResponseSignature" fields are | | -signature-hints | | | "QueryResponseSignature" fields are |
| | | | candidates for capture or omitted, see | | | | | omitted; see Section 7.3.2.3.2. If a |
| | | | section Section 7.5.3.2. If a bit is | | | | | bit is unset, the field is omitted |
| | | | unset, the field is omitted from the | | | | | from the capture. |
| | | | capture. | | | | | Bit 0. server-address-index |
| | | | Bit 0. server-address | | | | | Bit 1. server-port |
| | | | Bit 1. server-port | | | | | Bit 2. qr-transport-flags |
| | | | Bit 2. qr-transport-flags | | | | | Bit 3. qr-type |
| | | | Bit 3. qr-type | | | | | Bit 4. qr-sig-flags |
| | | | Bit 4. qr-sig-flags | | | | | Bit 5. query-opcode |
| | | | Bit 5. query-opcode | | | | | Bit 6. qr-dns-flags |
| | | | Bit 6. dns-flags | | | | | Bit 7. query-rcode |
| | | | Bit 7. query-rcode | | | | | Bit 8. query-classtype-index |
| | | | Bit 8. query-class-type | | | | | Bit 9. query-qdcount |
| | | | Bit 9. query-qdcount | | | | | Bit 10. query-ancount |
| | | | Bit 10. query-ancount | | | | | Bit 11. query-nscount |
| | | | Bit 11. query-nscount | | | | | Bit 12. query-arcount |
| | | | Bit 12. query-arcount | | | | | Bit 13. query-edns-version |
| | | | Bit 13. query-edns-version | | | | | Bit 14. query-udp-size |
| | | | Bit 14. query-udp-size | | | | | Bit 15. query-opt-rdata-index |
| | | | Bit 15. query-opt-rdata | | | | | Bit 16. response-rcode |
| | | | Bit 16. response-rcode |
| | | | | | | | | |
| rr-hints | X | U | Hints indicating which optional "RR" | | rr-hints | X | U | Hints indicating which optional "RR" |
| | | | fields are candidates for capture or | | | | | fields are omitted; see Section |
| | | | omitted, see Section 7.5.3.4. If a bit | | | | | 7.3.2.3.4. If a bit is unset, the |
| | | | is unset, the field is omitted from | | | | | field is omitted from the capture. |
| | | | the capture. | | | | | Bit 0. ttl |
| | | | Bit 0. ttl | | | | | Bit 1. rdata-index |
| | | | Bit 1. rdata-index | | other-data-hints | X | U | Hints indicating which other datatypes |
| other-data-hints | X | U | Hints indicating which other data | | | | | are omitted. If a bit is unset, the |
| | | | types are omitted. If a bit is unset, | | | | | datatype is omitted from the capture. |
| | | | the the data type is omitted from the | | | | | Bit 0. malformed-messages |
| | | | capture. | | | | | Bit 1. address-event-counts |
| | | | Bit 0. malformed-messages |
| | | | Bit 1. address-event-counts |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.4.2. "CollectionParameters" 7.3.1.1.2. "CollectionParameters"
Parameters providing information to how data in the file was Parameters providing information regarding how data in the file was
collected (applicable for some, but not all collection environments). collected (applicable for some, but not all, collection
The values are informational only and serve as hints to downstream environments). The values are informational only and serve as
analysers as to the configuration of a collecting implementation. metadata to downstream analyzers as to the configuration of a
They can provide context when interpreting what data is present/ collecting implementation. They can provide context when
absent from the capture but cannot necessarily be validated against interpreting what data is present/absent from the capture but cannot
the data captured. necessarily be validated against the data captured.
These parameters have no default. If they do not appear, nothing can These parameters have no default. If they do not appear, nothing can
be inferred about their value. be inferred about their value.
A map containing the following items: A map containing the following items:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| query-timeout | | U | To be matched with a query, a response | | query-timeout | | U | To be matched with a Query, a Response |
| | | | must arrive within this number of | | | | | must arrive within this number of |
| | | | seconds. | | | | | milliseconds. |
| | | | | | | | | |
| skew-timeout | | U | The network stack may report a | | skew-timeout | | U | The network stack may report a |
| | | | response before the corresponding | | | | | Response before the corresponding |
| | | | query. A response is not considered to | | | | | Query. A Response is not considered |
| | | | be missing a query until after this | | | | | to be missing a Query until after this |
| | | | many micro-seconds. | | | | | many microseconds. |
| | | | | | | | | |
| snaplen | | U | Collect up to this many bytes per | | snaplen | | U | Collect up to this many bytes per |
| | | | packet. | | | | | packet. |
| | | | | | | | | |
| promisc | | B | "true" if promiscuous mode | | promisc | | B | "true" if promiscuous mode |
| | | | [pcap-options] was enabled on the | | | | | [pcap-options] was enabled on the |
| | | | interface, "false" otherwise. | | | | | interface, "false" otherwise. |
| | | | | | | | | |
| interfaces | | A | Array of identifiers (of type text | | interfaces | | A | Array of identifiers (of type text |
| | | | string) of the interfaces used for | | | | | string) of the interfaces used for |
| | | | collection. | | | | | collection. |
| | | | | | | | | |
| server-addresses | | A | Array of server collection IP | | server-addresses | | A | Array of server collection IP |
| | | | addresses (of type byte string). Hint | | | | | addresses (of type byte string). |
| | | | for downstream analysers; does not | | | | | Metadata for downstream analyzers; |
| | | | affect collection. | | | | | does not affect collection. |
| | | | | | | | | |
| vlan-ids | | A | Array of identifiers (of type unsigned | | vlan-ids | | A | Array of identifiers (of type unsigned |
| | | | integer, each in the range 1 to 4094 | | | | | integer, each in the range 1 to 4094 |
| | | | inclusive) of VLANs [IEEE802.1Q] | | | | | inclusive) of VLANs [IEEE802.1Q] |
| | | | selected for collection. VLAN IDs are | | | | | selected for collection. VLAN IDs are |
| | | | unique only within an administrative | | | | | unique only within an administrative |
| | | | domain. | | | | | domain. |
| | | | | | | | | |
| filter | | T | "tcpdump" [pcap-filter] style filter | | filter | | T | Filter for input, in "tcpdump" |
| | | | for input. | | | | | [pcap-filter] style. |
| | | | | | | | | |
| generator-id | | T | Implementation specific human-readable | | generator-id | | T | Implementation-specific human-readable |
| | | | string identifying the collection | | | | | string identifying the collection |
| | | | method. | | | | | method. |
| | | | | | | | | |
| host-id | | T | String identifying the collecting | | host-id | | T | String identifying the collecting |
| | | | host. Empty if converting an existing | | | | | host. |
| | | | packet capture file. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.5. "Block" 7.3.2. "Block"
Container for data with common collection and storage parameters. A Container for data with common collection and storage parameters. A
map containing the following: map containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| block-preamble | X | M | Overall information for the "Block" | | block-preamble | X | M | Overall information for the "Block" |
| | | | item. Map of type "BlockPreamble", | | | | | item. Map of type "BlockPreamble"; |
| | | | see Section 7.5.1. | | | | | see Section 7.3.2.1. |
| | | | | | | | | |
| block-statistics | | M | Statistics about the "Block" item. | | block-statistics | | M | Statistics about the "Block" item. |
| | | | Map of type "BlockStatistics", see | | | | | Map of type "BlockStatistics"; see |
| | | | Section 7.5.2. | | | | | Section 7.3.2.2. |
| | | | | | | | | |
| block-tables | | M | The arrays containing data | | block-tables | | M | The arrays containing data |
| | | | referenced by individual | | | | | referenced by individual |
| | | | "QueryResponse" or | | | | | "QueryResponse" or |
| | | | "MalformedMessage" items. Map of | | | | | "MalformedMessage" items. Map of |
| | | | type "BlockTables", see Section | | | | | type "BlockTables"; see Section |
| | | | 7.5.3. | | | | | 7.3.2.3. |
| | | | | | | | | |
| query-responses | | A | Details of individual DNS Q/R data | | query-responses | | A | Details of individual C-DNS Q/R data |
| | | | items. Array of items of type | | | | | items. Array of items of type |
| | | | "QueryResponse", see Section 7.6. If | | | | | "QueryResponse"; see Section |
| | | | present, the array must not be | | | | | 7.3.2.4. If present, the array must |
| | | | empty. | | | | | not be empty. |
| | | | | | | | | |
| address-event | | A | Per client counts of ICMP messages | | address-event | | A | Per-client counts of ICMP messages |
| -counts | | | and TCP resets. Array of items of | | -counts | | | and TCP resets. Array of items of |
| | | | type "AddressEventCount", see | | | | | type "AddressEventCount"; see |
| | | | Section 7.7. If present, the array | | | | | Section 7.3.2.5. If present, the |
| | | | must not be empty. | | | | | array must not be empty. |
| | | | | | | | | |
| malformed-messages | | A | Details of malformed DNS messages. | | malformed-messages | | A | Details of malformed DNS messages. |
| | | | Array of items of type | | | | | Array of items of type |
| | | | "MalformedMessage", see Section 7.8. | | | | | "MalformedMessage"; see Section |
| | | | If present, the array must not be | | | | | 7.3.2.6. If present, the array must |
| | | | empty. | | | | | not be empty. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.5.1. "BlockPreamble" 7.3.2.1. "BlockPreamble"
Overall information for a "Block" item. A map containing the Overall information for a "Block" item. A map containing the
following: following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| earliest-time | C | A | A timestamp (2 unsigned integers, | | earliest-time | C | A | A timestamp (two unsigned integers, of |
| | | | "Timestamp") for the earliest record | | | | | type "Timestamp") for the earliest |
| | | | in the "Block" item. The first integer | | | | | record in the "Block" item. The first |
| | | | is the number of seconds since the | | | | | integer is the number of seconds since |
| | | | POSIX epoch [posix-time] ("time_t"), | | | | | the POSIX epoch [posix-time] |
| | | | excluding leap seconds. The second | | | | | ("time_t"), excluding leap seconds. |
| | | | integer is the number of ticks (see | | | | | The second integer is the number of |
| | | | Section 7.4.1.1) since the start of | | | | | ticks (see Section 7.3.1.1.1) since |
| | | | the second. This field is mandatory | | | | | the start of the second. This field |
| | | | unless all block items containing a | | | | | is mandatory unless all block items |
| | | | time offset from the start of the | | | | | containing a time offset from the |
| | | | block also omit that time offset. | | | | | start of the Block also omit that time |
| | | | offset. |
| | | | | | | | | |
| block-parameters | | U | The index of the item in the "block- | | block-parameters | | U | The index of the item in the |
| -index | | | parameters" array (in the "file- | | -index | | | "block-parameters" array (in the |
| | | | premable" item) applicable to this | | | | | "file-preamble" item) applicable to |
| | | | block. If not present, index 0 is | | | | | this block. If not present, index 0 |
| | | | used. See Section 7.4.1. | | | | | is used. See Section 7.3.1. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.5.2. "BlockStatistics" 7.3.2.2. "BlockStatistics"
Basic statistical information about a "Block" item. A map containing Basic statistical information about a "Block" item. A map containing
the following: the following:
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
| processed-messages | | U | Total number of DNS messages | | processed-messages | | U | Total number of well-formed DNS |
| | | | processed from the input traffic | | | | | messages processed from the input |
| | | | stream during collection of data in | | | | | traffic stream during collection of |
| | | | this "Block" item. | | | | | data in this "Block" item. |
| | | | | | | | | |
| qr-data-items | | U | Total number of Q/R data items in | | qr-data-items | | U | Total number of Q/R data items in |
| | | | this "Block" item. | | | | | this "Block" item. |
| | | | | | | | | |
| unmatched-queries | | U | Number of unmatched queries in this | | unmatched-queries | | U | Number of unmatched Queries in this |
| | | | "Block" item. | | | | | "Block" item. |
| | | | | | | | | |
| unmatched-responses | | U | Number of unmatched responses in | | unmatched-responses | | U | Number of unmatched Responses in |
| | | | this "Block" item. | | | | | this "Block" item. |
| | | | | | | | | |
| discarded-opcode | | U | Number of DNS messages processed | | discarded-opcode | | U | Number of DNS messages processed |
| | | | from the input traffic stream | | | | | from the input traffic stream |
| | | | during collection of data in this | | | | | during collection of data in this |
| | | | "Block" item but not recorded | | | | | "Block" item but not recorded |
| | | | because their OPCODE is not in the | | | | | because their OPCODE is not in the |
| | | | list to be collected. | | | | | list to be collected. |
| | | | | | | | | |
| malformed-items | | U | Number of malformed messages found | | malformed-items | | U | Number of malformed messages |
| | | | in input for this "Block" item. | | | | | processed from the input traffic |
| | | | stream during collection of data in |
| | | | this "Block" item. |
+---------------------+---+---+-------------------------------------+ +---------------------+---+---+-------------------------------------+
7.5.3. "BlockTables" 7.3.2.3. "BlockTables"
Map of arrays containing data referenced by individual Map of arrays containing data referenced by individual
"QueryResponse" or "MalformedMessage" items in this "Block". Each "QueryResponse" or "MalformedMessage" items in this "Block". Each
element is an array which, if present, must not be empty. element is an array that, if present, must not be empty.
An item in the "qlist" array contains indexes to values in the "qrr" An item in the "qlist" array contains indexes to values in the "qrr"
array. Therefore, if "qlist" is present, "qrr" must also be present. array. Therefore, if "qlist" is present, "qrr" must also be present.
Similarly, if "rrlist" is present, "rr" must also be present. Similarly, if "rrlist" is present, "rr" must also be present.
The map contains the following items: The map contains the following items:
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
| ip-address | | A | Array of IP addresses, in network | | ip-address | | A | Array of IP addresses, in network |
| | | | byte order (of type byte string). If | | | | | byte order (of type byte string). If |
| | | | client or server address prefixes are | | | | | client or server address prefixes are |
| | | | set, only the address prefix bits are | | | | | set, only the address prefix bits are |
| | | | stored. Each string is therefore up | | | | | stored. Each string is therefore up |
| | | | to 4 bytes long for an IPv4 address, | | | | | to 4 bytes long for an IPv4 address, |
| | | | or up to 16 bytes long for an IPv6 | | | | | or up to 16 bytes long for an IPv6 |
| | | | address. See Section 7.4.1.1. | | | | | address. See Section 7.3.1.1.1. |
| | | | | | | | | |
| classtype | | A | Array of RR class and type | | classtype | | A | Array of RR CLASS and TYPE |
| | | | information. Type is "ClassType", see | | | | | information. Type is "ClassType". |
| | | | Section 7.5.3.1. | | | | | See Section 7.3.2.3.1. |
| | | | | | | | | |
| name-rdata | | A | Array where each entry is the | | name-rdata | | A | Array where each entry is the |
| | | | contents of a single NAME or RDATA in | | | | | contents of a single NAME or RDATA in |
| | | | wire format (of type byte string). | | | | | wire format (of type byte string). |
| | | | Note that NAMEs, and labels within | | | | | Note that NAMEs, and labels within |
| | | | RDATA contents, are full domain names | | | | | RDATA contents, are full domain names |
| | | | or labels; no [RFC1035] name | | | | | or labels; no name compression (per |
| | | | compression is used on the individual | | | | | [RFC1035]) is used on the individual |
| | | | names/labels within the format. | | | | | names/labels within the format. |
| | | | | | | | | |
| qr-sig | | A | Array Q/R data item signatures. Type | | qr-sig | | A | Array of Q/R data item signatures. |
| | | | is "QueryResponseSignature", see | | | | | Type is "QueryResponseSignature". |
| | | | Section 7.5.3.2. | | | | | See Section 7.3.2.3.2. |
| | | | | | | | | |
| qlist | | A | Array of type "QuestionList". A | | qlist | | A | Array of type "QuestionList". A |
| | | | "QuestionList" is an array of | | | | | "QuestionList" is an array of |
| | | | unsigned integers, indexes to | | | | | unsigned integers, indexes to |
| | | | "Question" items in the "qrr" array. | | | | | "Question" items in the "qrr" array. |
| | | | | | | | | |
| qrr | | A | Array of type "Question". Each entry | | qrr | | A | Array of type "Question". Each entry |
| | | | is the contents of a single question, | | | | | is the contents of a single Question, |
| | | | where a question is the second or | | | | | where a Question is the second or |
| | | | subsequent question in a query. See | | | | | subsequent Question in a Query. See |
| | | | Section 7.5.3.3. | | | | | Section 7.3.2.3.3. |
| | | | | | | | | |
| rrlist | | A | Array of type "RRList". An "RRList" | | rrlist | | A | Array of type "RRList". An "RRList" |
| | | | is an array of unsigned integers, | | | | | is an array of unsigned integers, |
| | | | indexes to "RR" items in the "rr" | | | | | indexes to "RR" items in the "rr" |
| | | | array. | | | | | array. |
| | | | | | | | | |
| rr | | A | Array of type "RR". Each entry is the | | rr | | A | Array of type "RR". Each entry is |
| | | | contents of a single RR. See Section | | | | | the contents of a single RR. See |
| | | | 7.5.3.4. | | | | | Section 7.3.2.3.4. |
| | | | | | | | | |
| malformed-message | | A | Array of the contents of malformed | | malformed-message | | A | Array of the contents of malformed |
| -data | | | messages. Array of type | | -data | | | messages. Array of type |
| | | | "MalformedMessageData", see Section | | | | | "MalformedMessageData". See Section |
| | | | 7.5.3.5. | | | | | 7.3.2.3.5. |
+-------------------+---+---+---------------------------------------+ +-------------------+---+---+---------------------------------------+
7.5.3.1. "ClassType" 7.3.2.3.1. "ClassType"
RR class and type information. A map containing the following: RR CLASS and TYPE information. A map containing the following:
+-------+---+---+--------------------------+ +-------+---+---+--------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+-------+---+---+--------------------------+ +-------+---+---+--------------------------+
| type | X | U | TYPE value [rrtypes]. | | type | X | U | TYPE value [rrtypes]. |
| | | | | | | | | |
| class | X | U | CLASS value [rrclasses]. | | class | X | U | CLASS value [rrclasses]. |
+-------+---+---+--------------------------+ +-------+---+---+--------------------------+
7.5.3.2. "QueryResponseSignature" 7.3.2.3.2. "QueryResponseSignature"
Elements of a Q/R data item that are often common between multiple Elements of a Q/R data item that are often common between multiple
individual Q/R data items. A map containing the following: individual Q/R data items. A map containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| server-address | | U | The index in the item in the "ip- | | server-address | | U | The index in the "ip-address" array |
| -index | | | address" array of the server IP | | -index | | | of the server IP address. See |
| | | | address. See Section 7.5.3. | | | | | Section 7.3.2.3. |
| | | | | | | | | |
| server-port | | U | The server port. | | server-port | | U | The server port. |
| | | | | | | | | |
| qr-transport-flags | C | U | Bit flags describing the transport | | qr-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the query. Same | | | | | used to service the Query. Same |
| | | | definition as "mm-transport-flags" | | | | | definition as "mm-transport-flags" |
| | | | in Section 7.5.3.5, with an | | | | | in Section 7.3.2.3.5, with an |
| | | | additional indicator for trailing | | | | | additional indicator for trailing |
| | | | bytes, see Appendix A. | | | | | bytes. See Appendix A. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if | | | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6. See Section 6.2.4. | | | | | IPv6. See Section 6.2.4. |
| | | | Bit 1-4. Transport. 4 bit unsigned | | | | | Bits 1-4. Transport. 4-bit |
| | | | value where 0 = UDP, 1 = TCP, 2 = | | | | | unsigned value where |
| | | | TLS, 3 = DTLS [RFC7858], 4 = DoH | | | | | 0 = UDP [RFC1035] |
| | | | [RFC8484]. Values 5-15 are reserved | | | | | 1 = TCP [RFC1035] |
| | | | for future use. | | | | | 2 = TLS [RFC7858] |
| | | | Bit 5. 1 if trailing bytes in query | | | | | 3 = DTLS [RFC8094] |
| | | | packet. See Section 11.2. | | | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport (see |
| | | | below) |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | Bit 5. 1 if trailing bytes in Query |
| | | | packet. See Section 11.2. |
| | | | | | | | | |
| qr-type | | U | Type of Query/Response transaction. | | qr-type | | U | Type of Query/Response transaction |
| | | | 0 = Stub. A query from a stub | | | | | based on the definitions in the |
| | | | dnstap schema [dnstap-schema]. |
| | | | 0 = Stub. A transaction between a |
| | | | stub resolver and a DNS server from |
| | | | the perspective of the stub |
| | | | resolver. | | | | | resolver. |
| | | | 1 = Client. An incoming query to a | | | | | 1 = Client. A transaction between a |
| | | | recursive resolver. | | | | | client and a DNS server (a proxy or |
| | | | 2 = Resolver. A query sent from a | | | | | full recursive resolver) from the |
| | | | recursive resolver to an authorative | | | | | perspective of the DNS server. |
| | | | 2 = Resolver. A transaction between |
| | | | a recursive resolver and an |
| | | | authoritative server from the |
| | | | perspective of the recursive |
| | | | resolver. | | | | | resolver. |
| | | | 3 = Authorative. A query to an | | | | | 3 = Authoritative. A transaction |
| | | | authorative resolver. | | | | | between a recursive resolver and an |
| | | | 4 = Forwarder. A query sent from a | | | | | authoritative server from the |
| | | | recursive resolver to an upstream | | | | | perspective of the authoritative |
| | | | recursive resolver. | | | | | server. |
| | | | 5 = Tool. A query sent to a server | | | | | 4 = Forwarder. A transaction |
| | | | by a server tool. | | | | | between a downstream forwarder and |
| | | | an upstream DNS server (a recursive |
| | | | resolver) from the perspective of |
| | | | the downstream forwarder. |
| | | | 5 = Tool. A transaction between a |
| | | | DNS software tool and a DNS server, |
| | | | from the perspective of the tool. |
| | | | | | | | | |
| qr-sig-flags | | U | Bit flags explicitly indicating | | qr-sig-flags | | U | Bit flags explicitly indicating |
| | | | attributes of the message pair | | | | | attributes of the message pair |
| | | | represented by this Q/R data item | | | | | represented by this Q/R data item |
| | | | (not all attributes may be recorded | | | | | (not all attributes may be recorded |
| | | | or deducible). | | | | | or deducible). |
| | | | Bit 0. 1 if a Query was present. | | | | | Bit 0. 1 if a Query was present. |
| | | | Bit 1. 1 if a Response was present. | | | | | Bit 1. 1 if a Response was present. |
| | | | Bit 2. 1 if a Query was present and | | | | | Bit 2. 1 if a Query was present and |
| | | | it had an OPT Resource Record. | | | | | it had an OPT RR. |
| | | | Bit 3. 1 if a Response was present | | | | | Bit 3. 1 if a Response was present |
| | | | and it had an OPT Resource Record. | | | | | and it had an OPT RR. |
| | | | Bit 4. 1 if a Query was present but | | | | | Bit 4. 1 if a Query was present but |
| | | | had no Question. | | | | | had no Question. |
| | | | Bit 5. 1 if a Response was present | | | | | Bit 5. 1 if a Response was present |
| | | | but had no Question (only one query- | | | | | but had no Question (only one |
| | | | name-index is stored per Q/R item). | | | | | query-name-index is stored per Q/R |
| | | | data item). |
| | | | | | | | | |
| query-opcode | | U | Query OPCODE. | | query-opcode | | U | Query OPCODE. |
| | | | | | | | | |
| qr-dns-flags | | U | Bit flags with values from the Query | | qr-dns-flags | | U | Bit flags with values from the Query |
| | | | and Response DNS flags. Flag values | | | | | and Response DNS flags. Flag values |
| | | | are 0 if the Query or Response is | | | | | are 0 if the Query or Response is |
| | | | not present. | | | | | not present. |
| | | | Bit 0. Query Checking Disabled (CD). | | | | | Bit 0. Query Checking Disabled |
| | | | Bit 1. Query Authenticated Data | | | | | (CD). |
| | | | Bit 1. Query Authenticated Data |
| | | | (AD). | | | | | (AD). |
| | | | Bit 2. Query reserved (Z). | | | | | Bit 2. Query reserved (Z). |
| | | | Bit 3. Query Recursion Available | | | | | Bit 3. Query Recursion Available |
| | | | (RA). | | | | | (RA). |
| | | | Bit 4. Query Recursion Desired (RD). | | | | | Bit 4. Query Recursion Desired |
| | | | Bit 5. Query TrunCation (TC). | | | | | (RD). |
| | | | Bit 6. Query Authoritative Answer | | | | | Bit 5. Query TrunCation (TC). |
| | | | Bit 6. Query Authoritative Answer |
| | | | (AA). | | | | | (AA). |
| | | | Bit 7. Query DNSSEC answer OK (DO). | | | | | Bit 7. Query DNSSEC answer OK (DO). |
| | | | Bit 8. Response Checking Disabled | | | | | Bit 8. Response Checking Disabled |
| | | | (CD). | | | | | (CD). |
| | | | Bit 9. Response Authenticated Data | | | | | Bit 9. Response Authenticated Data |
| | | | (AD). | | | | | (AD). |
| | | | Bit 10. Response reserved (Z). | | | | | Bit 10. Response reserved (Z). |
| | | | Bit 11. Response Recursion Available | | | | | Bit 11. Response Recursion |
| | | | (RA). | | | | | Available (RA). |
| | | | Bit 12. Response Recursion Desired | | | | | Bit 12. Response Recursion Desired |
| | | | (RD). | | | | | (RD). |
| | | | Bit 13. Response TrunCation (TC). | | | | | Bit 13. Response TrunCation (TC). |
| | | | Bit 14. Response Authoritative | | | | | Bit 14. Response Authoritative |
| | | | Answer (AA). | | | | | Answer (AA). |
| | | | | | | | | |
| query-rcode | | U | Query RCODE. If the Query contains | | query-rcode | | U | Query RCODE. If the Query contains |
| | | | OPT [RFC6891], this value | | | | | an OPT RR [RFC6891], this value |
| | | | incorporates any | | | | | incorporates any EXTENDED-RCODE |
| | | | EXTENDED_RCODE_VALUE [rcodes]. | | | | | value [rcodes]. |
| | | | | | | | | |
| query-classtype | | U | The index to the item in the the | | query-classtype | | U | The index in the "classtype" array |
| -index | | | "classtype" array of the CLASS and | | -index | | | of the CLASS and TYPE of the first |
| | | | TYPE of the first Question. See | | | | | Question. See Section 7.3.2.3. |
| | | | Section 7.5.3. |
| | | | | | | | | |
| query-qd-count | | U | The QDCOUNT in the Query, or | | query-qdcount | | U | The QDCOUNT in the Query, or |
| | | | Response if no Query present. | | | | | Response if no Query present. |
| | | | | | | | | |
| query-an-count | | U | Query ANCOUNT. | | query-ancount | | U | Query ANCOUNT. |
| | | | | | | | | |
| query-ns-count | | U | Query NSCOUNT. | | query-nscount | | U | Query NSCOUNT. |
| | | | | | | | | |
| query-ar-count | | U | Query ARCOUNT. | | query-arcount | | U | Query ARCOUNT. |
| | | | | | | | | |
| edns-version | | U | The Query EDNS version. | | query-edns-version | | U | The Query EDNS version. ("EDNS" |
| | | | stands for Extension Mechanisms for |
| | | | DNS.) |
| | | | | | | | | |
| udp-buf-size | | U | The Query EDNS sender's UDP payload | | query-udp-size | | U | The Query EDNS sender's UDP payload |
| | | | size. | | | | | size. |
| | | | | | | | | |
| opt-rdata-index | | U | The index in the "name-rdata" array | | query-opt-rdata | | U | The index in the "name-rdata" array |
| | | | of the OPT RDATA. See Section 7.5.3. | | -index | | | of the OPT RDATA. See Section |
| | | | 7.3.2.3. |
| | | | | | | | | |
| response-rcode | | U | Response RCODE. If the Response | | response-rcode | | U | Response RCODE. If the Response |
| | | | contains OPT [RFC6891], this value | | | | | contains an OPT RR [RFC6891], this |
| | | | incorporates any | | | | | value incorporates any EXTENDED- |
| | | | EXTENDED_RCODE_VALUE [rcodes]. | | | | | RCODE value [rcodes]. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.5.3.3. "Question" Version 1.0 of C-DNS supports transport values corresponding to DNS
transports defined in IETF Standards Track documents at the time of
writing. There are numerous non-standard methods of sending DNS
messages over various transports using a variety of protocols, but
they are out of scope for this document. With the current
specification, these can be generically stored using value 15
(Non-standard transport), or implementations are free to use the
negative integer map keys to define their own mappings. Such
non-standard transports may also be the subject of a future extension
to the specification.
7.3.2.3.3. "Question"
Details on individual Questions in a Question section. A map Details on individual Questions in a Question section. A map
containing the following: containing the following:
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| name-index | X | U | The index in the "name-rdata" array of | | name-index | X | U | The index in the "name-rdata" array of |
| | | | the QNAME. See Section 7.5.3. | | | | | the QNAME. See Section 7.3.2.3. |
| | | | | | | | | |
| classtype-index | X | U | The index in the "classtype" array of | | classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the Question. See | | | | | the CLASS and TYPE of the Question. |
| | | | Section 7.5.3. | | | | | See Section 7.3.2.3. |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
7.5.3.4. "RR" 7.3.2.3.4. "RR"
Details on individual Resource Records in RR sections. A map Details on individual RRs in RR sections. A map containing the
containing the following: following:
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| name-index | X | U | The index in the "name-rdata" array of | | name-index | X | U | The index in the "name-rdata" array of |
| | | | the NAME. See Section 7.5.3. | | | | | the NAME. See Section 7.3.2.3. |
| | | | | | | | | |
| classtype-index | X | U | The index in the "classtype" array of | | classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the RR. See | | | | | the CLASS and TYPE of the RR. See |
| | | | Section 7.5.3. | | | | | Section 7.3.2.3. |
| | | | | | | | | |
| ttl | | U | The RR Time to Live. | | ttl | | U | The RR Time to Live. |
| | | | | | | | | |
| rdata-index | | U | The index in the "name-rdata" array of | | rdata-index | | U | The index in the "name-rdata" array of |
| | | | the RR RDATA. See Section 7.5.3. | | | | | the RR RDATA. See Section 7.3.2.3. |
+-----------------+---+---+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
7.5.3.5. "MalformedMessageData" 7.3.2.3.5. "MalformedMessageData"
Details on malformed message items in this "Block" item. A map Details on malformed DNS messages stored in this "Block" item. A map
containing the following: containing the following:
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
| server-address | | U | The index in the "ip-address" array | | server-address | | U | The index in the "ip-address" array |
| -index | | | of the server IP address. See | | -index | | | of the server IP address. See |
| | | | Section 7.5.3. | | | | | Section 7.3.2.3. |
| | | | | | | | | |
| server-port | | U | The server port. | | server-port | | U | The server port. |
| | | | | | | | | |
| mm-transport-flags | C | U | Bit flags describing the transport | | mm-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the query, see | | | | | used to service the Query. See |
| | | | Section 6.2.4. | | | | | Section 6.2.4. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if | | | | | Bits 1-4. Transport. 4-bit |
| | | | IPv6 | | | | | unsigned value where |
| | | | Bit 1-4. Transport. 4 bit unsigned | | | | | 0 = UDP [RFC1035] |
| | | | value where 0 = UDP, 1 = TCP, 2 = | | | | | 1 = TCP [RFC1035] |
| | | | TLS, 3 = DTLS [RFC7858], 4 = DoH | | | | | 2 = TLS [RFC7858] |
| | | | [RFC8484]. Values 5-15 are reserved | | | | | 3 = DTLS [RFC8094] |
| | | | for future use. | | | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | | | | | | |
| mm-payload | | S | The payload (raw bytes) of the DNS | | mm-payload | | S | The payload (raw bytes) of the DNS |
| | | | message. | | | | | message. |
+--------------------+---+---+--------------------------------------+ +--------------------+---+---+--------------------------------------+
7.6. "QueryResponse" 7.3.2.4. "QueryResponse"
Details on individual Q/R data items. Details on individual Q/R data items.
Note that there is no requirement that the elements of the "query- Note that there is no requirement that the elements of the
responses" array are presented in strict chronological order. "query-responses" array are presented in strict chronological order.
A map containing the following items: A map containing the following items:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| time-offset | | U | Q/R timestamp as an offset in | | time-offset | | U | Q/R timestamp as an offset in |
| | | | ticks (see Section 7.4.1.1) from | | | | | ticks (see Section 7.3.1.1.1) from |
| | | | "earliest-time". The timestamp is | | | | | "earliest-time". The timestamp is |
| | | | the timestamp of the Query, or the | | | | | the timestamp of the Query, or the |
| | | | Response if there is no Query. | | | | | Response if there is no Query. |
| | | | | | | | | |
| client-address-index | | U | The index in the "ip-address" | | client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. | | | | | array of the client IP address. |
| | | | See Section 7.5.3. | | | | | See Section 7.3.2.3. |
| | | | | | | | | |
| client-port | | U | The client port. | | client-port | | U | The client port. |
| | | | | | | | | |
| transaction-id | | U | DNS transaction identifier. | | transaction-id | | U | DNS transaction identifier. |
| | | | | | | | | |
| qr-signature-index | | U | The index in the "qr-sig" array of | | qr-signature-index | | U | The index in the "qr-sig" array of |
| | | | the "QueryResponseSignature" item. | | | | | the "QueryResponseSignature" item. |
| | | | See Section 7.5.3. | | | | | See Section 7.3.2.3. |
| | | | | | | | | |
| client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from | | client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from |
| | | | the Query packet. | | | | | the Query packet. |
| | | | | | | | | |
| response-delay | | I | The time difference between Query | | response-delay | | I | The time difference between Query |
| | | | and Response, in ticks (see | | | | | and Response, in ticks. See |
| | | | Section 7.4.1.1). Only present if | | | | | Section 7.3.1.1.1. Only present |
| | | | there is a query and a response. | | | | | if there is a Query and a |
| | | | The delay can be negative if the | | | | | Response. The delay can be |
| | | | network stack/capture library | | | | | negative if the network |
| | | | returns packets out of order. | | | | | stack/capture library returns |
| | | | packets out of order. |
| | | | | | | | | |
| query-name-index | | U | The index in the "name-rdata" | | query-name-index | | U | The index in the "name-rdata" |
| | | | array of the item containing the | | | | | array of the item containing the |
| | | | QNAME for the first Question. See | | | | | QNAME for the first Question. See |
| | | | Section 7.5.3. | | | | | Section 7.3.2.3. |
| | | | | | | | | |
| query-size | | U | DNS query message size (see | | query-size | | U | DNS Query message size (see |
| | | | below). | | | | | below). |
| | | | | | | | | |
| response-size | | U | DNS response message size (see | | response-size | | U | DNS Response message size (see |
| | | | below). | | | | | below). |
| | | | | | | | | |
| response-processing | | M | Data on response processing. Map | | response-processing | | M | Data on Response processing. Map |
| -data | | | of type "ResponseProcessingData", | | -data | | | of type "ResponseProcessingData". |
| | | | see Section 7.6.1. | | | | | See Section 7.3.2.4.1. |
| | | | | | | | | |
| query-extended | | M | Extended Query data. Map of type | | query-extended | | M | Extended Query data. Map of type |
| | | | "QueryResponseExtended", see | | | | | "QueryResponseExtended". See |
| | | | Section 7.6.2. | | | | | Section 7.3.2.4.2. |
| | | | | | | | | |
| response-extended | | M | Extended Response data. Map of | | response-extended | | M | Extended Response data. Map of |
| | | | type "QueryResponseExtended", see | | | | | type "QueryResponseExtended". See |
| | | | Section 7.6.2. | | | | | Section 7.3.2.4.2. |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
The "query-size" and "response-size" fields hold the DNS message The "query-size" and "response-size" fields hold the DNS message
size. For UDP this is the size of the UDP payload that contained the size. For UDP, this is the size of the UDP payload that contained
DNS message. For TCP it is the size of the DNS message as specified the DNS message. For TCP, it is the size of the DNS message as
in the two-byte message length header. Trailing bytes in UDP queries specified in the two-byte message length header. Trailing bytes in
are routinely observed in traffic to authoritative servers and this UDP Queries are routinely observed in traffic to authoritative
value allows a calculation of how many trailing bytes were present. servers, and this value allows a calculation of how many trailing
bytes were present.
7.6.1. "ResponseProcessingData" 7.3.2.4.1. "ResponseProcessingData"
Information on the server processing that produced the response. A Information on the server processing that produced the Response. A
map containing the following: map containing the following:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| bailiwick-index | | U | The index in the "name-rdata" array of | | bailiwick-index | | U | The index in the "name-rdata" array of |
| | | | the owner name for the response | | | | | the owner name for the Response |
| | | | bailiwick. See Section 7.5.3. | | | | | bailiwick. See Section 7.3.2.3. |
| | | | | | | | | |
| processing-flags | | U | Flags relating to response processing. | | processing-flags | | U | Flags relating to Response processing. |
| | | | Bit 0. 1 if the response came from | | | | | Bit 0. 1 if the Response came from |
| | | | cache. | | | | | cache. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.6.2. "QueryResponseExtended" 7.3.2.4.2. "QueryResponseExtended"
Extended data on the Q/R data item. Extended data on the Q/R data item.
Each item in the map is present only if collection of the relevant Each item in the map is present only if collection of the relevant
details is configured. details is configured.
A map containing the following items: A map containing the following items:
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
| question-index | | U | The index in the "qlist" array of the | | question-index | | U | The index in the "qlist" array of the |
| | | | entry listing any second and | | | | | entry listing any second and |
| | | | subsequent Questions in the Question | | | | | subsequent Questions in the Question |
| | | | section for the Query or Response. See | | | | | section for the Query or Response. |
| | | | Section 7.5.3. | | | | | See Section 7.3.2.3. |
| | | | | | | | | |
| answer-index | | U | The index in the "rrlist" array of the | | answer-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Answer Resource | | | | | entry listing the Answer RR sections |
| | | | Record sections for the Query or | | | | | for the Query or Response. See |
| | | | Response. See Section 7.5.3. | | | | | Section 7.3.2.3. |
| | | | | | | | | |
| authority-index | | U | The index in the "rrlist" array of the | | authority-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Authority Resource | | | | | entry listing the Authority RR |
| | | | Record sections for the Query or | | | | | sections for the Query or Response. |
| | | | Response. See Section 7.5.3. | | | | | See Section 7.3.2.3. |
| | | | | | | | | |
| additional-index | | U | The index in the "rrlist" array of the | | additional-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Additional Resource | | | | | entry listing the Additional RR |
| | | | Record sections for the Query or | | | | | sections for the Query or Response. |
| | | | Response. See Section 7.5.3. Note that | | | | | See Section 7.3.2.3. Note that Query |
| | | | Query OPT RR data can be optionally | | | | | OPT RR data can optionally be stored |
| | | | stored in the QuerySignature. | | | | | in the QuerySignature. |
+------------------+---+---+----------------------------------------+ +------------------+---+---+----------------------------------------+
7.7. "AddressEventCount" 7.3.2.5. "AddressEventCount"
Counts of various IP related events relating to traffic with Counts of various IP-related events relating to traffic with
individual client addresses. A map containing the following: individual client addresses. A map containing the following:
+------------------+---+---+----------------------------------------+ +--------------------+---+---+--------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+------------------+---+---+----------------------------------------+ +--------------------+---+---+--------------------------------------+
| ae-type | X | U | The type of event. The following | | ae-type | X | U | The type of event. The following |
| | | | events types are currently defined: | | | | | event types are currently defined: |
| | | | 0. TCP reset. | | | | | 0. TCP reset. |
| | | | 1. ICMP time exceeded. | | | | | 1. ICMP time exceeded. |
| | | | 2. ICMP destination unreachable. | | | | | 2. ICMP destination unreachable. |
| | | | 3. ICMPv6 time exceeded. | | | | | 3. ICMPv6 time exceeded. |
| | | | 4. ICMPv6 destination unreachable. | | | | | 4. ICMPv6 destination unreachable. |
| | | | 5. ICMPv6 packet too big. | | | | | 5. ICMPv6 packet too big. |
| | | | | | | | | |
| ae-code | | U | A code relating to the event. For ICMP | | ae-code | | U | A code relating to the event. For |
| | | | or ICMPv6 events, this MUST be the | | | | | ICMP or ICMPv6 events, this MUST be |
| | | | ICMP [RFC0792] or ICMPv6 [RFC4443] | | | | | the ICMP [RFC792] or ICMPv6 |
| | | | code. For other events the contents | | | | | [RFC4443] code. For other events, |
| | | | are undefined. | | | | | the contents are undefined. |
| | | | | | | | | |
| ae-address-index | X | U | The index in the "ip-address" array of | | ae-transport-flags | C | U | Bit flags describing the transport |
| | | | the client address. See Section 7.5.3. | | | | | used to service the event. See |
| | | | | | | | | Section 6.2.4. |
| ae-count | X | U | The number of occurrences of this | | | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | event during the block collection | | | | | IPv6. |
| | | | period. | | | | | Bits 1-4. Transport. 4-bit |
+------------------+---+---+----------------------------------------+ | | | | unsigned value where |
| | | | 0 = UDP [RFC1035] |
| | | | 1 = TCP [RFC1035] |
| | | | 2 = TLS [RFC7858] |
| | | | 3 = DTLS [RFC8094] |
| | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | |
| ae-address-index | X | U | The index in the "ip-address" array |
| | | | of the client address. See Section |
| | | | 7.3.2.3. |
| | | | |
| ae-count | X | U | The number of occurrences of this |
| | | | event during the Block collection |
| | | | period. |
+--------------------+---+---+--------------------------------------+
7.8. "MalformedMessage" 7.3.2.6. "MalformedMessage"
Details of malformed messages. A map containing the following: Details on Malformed Message data items. A map containing the
following:
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| Field | M | T | Description | | Field | M | T | Description |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
| time-offset | | U | Message timestamp as an offset in | | time-offset | | U | Message timestamp as an offset in |
| | | | ticks (see Section 7.4.1.1) from | | | | | ticks (see Section 7.3.1.1.1) from |
| | | | "earliest-time". | | | | | "earliest-time". |
| | | | | | | | | |
| client-address-index | | U | The index in the "ip-address" | | client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. | | | | | array of the client IP address. |
| | | | See Section 7.5.3. | | | | | See Section 7.3.2.3. |
| | | | | | | | | |
| client-port | | U | The client port. | | client-port | | U | The client port. |
| | | | | | | | | |
| message-data-index | | U | The index in the "malformed- | | message-data-index | | U | The index in the "malformed- |
| | | | message-data" array of the message | | | | | message-data" array of the message |
| | | | data for this message. See Section | | | | | data for this message. See |
| | | | 7.5.3. | | | | | Section 7.3.2.3. |
+----------------------+---+---+------------------------------------+ +----------------------+---+---+------------------------------------+
8. Versioning 8. Versioning
The C-DNS file preamble includes a file format version; a major and The C-DNS File Preamble includes a file Format Version; a major and
minor version number are required fields. The document defines minor version number are required fields. This document defines
version 1.0 of the C-DNS specification. This section describes the version 1.0 of the C-DNS specification. This section describes the
intended use of these version numbers in future specifications. intended use of these version numbers in future specifications.
It is noted that version 1.0 includes many optional fields and It is noted that version 1.0 includes many optional fields;
therefore consumers of version 1.0 should be inherently robust to therefore, consumers of version 1.0 should be inherently robust to
parsing files with variable data content. parsing files with variable data content.
Within a major version, a new minor version MUST be a strict superset Within a major version, a new minor version MUST be a strict superset
of the previous minor version, with no semantic changes to existing of the previous minor version, with no semantic changes to existing
fields. New keys MAY be added to existing maps, and new maps MAY be fields. New keys MAY be added to existing maps, and new maps MAY be
added. A consumer capable of reading a particular major.minor added. A consumer capable of reading a particular major.minor
version MUST also be capable of reading all previous minor versions version MUST also be capable of reading all previous minor versions
of the same major version. It SHOULD also be capable of parsing all of the same major version. It SHOULD also be capable of parsing all
subsequent minor versions ignoring any keys or maps that it does not subsequent minor versions, ignoring any keys or maps that it does not
recognise. recognize.
A new major version indicates changes to the format that are not A new major version indicates changes to the format that are not
backwards compatible with previous major versions. A consumer backwards compatible with previous major versions. A consumer
capable of only reading a particular major version (greater than 1) capable of only reading a particular major version (greater than 1)
is not required to and has no expectation to be capable of reading a is neither required nor expected to be capable of reading a previous
previous major version. major version.
9. C-DNS to PCAP 9. C-DNS to PCAP
It is possible to re-construct PCAP files from the C-DNS format in a It is usually possible to reconstruct PCAP files from the C-DNS
lossy fashion. Some of the issues with reconstructing both the DNS format in a lossy fashion. Some of the issues with reconstructing
payload and the full packet stream are outlined here. both the DNS payload and the full packet stream are outlined here.
The reconstruction depends on whether or not all the optional The reconstruction of well-formed DNS messages depends on two
sections of both the query and response were captured in the C-DNS factors:
file. Clearly, if they were not all captured, the reconstruction
will be imperfect.
Even if all sections of the response were captured, one cannot 1. Whether or not a particular subset of the optional fields were
reconstruct the DNS response payload exactly due to the fact that captured in the C-DNS file, specifically the data fields
necessary to reconstruct a valid IP header and DNS payload for
both Query and Response (see Appendix D.1). Clearly, if not all
these data fields were captured, the reconstruction is likely to
be imperfect even if reasonable defaults are provided for the
reconstruction.
2. Whether or not at least one field was captured that unambiguously
identifies the Query/Response data item as containing just a
Query, just a Response, or a Query/Response pair. Obviously, the
qr-sig-flags defined in Section 7.3.2.3.2 is such a field;
however, this field is optional. For more details, see
Appendix D.2.
It is noted again that simply having hints that indicate that certain
data fields were not omitted does not guarantee that those data
fields were actually captured. Therefore, the ability to reconstruct
PCAP data (in the absence of defaults) can in principle vary for each
record captured in a C-DNS file, and between Blocks that have
differing hints.
Even if all sections of the Response were captured, one cannot
reconstruct the DNS Response payload exactly, due to the fact that
some DNS names in the message on the wire may have been compressed. some DNS names in the message on the wire may have been compressed.
Section 9.1 discusses this is more detail. Section 9.1 discusses this in more detail.
Some transport information is not captured in the C-DNS format. For Some transport information is not captured in the C-DNS format. For
example, the following aspects of the original packet stream cannot example, the following aspects of the original packet stream cannot
be re-constructed from the C-DNS format: be reconstructed from the C-DNS format:
o IP fragmentation o IP fragmentation
o TCP stream information: o TCP stream information:
* Multiple DNS messages may have been sent in a single TCP * Multiple DNS messages may have been sent in a single TCP
segment segment
* A DNS payload may have been split across multiple TCP segments * A DNS payload may have been split across multiple TCP segments
* Multiple DNS messages may have been sent on a single TCP * Multiple DNS messages may have been sent on a single TCP
session session
skipping to change at page 38, line 14 skipping to change at page 41, line 21
o TCP stream information: o TCP stream information:
* Multiple DNS messages may have been sent in a single TCP * Multiple DNS messages may have been sent in a single TCP
segment segment
* A DNS payload may have been split across multiple TCP segments * A DNS payload may have been split across multiple TCP segments
* Multiple DNS messages may have been sent on a single TCP * Multiple DNS messages may have been sent on a single TCP
session session
o TLS session information:
* TLS version or cipher suites
* TLS-related features such as TCP Fast Open (TFO) [RFC7413] or
TLS session resumption [RFC5077]
o DNS-over-HTTPS [RFC8484] message details:
* Whether the message used POST or GET
* HTTPS Headers
o Malformed DNS messages if the wire format is not recorded o Malformed DNS messages if the wire format is not recorded
o Any Non-DNS messages that were in the original packet stream e.g. o Any non-DNS messages that were in the original packet stream,
ICMP e.g., ICMP
Simple assumptions can be made on the reconstruction: fragmented and Simple assumptions can be made on the reconstruction: fragmented and
DNS-over-TCP messages can be reconstructed into single packets and a DNS-over-TCP messages can be reconstructed into single packets, and a
single TCP session can be constructed for each TCP packet. single TCP session can be constructed for each TCP packet.
Additionally, if malformed messages and Non-DNS packets are captured Additionally, if malformed messages and non-DNS packets are captured
separately, they can be merged with packet captures reconstructed separately, they can be merged with packet captures reconstructed
from C-DNS to produce a more complete packet stream. from C-DNS to produce a more complete packet stream.
9.1. Name compression 9.1. Name Compression
All the names stored in the C-DNS format are full domain names; no All the names stored in the C-DNS format are full domain names; no
[RFC1035] name compression is used on the individual names within the name compression (per [RFC1035]) is used on the individual names
format. Therefore when reconstructing a packet, name compression within the format. Therefore, when reconstructing a packet, name
must be used in order to reproduce the on the wire representation of compression must be used in order to reproduce the on-the-wire
the packet. representation of the packet.
[RFC1035] name compression works by substituting trailing sections of Name compression per [RFC1035] works by substituting trailing
a name with a reference back to the occurrence of those sections sections of a name with a reference back to the occurrence of those
earlier in the message. Not all name server software uses the same sections earlier in the message. Not all name server software uses
algorithm when compressing domain names within the responses. Some the same algorithm when compressing domain names within the
attempt maximum recompression at the expense of runtime resources, Responses. Some attempt maximum recompression at the expense of
others use heuristics to balance compression and speed and others use runtime resources, others use heuristics to balance compression and
different rules for what is a valid compression target. speed, and others use different rules for what is a valid compression
target.
This means that responses to the same question from different name This means that Responses to the same Query from different name
server software which match in terms of DNS payload content (header, server software that match in terms of DNS payload content (header,
counts, RRs with name compression removed) do not necessarily match counts, RRs with name compression removed) do not necessarily match
byte-for-byte on the wire. byte for byte on the wire.
Therefore, it is not possible to ensure that the DNS response payload Therefore, it is not possible to ensure that the DNS Response payload
is reconstructed byte-for-byte from C-DNS data. However, it can at is reconstructed byte for byte from C-DNS data. However, it can at
least, in principle, be reconstructed to have the correct payload least, in principle, be reconstructed to have the correct payload
length (since the original response length is captured) if there is length (since the original Response length is captured) if there is
enough knowledge of the commonly implemented name compression enough knowledge of the commonly implemented name compression
algorithms. For example, a simplistic approach would be to try each algorithms. For example, a simplistic approach would be to try each
algorithm in turn to see if it reproduces the original length, algorithm in turn to see if it reproduces the original length,
stopping at the first match. This would not guarantee the correct stopping at the first match. This would not guarantee that the
algorithm has been used as it is possible to match the length whilst correct algorithm has been used, as it is possible to match the
still not matching the on the wire bytes but, without further length whilst still not matching the on-the-wire bytes; however,
information added to the C-DNS data, this is the best that can be without further information added to the C-DNS data, this is the best
achieved. that can be achieved.
Appendix B presents an example of two different compression Appendix B presents an example of two different compression
algorithms used by well-known name server software. algorithms used by well-known name server software.
10. Data collection 10. Data Collection
This section describes a non-normative proposed algorithm for the This section describes a non-normative proposed algorithm for the
processing of a captured stream of DNS queries and responses and processing of a captured stream of DNS Queries and Responses and
production of a stream of query/response items, matching queries/ production of a stream of Q/R data items, matching Queries and
responses where possible. Responses where possible.
For the purposes of this discussion, it is assumed that the input has For the purposes of this discussion, it is assumed that the input has
been pre-processed such that: been preprocessed such that:
1. All IP fragmentation reassembly, TCP stream reassembly, and so 1. All IP fragmentation reassembly, TCP stream reassembly, and
on, has already been performed. so on, have already been performed.
2. Each message is associated with transport metadata required to 2. Each message is associated with transport metadata required to
generate the Primary ID (see Section 10.2.1). generate the Primary ID (see Section 10.2.1).
3. Each message has a well-formed DNS header of 12 bytes and (if 3. Each message has a well-formed DNS Header of 12 bytes, and (if
present) the first Question in the Question section can be parsed present) the first Question in the Question section can be parsed
to generate the Secondary ID (see below). As noted earlier, this to generate the Secondary ID (see below). As noted earlier, this
requirement can result in a malformed query being removed in the requirement can result in a malformed Query being removed in the
pre-processing stage, but the correctly formed response with preprocessing stage, but the correctly formed Response with RCODE
RCODE of FORMERR being present. of FORMERR being present.
DNS messages are processed in the order they are delivered to the DNS messages are processed in the order they are delivered to the
implementation. implementation.
It should be noted that packet capture libraries do not necessarily It should be noted that packet capture libraries do not necessarily
provide packets in strict chronological order. This can, for provide packets in strict chronological order. This can, for
example, arise on multi-core platforms where packets arriving at a example, arise on multi-core platforms where packets arriving at a
network device are processed by different cores. On systems where network device are processed by different cores. On systems where
this behaviour has been observed, the timestamps associated with each this behavior has been observed, the timestamps associated with each
packet are consistent; queries always have a timestamp prior to the packet are consistent; Queries always have a timestamp prior to the
response timestamp. However, the order in which these packets appear Response timestamp. However, the order in which these packets appear
in the packet capture stream is not necessarily strictly in the packet capture stream is not necessarily strictly
chronological; a response can appear in the capture stream before the chronological; a Response can appear in the capture stream before the
query that provoked the response. For this discussion, this non- Query that provoked the Response. For this discussion, this
chronological delivery is termed "skew". non-chronological delivery is termed "skew".
In the presence of skew, a response packets can arrive for matching In the presence of skew, Response packets can arrive for matching
before the corresponding query. To avoid generating false instances before the corresponding Query. To avoid generating false instances
of responses without a matching query, and queries without a matching of Responses without a matching Query, and Queries without a matching
response, the matching algorithm must take account of the possibility Response, the matching algorithm must take the possibility of skew
of skew. into account.
10.1. Matching algorithm 10.1. Matching Algorithm
A schematic representation of the algorithm for matching Q/R data A schematic representation of the algorithm for matching Q/R data
items is shown in Figure 3. It takes individual DNS query or items is shown in Figure 3. It takes individual DNS Query or
response messages as input, and outputs matched Q/R items. The Response messages as input, and it outputs matched Q/R data items.
numbers in the figure identify matching operations listed in Table 1. The numbers in the figure identify matching operations listed in
Specific details of the algorithm, for example queues, timers and Table 1. Specific details of the algorithm -- for example, queues,
identifiers, are given in the following sections. timers, and identifiers -- are given in the following sections.
.----------------------. .----------------------.
| Process next message |<------------------+ | Process next message |<------------------+
`----------------------' | `----------------------' |
| | | |
+------------------------------+ | +------------------------------+ |
| Generate message identifiers | | | Generate message identifiers | |
+------------------------------+ | +------------------------------+ |
| | | |
Response | Query | Response | Query |
+--------------< >---------------+ | +--------------< >---------------+ |
| | | | | |
+--------------------+ +--------------------+ | +--------------------+ +--------------------+ |
| Find earliest QR | | Create QR item [2] | | | Find earliest QR | | Create QR item (2) | |
| item in OFIFO [1] | +--------------------+ | | item in OFIFO (1) | +--------------------+ |
+--------------------+ | | +--------------------+ | |
| +---------------+ | | +---------------+ |
Match | No match | Append new QR | | Match | No match | Append new QR | |
+--------< >------+ | item to OFIFO | | +--------< >------+ | item to OFIFO | |
| | +---------------+ | | | +---------------+ |
+-----------+ +--------+ | | +-----------+ +--------+ | |
| Update QR | | Add to | +-------------------+ | | Update QR | | Add to | +-------------------+ |
| item [3] | | RFIFO | | Find earliest QR | | | item (3) | | RFIFO | | Find earliest QR | |
+-----------+ +--------+ | item in RFIFO [1] | | +-----------+ +--------+ | item in RFIFO (1) | |
| | +-------------------+ | | | +-------------------+ |
+-----------------+ | | +-----------------+ | |
| | | | | |
| +----------------+ Match | No match | | +----------------+ Match | No match |
| | Remove R |-------< >-----+ | | | Remove R |-------< >-----+ |
| | from RFIFO [3] | | | | | from RFIFO (3) | | |
| +----------------+ | | | +----------------+ | |
| | | | | | | |
+--------------+-----------------------+ | +--------------+-----------------------+ |
| | | |
+----------------------------------------------+ | +----------------------------------------------+ |
| Update all timed out (QT) OFIFO QR items [4] | | | Update all timed-out (QT) OFIFO QR items (4) | |
+----------------------------------------------+ | +----------------------------------------------+ |
| | | |
+--------------------------------+ | +--------------------------------+ |
| Remove all timed out (ST) R | | | Remove all timed-out (ST) R | |
| from RFIFO, create QR item [5] | | | from RFIFO, create QR item (5) | |
+--------------------------------+ | +--------------------------------+ |
____________________|_______________________ | ____________________|_______________________ |
/ / | / / |
/ Remove all consecutive done entries from /-------+ / Remove all consecutive done entries from /-------+
/ front of OFIFO for further processing / / front of OFIFO for further processing /
/____________________________________________/ /____________________________________________/
OFIFO = output FIFO containing Q/R data items (Section 10.6)
RFIFO = Response FIFO containing unmatched Response items
(Section 10.6)
QT = Query Timeout (Section 10.3)
ST = Skew Timeout (Section 10.3)
Figure 3: Query/Response matching algorithm Figure 3: Query/Response Matching Algorithm
+-----+-------------------------------------------+ +-----------+-------------------------------------------+
| Ref | Operation | | Reference | Operation |
+-----+-------------------------------------------+ +-----------+-------------------------------------------+
| [1] | Find earliest QR item in FIFO where: | | (1) | Find earliest QR item in FIFO where: |
| | * QR.done = false | | | * QR.done = false |
| | * QR.Q.PrimaryID == R.PrimaryID | | | * QR.Q.PrimaryID == R.PrimaryID |
| | and, if both QR.Q and R have SecondaryID: | | | and, if both QR.Q and R have SecondaryID: |
| | * QR.Q.SecondaryID == R.SecondaryID | | | * QR.Q.SecondaryID == R.SecondaryID |
| | | | | |
| [2] | Set: | | (2) | Set: |
| | QR.Q := Q | | | QR.Q := Q |
| | QR.R := nil | | | QR.R := nil |
| | QR.done := false | | | QR.done := false |
| | | | | |
| [3] | Set: | | (3) | Set: |
| | QR.R := R | | | QR.R := R |
| | QR.done := true | | | QR.done := true |
| | | | | |
| [4] | Set: | | (4) | Set: |
| | QR.done := true | | | QR.done := true |
| | | | | |
| [5] | Set: | | (5) | Set: |
| | QR.Q := nil | | | QR.Q := nil |
| | QR.R := R | | | QR.R := R |
| | QR.done := true | | | QR.done := true |
+-----+-------------------------------------------+ +-----------+-------------------------------------------+
Table 1: Operations used in the matching algorithm Table 1: Operations Used in the Matching Algorithm
10.2. Message identifiers 10.2. Message Identifiers
10.2.1. Primary ID (required) 10.2.1. Primary ID (Required)
A Primary ID is constructed for each message. It is composed of the A Primary ID is constructed for each message. It is composed of the
following data: following data:
1. Source IP Address 1. Source IP Address
2. Destination IP Address 2. Destination IP Address
3. Source Port 3. Source Port
4. Destination Port 4. Destination Port
5. Transport 5. Transport
6. DNS Message ID 6. DNS Message ID
10.2.2. Secondary ID (optional) 10.2.2. Secondary ID (Optional)
If present, the first Question in the Question section is used as a If present, the first Question in the Question section is used as a
secondary ID for each message. Note that there may be well formed Secondary ID for each message. Note that there may be well-formed
DNS queries that have a QDCOUNT of 0, and some responses may have a DNS Queries that have a QDCOUNT of 0, and some Responses may have a
QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). QDCOUNT of 0 (for example, Responses with RCODE=FORMERR or NOTIMP).
In this case the secondary ID is not used in matching. In this case, the Secondary ID is not used in matching.
10.3. Algorithm parameters 10.3. Algorithm Parameters
1. Query timeout, QT. A query arrives with timestamp t1. If no 1. Query Timeout (QT). A Query arrives with timestamp t1. If no
response matching that query has arrived before other input Response matching that Query has arrived before other input
arrives timestamped later than (t1 + QT), a query/response item arrives timestamped later than (t1 + QT), a Q/R data item
containing only a query item is recorded. The query timeout containing only a Query is recorded. The QT value is typically
value is typically of the order of 5 seconds. on the order of 5 seconds.
2. Skew timeout, ST. A response arrives with timestamp t2. If a 2. Skew Timeout (ST). A Response arrives with timestamp t2. If a
response has not been matched by a query before input arrives Response has not been matched by a Query before input arrives
timestamped later than (t2 + ST), a query/response item timestamped later than (t2 + ST), a Q/R data item containing only
containing only a response is recorded. The skew timeout value a Response is recorded. The ST value is typically a few
is typically a few microseconds. microseconds.
10.4. Algorithm requirements 10.4. Algorithm Requirements
The algorithm is designed to handle the following input data: The algorithm is designed to handle the following input data:
1. Multiple queries with the same Primary ID (but different 1. Multiple Queries with the same Primary ID (but different
Secondary ID) arriving before any responses for these queries are Secondary ID) arriving before any Responses for these Queries
seen. are seen.
2. Multiple queries with the same Primary and Secondary ID arriving 2. Multiple Queries with the same Primary and Secondary ID arriving
before any responses for these queries are seen. before any Responses for these Queries are seen.
3. Queries for which no later response can be found within the 3. Queries for which no later Response can be found within the
specified timeout. specified timeout.
4. Responses for which no previous query can be found within the 4. Responses for which no previous Query can be found within the
specified timeout. specified timeout.
10.5. Algorithm limitations 10.5. Algorithm Limitations
For cases 1 and 2 listed in the above requirements, it is not For cases 1 and 2 listed in the above requirements, it is not
possible to unambiguously match queries with responses. This possible to unambiguously match Queries with Responses. This
algorithm chooses to match to the earliest query with the correct algorithm chooses to match to the earliest Query with the correct
Primary and Secondary ID. Primary and Secondary ID.
10.6. Workspace 10.6. Workspace
The algorithm employs two FIFO queues: The algorithm employs two FIFO queues:
o OFIFO, an output FIFO containing Q/R items in chronological order, o OFIFO: an output FIFO containing Q/R data items in chronological
order.
o RFIFO, a FIFO holding responses without a matching query in order o RFIFO: a FIFO holding Responses without a matching Query in order
of arrival. of arrival.
10.7. Output 10.7. Output
The output is a list of Q/R data items. Both the Query and Response The output is a list of Q/R data items. Both the Query and Response
elements are optional in these items, therefore Q/R data items have elements are optional in these items; therefore, Q/R data items have
one of three types of content: one of three types of content:
1. A matched pair of query and response messages 1. A matched pair of Query and Response messages
2. A query message with no response 2. A Query message with no Response
3. A response message with no query 3. A Response message with no Query
The timestamp of a list item is that of the query for cases 1 and 2 The timestamp of a list item is that of the Query for cases 1 and 2
and that of the response for case 3. and that of the Response for case 3.
10.8. Post processing 10.8. Post-Processing
When ending capture, all items in the responses FIFO are timed out When ending a capture, all items in the RFIFO are timed out
immediately, generating response-only entries to the Q/R data item immediately, generating Response only entries to the OFIFO. These
FIFO. These and all other remaining entries in the Q/R data item and all other remaining entries in the OFIFO should be treated as
FIFO should be treated as timed out queries. timed-out Queries.
11. Implementation guidance 11. Implementation Guidance
Whilst this document makes no specific recommendations with respect Whilst this document makes no specific recommendations with respect
to Canonical CBOR (see Section 3.9 of [RFC7049]) the following to "Canonical CBOR" (see Section 3.9 of [RFC7049]), the following
guidance may be of use to implementors. guidance may be of use to implementers.
Adherence to the first two rules given in Section 3.9 of [RFC7049] Adherence to the first two rules given in Section 3.9 of [RFC7049]
will minimise file sizes. will minimize file sizes.
Adherence to the last two rules given in Section 3.9 of [RFC7049] for Adherence to the last two rules given in Section 3.9 of [RFC7049] for
all maps and arrays would unacceptably constrain implementations, for all maps and arrays would unacceptably constrain implementations --
example, in the use case of real-time data collection in constrained for example, in the use case of real-time data collection in
environments where outputting block tables after query/response data constrained environments where outputting Block Tables after Q/R data
and allowing indefinite length maps and arrays could reduce memory items and allowing indefinite-length maps and arrays could reduce
requirements. memory requirements.
11.1. Optional data It is recommended that implementations that have fundamental
restrictions on what data fields they can collect SHOULD always store
hints with the bits unset for those fields, i.e., they unambiguously
indicate that those data fields will be omitted from captured C-DNS.
When decoding C-DNS data some of the items required for a particular 11.1. Optional Data
When decoding C-DNS data, some of the items required for a particular
function that the consumer wishes to perform may be missing. function that the consumer wishes to perform may be missing.
Consumers should consider providing configurable default values to be Consumers should consider providing configurable default values to be
used in place of the missing values in their output. used in place of the missing values in their output.
11.2. Trailing bytes 11.2. Trailing Bytes
A DNS query message in a UDP or TCP payload can be followed by some A DNS Query message in a UDP or TCP payload can be followed by some
additional (spurious) bytes, which are not stored in C-DNS. additional (spurious) bytes, which are not stored in C-DNS.
When DNS traffic is sent over TCP, each message is prefixed with a When DNS traffic is sent over TCP, each message is prefixed with a
two byte length field which gives the message length, excluding the two-byte length field, which gives the message length, excluding the
two byte length field. In this context, trailing bytes can occur in two-byte length field. In this context, trailing bytes can occur in
two circumstances with different results: two circumstances, with different results:
1. The number of bytes consumed by fully parsing the message is less 1. The number of bytes consumed by fully parsing the message is less
than the number of bytes given in the length field (i.e. the than the number of bytes given in the length field (i.e., the
length field is incorrect and too large). In this case, the length field is incorrect and too large). In this case, the
surplus bytes are considered trailing bytes in an analogous surplus bytes are considered trailing bytes in a manner analogous
manner to UDP and recorded as such. If only this case occurs it to UDP and recorded as such. If only this case occurs, it is
is possible to process a packet containing multiple DNS messages possible to process a packet containing multiple DNS messages
where one or more has trailing bytes. where one or more have trailing bytes.
2. There are surplus bytes between the end of a well-formed message 2. There are surplus bytes between the end of a well-formed message
and the start of the length field for the next message. In this and the start of the length field for the next message. In this
case the first of the surplus bytes will be processed as the case, the first of the surplus bytes will be processed as the
first byte of the next length field, and parsing will proceed first byte of the next length field, and parsing will proceed
from there, almost certainly leading to the next and any from there, almost certainly leading to the next and any
subsequent messages in the packet being considered malformed. subsequent messages in the packet being considered malformed.
This will not generate a trailing bytes record for the processed This will not generate a trailing-bytes record for the processed
well-formed message. well-formed message.
11.3. Limiting collection of RDATA 11.3. Limiting Collection of RDATA
Implementations should consider providing a configurable maximum Implementations should consider providing a configurable maximum
RDATA size for capture, for example, to avoid memory issues when RDATA size for captures -- for example, to avoid memory issues when
confronted with large XFR records. confronted with large zone transfer records.
11.4. Timestamps 11.4. Timestamps
The preamble to each block includes a timestamp of the earliest The preamble to each block includes a timestamp of the earliest
record in the block. As described in Section 7.5.1, the timestamp is record in the Block. As described in Section 7.3.2.1, the timestamp
an array of 2 unsigned integers. The first is a POSIX "time_t" is an array of two unsigned integers. The first is a POSIX "time_t"
[posix-time]. Consumers of C-DNS should be aware of this as it [posix-time]. Consumers of C-DNS should be aware of this, as it
excludes leap seconds and therefore may cause minor anomalies in the excludes leap seconds and therefore may cause minor anomalies in the
data e.g. when calculating query throughput. data, e.g., when calculating Query throughput.
12. Implementation status
[Note to RFC Editor: please remove this section and reference to
[RFC7942] prior to publication.]
This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to
assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation
here does not imply endorsement by the IETF. Furthermore, no effort
has been spent to verify the information presented here that was
supplied by IETF contributors. This is not intended as, and must not
be construed to be, a catalog of available implementations or their
features. Readers are advised to note that other implementations may
exist.
According to [RFC7942], "this will allow reviewers and working groups
to assign due consideration to documents that have the benefit of
running code, which may serve as evidence of valuable experimentation
and feedback that have made the implemented protocols more mature.
It is up to the individual working groups to use this information as
they see fit".
12.1. DNS-STATS Compactor
ICANN/Sinodun IT have developed an open source implementation called
DNS-STATS Compactor. The Compactor is a suite of tools which can
capture DNS traffic (from either a network interface or a PCAP file)
and store it in the Compacted-DNS (C-DNS) file format. PCAP files
for the captured traffic can also be reconstructed. See Compactor
[1].
This implementation:
o covers the whole of the specification described in the -03 draft
with the exception of support for malformed messages and pico
second time resolution. (Note: this implementation does allow
malformed messages to be recorded separately in a PCAP file).
o is released under the Mozilla Public License Version 2.0.
o has a users mailing list available, see dns-stats-users [2].
There is also some discussion of issues encountered during
development available at Compressing Pcap Files [3] and Packet
Capture [4].
This information was last updated on 3rd of May 2018.
13. IANA considerations 12. IANA Considerations
IANA is requested to create a registry "C-DNS DNS Capture Format" IANA has created a registry "C-DNS DNS Capture Format" containing the
containing the subregistries defined in sections Section 13.1 to subregistries defined in Sections 12.1 to 12.4 inclusive.
Section 13.4 inclusive.
In all cases, new entries may be added to the subregistries by Expert In all cases, new entries may be added to the subregistries by Expert
Review as defined in [RFC8126]. Experts are expected to exercise Review as defined in [RFC8126]. Experts are expected to exercise
their own expert judgement, and should consider the following general their own expert judgment and should consider the following general
guidelines in addition to any guidelines given particular to a guidelines in addition to any provided guidelines that are particular
subregistry. to a subregistry.
o There should be a real and compelling use for any new value. o There should be a real and compelling use for any new value.
o Values assigned should be carefully chosen to minimise storage o Values assigned should be carefully chosen to minimize storage
requirements for common cases. requirements for common cases.
13.1. Transport types 12.1. Transport Types
IANA is requested to create a registry "C-DNS Transports" of C-DNS IANA has created a registry "C-DNS Transports" of C-DNS transport
transport type identifiers. The primary purpose of this registry is type identifiers. The primary purpose of this registry is to provide
to provide unique identifiers for all transports used for DNS unique identifiers for all transports used for DNS Queries.
queries.
The following note is included in this registry: "In version 1.0 of The following note is included in this registry: "In version 1.0 of
C-DNS [[this RFC]], there is a field to identify the type of DNS C-DNS [RFC8618], there is a field to identify the type of DNS
transport. This field is 4 bits in size." transport. This field is 4 bits in size."
The initial contents of the registry are as follows. See
Sections 7.3.2.3.2, 7.3.2.3.5, and 7.3.2.5 of this document:
The initial contents of the registry are as follows - see sections +------------+------------------------+-----------+
Section 7.5.3.2 and Section 7.5.3.5 of [[this RFC]]: | Identifier | Name | Reference |
+------------+------------------------+-----------+
+------------+------------+--------------+ | 0 | UDP | RFC 8618 |
| Identifier | Name | Reference | | 1 | TCP | RFC 8618 |
+------------+------------+--------------+ | 2 | TLS | RFC 8618 |
| 0 | UDP | [[this RFC]] | | 3 | DTLS | RFC 8618 |
| 1 | TCP | [[this RFC]] | | 4 | HTTPS | RFC 8618 |
| 2 | TLS | [[this RFC]] | | 5-14 | Unassigned | |
| 3 | DTLS | [[this RFC]] | | 15 | Non-standard transport | RFC 8618 |
| 4 | DoH | [[this RFC]] | +------------+------------------------+-----------+
| 5-15 | Unassigned | |
+------------+------------+--------------+
Expert reviewers should take the following points into consideration:
o Is the requested DNS transport described by a Standards Track RFC? Expert reviewers should take the following point into consideration:
Is the requested DNS transport described by a Standards Track RFC?
13.2. Data storage flags 12.2. Data Storage Flags
IANA is requested to create a registry "C-DNS Storage Flags" of C-DNS IANA has created a registry "C-DNS Storage Flags" of C-DNS data
data storage flags. The primary purpose of this registry is to storage flags. The primary purpose of this registry is to provide
provide indicators giving hints on processing of the data stored. indicators giving hints on processing of the data stored.
The following note is included in this registry: "In version 1.0 of The following note is included in this registry: "In version 1.0 of
C-DNS [[this RFC]], there is a field describing attributes of the C-DNS [RFC8618], there is a field describing attributes of the data
data recorded. The field is a CBOR [RFC7049] unsigned integer recorded. The field is a CBOR [RFC7049] unsigned integer holding bit
holding bit flags." flags."
The initial contents of the registry are as follows - see section The initial contents of the registry are as follows. See
Section 7.4.1.1 of [[this RFC]]: Section 7.3.1.1.1 of this document:
+------+------------------+-----------------------------+-----------+ +------+------------------+-----------------------------+-----------+
| Bit | Name | Description | Reference | | Bit | Name | Description | Reference |
+------+------------------+-----------------------------+-----------+ +------+------------------+-----------------------------+-----------+
| 0 | anonymised-data | The data has been | [[this | | 0 | anonymized-data | The data has been | RFC 8618 |
| | | anonymised. | RFC]] | | | | anonymized. | |
| 1 | sampled-data | The data is sampled data. | [[this | | | | | |
| | | | RFC]] | | 1 | sampled-data | The data is sampled data. | RFC 8618 |
| 2 | normalized-names | Names in the data have been | [[this | | | | | |
| | | normalized. | RFC]] | | 2 | normalized-names | Names in the data have been | RFC 8618 |
| | | normalized. | |
| | | | |
| 3-63 | Unassigned | | | | 3-63 | Unassigned | | |
+------+------------------+-----------------------------+-----------+ +------+------------------+-----------------------------+-----------+
13.3. Response processing flags 12.3. Response-Processing Flags
IANA is requested to create a registry "C-DNS Response Flags" of IANA has created a registry "C-DNS Response Flags" of C-DNS response-
C-DNS response processing flags. The primary purpose of this processing flags. The primary purpose of this registry is to provide
registry is to provide indicators giving hints on the generation of a indicators giving hints on the generation of a particular Response.
particular response.
The following note is included in this registry: "In version 1.0 of The following note is included in this registry: "In version 1.0 of
C-DNS [[this RFC]], there is a field describing attributes of the C-DNS [RFC8618], there is a field describing attributes of the
responses recorded. The field is a CBOR [RFC7049] unsigned integer Responses recorded. The field is a CBOR [RFC7049] unsigned integer
holding bit flags." holding bit flags."
The initial contents of the registry are as follows - see section The initial contents of the registry are as follows. See
Section 7.6.1 of [[this RFC]]: Section 7.3.2.4.1 of this document:
+------+------------+-------------------------------+--------------+ +------+------------+-------------------------------+-----------+
| Bit | Name | Description | Reference | | Bit | Name | Description | Reference |
+------+------------+-------------------------------+--------------+ +------+------------+-------------------------------+-----------+
| 0 | from-cache | The response came from cache. | [[this RFC]] | | 0 | from-cache | The Response came from cache. | RFC 8618 |
| 1-63 | Unassigned | | | | 1-63 | Unassigned | | |
+------+------------+-------------------------------+--------------+ +------+------------+-------------------------------+-----------+
13.4. AddressEvent types 12.4. AddressEvent Types
IANA is requested to create a registry "C-DNS Address Event Types" of IANA has created a registry "C-DNS Address Event Types" of C-DNS
C-DNS AddressEvent types. The primary purpose of this registry is to AddressEvent types. The primary purpose of this registry is to
provide unique identifiers of different types of C-DNS address provide unique identifiers of different types of C-DNS address events
events, and so specify the contents of the optional companion field and so specify the contents of the optional companion field "ae-code"
"ae-code" for each type. for each type.
The following note is included in this registry: "In version 1.0 of The following note is included in this registry: "In version 1.0 of
C-DNS [[this RFC]], there is a field identify types of the events C-DNS [RFC8618], there is a field identifying types of the events
related to client addresses. This field is a CBOR [RFC7049] unsigned related to client addresses. This field is a CBOR [RFC7049] unsigned
integer. There is a related optional field "ae-code", which, if integer. There is a related optional field "ae-code", which, if
present, holds an additional CBOR unsigned integer giving additional present, holds an additional CBOR unsigned integer giving additional
information specific to the event type." information specific to the event type."
The initial contents of the registry are as follows. See
Section 7.3.2.5 of this document:
The initial contents of the registry are as follows - see section +------------------------+---------------+--------------+-----------+
Section 7.7: | Identifier | Event Type | ae-code | Reference |
| | | Contents | |
+------------+----------------------+-------------------+-----------+ +------------------------+---------------+--------------+-----------+
| Identifier | Event Type | ae-code contents | Reference | | 0 | TCP reset | None | RFC 8618 |
+------------+----------------------+-------------------+-----------+ | | | | |
| 0 | TCP reset | None | [[this | | 1 | ICMP time | ICMP code | RFC 8618 |
| | | | RFC]] | | | exceeded | [icmpcodes] | |
| 1 | ICMP time exceeded | ICMP code | [[this | | | | | |
| | | [icmpcodes] | RFC]] | | 2 | ICMP | ICMP code | RFC 8618 |
| 2 | ICMP destination | ICMP code | [[this | | | destination | [icmpcodes] | |
| | unreachable | [icmpcodes] | RFC]] | | | unreachable | | |
| 3 | ICMPv6 time exceeded | ICMPv6 code | [[this | | | | | |
| | | [icmp6codes] | RFC]] | | 3 | ICMPv6 time | ICMPv6 code | RFC 8618 |
| 4 | ICMPv6 destination | ICMPv6 code | [[this | | | exceeded | [icmp6codes] | |
| | unreachable | [icmp6codes] | RFC]] | | | | | |
| 5 | ICMPv6 packet too | ICMPv6 code | [[this | | 4 | ICMPv6 | ICMPv6 code | RFC 8618 |
| | big | [icmp6codes] | RFC]] | | | destination | [icmp6codes] | |
| >5 | Unassigned | | | | | unreachable | | |
+------------+----------------------+-------------------+-----------+ | | | | |
| 5 | ICMPv6 packet | ICMPv6 code | RFC 8618 |
Expert reviewers should take the following points into consideration: | | too big | [icmp6codes] | |
| | | | |
| 6-18446744073709551615 | Unassigned | | |
+------------------------+---------------+--------------+-----------+
o "ae-code" contents must be defined for a type, or if not Expert reviewers should take the following point into consideration:
appropriate specified as "None". A specification of "None" "ae-code" contents must be defined for a type or, if not appropriate,
requires less storage, and is therefore preferred. specified as "None". A specification of "None" requires less storage
and is therefore preferred.
14. Security considerations 13. Security Considerations
Any control interface MUST perform authentication and encryption. Any control interface MUST perform authentication and encryption.
Any data upload MUST be authenticated and encrypted. Any data upload MUST be authenticated and encrypted.
15. Privacy considerations 14. Privacy Considerations
Storage of DNS traffic by operators in PCAP and other formats is a Storage of DNS traffic by operators in PCAP and other formats is a
long standing and widespread practice. Section 2.5 of long-standing and widespread practice. Section 2.5 of
[I-D.bortzmeyer-dprive-rfc7626-bis] is an analysis of the risks to [DNS-Priv-Cons] provides an analysis of the risks to Internet users
Internet users of the storage of DNS traffic data in servers regarding the storage of DNS traffic data in servers (recursive
(recursive resolvers, authoritative and rogue servers). resolvers, authoritative servers, and rogue servers).
Section 5.2 of [I-D.dickinson-dprive-bcp-op] describes mitigations Section 5.2 of [DNS-Priv-Svc] describes mitigations for those risks
for those risks for data stored on recursive resolvers (but which for data stored on recursive resolvers (but that could by extension
could by extension apply to authoritative servers). These include apply to authoritative servers). These include data-handling
data handling practices and methods for data minimization, IP address practices and methods for data minimization, IP address
pseudonymization and anonymization. Appendix B of that document pseudonymization, and anonymization. Appendix C of [DNS-Priv-Svc]
presents an analysis of 7 published anonymization processes. In presents an analysis of seven published anonymization processes. In
addition, RSSAC have recently published RSSAC04: [5] " addition, the ICANN Root Server System Advisory Committee (RSSAC)
Recommendations on Anonymization Processes for Source IP Addresses have recently published [RSSAC04] ("Recommendations on Anonymization
Submitted for Future Analysis". Processes for Source IP Addresses Submitted for Future Analysis").
The above analyses consider full data capture (e.g using PCAP) as a The above analyses consider full data capture (e.g., using PCAP) as a
baseline for privacy considerations and therefore this format baseline for privacy considerations; therefore, this format
specification introduces no new user privacy issues beyond those of specification introduces no new user privacy issues beyond those of
full data capture (which are quite severe). It does provides full data capture (which are quite severe). It does provide
mechanisms to selectively record only certain fields at the time of mechanisms to selectively record only certain fields at the time of
data capture to improve user privacy and to explicitly indicate that data capture, to improve user privacy and to explicitly indicate that
data is sampled and or anonymized. It also provide flags to indicate data is sampled, anonymized, or both. It also provides flags to
if data normalization has been performed; data normalization indicate if data normalization has been performed; data normalization
increases user privacy by reducing the potential for fingerprinting increases user privacy by reducing the potential for fingerprinting
individuals, however, a trade-off is potentially reducing the individuals. However, a trade-off is the potential reduction of the
capacity to identify attack traffic via query name signatures. capacity to identify attack traffic via Query name signatures.
Operators should carefully consider their operational requirements Operators should carefully consider their operational requirements
and privacy policies and SHOULD capture at source the minimum user and privacy policies and SHOULD capture at the source the minimum
data required to meet their needs. user data required to meet their needs.
16. Acknowledgements
The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for
many useful discussions on binary formats, compression and packet
matching. Also Jan Vcelak and Wouter Wijngaards for discussions on
name compression and Paul Hoffman for a detailed review of the
document and the C-DNS CDDL.
Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson,
Stephane Bortzmeyer and many other members of DNSOP for review.
Also, Miek Gieben for mmark [6]
17. Changelog
draft-ietf-dnsop-dns-capture-format-10
o Add IANA Considerations
o Convert graph in C.6 to table
draft-ietf-dnsop-dns-capture-format-09
o Editorial changes arising from IESG review
o *-transport-flags and may be mandatory in some configurations
o Mark fields that are conditionally mandatory
o Change `promisc' flag CDDL data type to boolean
o Add ranges to configuration quantities where appropriate
draft-ietf-dnsop-dns-capture-format-08
o Convert diagrams to ASCII
o Describe versioning
o Fix unused group warning in CDDL
draft-ietf-dnsop-dns-capture-format-07
o Resolve outstanding questions and TODOs
o Make RR RDATA optional
o Update matching diagram and explain skew
o Add count of discarded messages to block statistics
o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-06
o Correct BlockParameters type to map
o Make RR ttl optional
o Add storage flag indicating name normalization
o Add storage parameter fields for sampling and anonymization
methods
o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-05
o Make all data items in Q/R, QuerySignature and Malformed Message
arrays optional
o Re-structure the FilePreamble and ConfigurationParameters into
BlockParameters
o BlockParameters has separate Storage and Collection Parameters
o Storage Parameters includes information on what optional fields
are present, and flags specifying anonymization or sampling
o Addresses can now be stored as prefixes.
o Switch to using a variable sub-second timing granularity
o Add response bailiwick and query response type
o Add specifics of how to record malformed messages
o Add implementation guidance
o Improve terminology and naming consistency
draft-ietf-dnsop-dns-capture-format-04
o Correct query-d0 to query-do in CDDL
o Clarify that map keys are unsigned integers
o Add Type to Class/Type table
o Clarify storage format in section 7.12
draft-ietf-dnsop-dns-capture-format-03
o Added an Implementation Status section
draft-ietf-dnsop-dns-capture-format-02
o Update qr_data_format.png to match CDDL
o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-01
o Many editorial improvements by Paul Hoffman
o Included discussion of malformed message handling
o Improved Appendix C on Comparison of Binary Formats
o Now using C-DNS field names in the tables in section 8
o A handful of new fields included (CDDL updated)
o Timestamps now include optional picoseconds
o Added details of block statistics
draft-ietf-dnsop-dns-capture-format-00
o Changed dnstap.io to dnstap.info
o qr_data_format.png was cut off at the bottom
o Update authors address
o Improve wording in Abstract
o Changed DNS-STAT to C-DNS in CDDL
o Set the format version in the CDDL
o Added a TODO: Add block statistics
o Added a TODO: Add extend to support pico/nano. Also do this for
Time offset and Response delay
o Added a TODO: Need to develop optional representation of malformed
messages within C-DNS and what this means for packet matching.
This may influence which fields are optional in the rest of the
representation.
o Added section on design goals to Introduction
o Added a TODO: Can Class be optimised? Should a class of IN be
inferred if not present?
draft-dickinson-dnsop-dns-capture-format-00
o Initial commit
18. References
18.1. Normative References 15. References
[I-D.ietf-cbor-cddl] 15.1. Normative References
Birkholz, H., Vigano, C., and C. Bormann, "Concise data
definition language (CDDL): a notational convention to
express CBOR and JSON data structures", draft-ietf-cbor-
cddl-06 (work in progress), November 2018.
[pcap-filter] [pcap-filter]
tcpdump.org, "Manpage of PCAP-FILTER", 2017, tcpdump.org, "Manpage of PCAP-FILTER", November 2017,
<http://www.tcpdump.org/manpages/pcap-filter.7.html>. <https://www.tcpdump.org/manpages/pcap-filter.7.html>.
[pcap-options] [pcap-options]
tcpdump.org, "Manpage of PCAP", 2018, tcpdump.org, "Manpage of PCAP", July 2018,
<http://www.tcpdump.org/manpages/pcap.3pcap.html>. <https://www.tcpdump.org/manpages/pcap.3pcap.html>.
[posix-time] [posix-time]
The Open Group, "Section 4.16, Base Definitions, Standard The Open Group, "IEEE Standard for Information
for Information Technology - Portable Operating System Technology--Portable Operating System Interface (POSIX(R))
Interface (POSIX(R)) Base Specifications, Issue 7", IEEE Base Specifications, Issue 7", IEEE Standard 1003.1-2017,
Standard 1003.1 2017 Edition, Section 4.16, DOI 10.1109/IEEESTD.2018.8277153.
DOI 10.1109/IEEESTD.2018.8277153, 2017.
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, [RFC792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, DOI 10.17487/RFC0792, September 1981, RFC 792, DOI 10.17487/RFC0792, September 1981,
<https://www.rfc-editor.org/info/rfc792>. <https://www.rfc-editor.org/info/rfc792>.
[RFC1035] Mockapetris, P., "Domain names - implementation and [RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>. November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
skipping to change at page 55, line 22 skipping to change at page 54, line 36
for DNS (EDNS(0))", STD 75, RFC 6891, for DNS (EDNS(0))", STD 75, RFC 6891,
DOI 10.17487/RFC6891, April 2013, DOI 10.17487/RFC6891, April 2013,
<https://www.rfc-editor.org/info/rfc6891>. <https://www.rfc-editor.org/info/rfc6891>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <https://www.rfc-editor.org/info/rfc7049>. October 2013, <https://www.rfc-editor.org/info/rfc7049>.
[RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D., [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D.,
and P. Hoffman, "Specification for DNS over Transport and P. Hoffman, "Specification for DNS over Transport
Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858,
2016, <https://www.rfc-editor.org/info/rfc7858>. May 2016, <https://www.rfc-editor.org/info/rfc7858>.
[RFC8094] Reddy, T., Wing, D., and P. Patil, "DNS over Datagram
Transport Layer Security (DTLS)", RFC 8094,
DOI 10.17487/RFC8094, February 2017,
<https://www.rfc-editor.org/info/rfc8094>.
[RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for
Writing an IANA Considerations Section in RFCs", BCP 26, Writing an IANA Considerations Section in RFCs", BCP 26,
RFC 8126, DOI 10.17487/RFC8126, June 2017, RFC 8126, DOI 10.17487/RFC8126, June 2017,
<https://www.rfc-editor.org/info/rfc8126>. <https://www.rfc-editor.org/info/rfc8126>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, RFC 2119 Key Words", BCP 14, RFC 8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>. DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.
[RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS [RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS
(DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018, (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018,
<https://www.rfc-editor.org/info/rfc8484>. <https://www.rfc-editor.org/info/rfc8484>.
18.2. Informative References [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/info/rfc8610>.
[ditl] DNS-OARC, "DITL", 2016, 15.2. Informative References
[Avro] The Apache Software Foundation, "Apache Avro(TM)", 2019,
<https://avro.apache.org/>.
[ditl] DNS-OARC, "DITL", 2018,
<https://www.dns-oarc.net/oarc/data/ditl>. <https://www.dns-oarc.net/oarc/data/ditl>.
[dnscap] DNS-OARC, "DNSCAP", 2016, [DNS-Priv-Cons]
Bortzmeyer, S. and S. Dickinson, "DNS Privacy
Considerations", Work in Progress,
draft-ietf-dprive-rfc7626-bis-00, July 2019.
[DNS-Priv-Svc]
Dickinson, S., Overeinder, B., van Rijswijk-Deij, R., and
A. Mankin, "Recommendations for DNS Privacy Service
Operators", Work in Progress, draft-ietf-dprive-bcp-op-03,
July 2019.
[dnscap] DNS-OARC, "DNSCAP", 2018,
<https://www.dns-oarc.net/tools/dnscap>. <https://www.dns-oarc.net/tools/dnscap>.
[dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dnstap] "dnstap", 2016, <https://dnstap.info/>.
[dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, [dnstap-schema]
<https://www.dns-oarc.net/tools/dsc>. "dnstap schema", commit d860ec1, November 2016,
<https://github.com/dnstap/dnstap.pb/blob/master/
dnstap.proto>.
[I-D.bortzmeyer-dprive-rfc7626-bis] [dnsxml] Daley, J., Ed., Morris, S., and J. Dickinson, "dnsxml - A
Bortzmeyer, S. and S. Dickinson, "DNS Privacy standard XML representation of DNS data", Work in
Considerations", draft-bortzmeyer-dprive-rfc7626-bis-01 Progress, draft-daley-dnsxml-00, July 2013.
(work in progress), July 2018.
[I-D.daley-dnsxml] [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
Daley, J., Morris, S., and J. Dickinson, "dnsxml - A <https://www.dns-oarc.net/tools/dsc>.
standard XML representation of DNS data", draft-daley-
dnsxml-00 (work in progress), July 2013.
[I-D.dickinson-dprive-bcp-op] [gzip] "gzip", <https://www.gzip.org/>.
Dickinson, S., Overeinder, B., Rijswijk-Deij, R., and A.
Mankin, "Recommendations for DNS Privacy Service
Operators", draft-dickinson-dprive-bcp-op-01 (work in
progress), July 2018.
[icmp6codes] [icmp6codes]
IANA, "ICMPv6 "Code" Fields", 2018, IANA, "ICMPv6 "Code" Fields",
<https://www.iana.org/assignments/icmpv6-parameters/ <https://www.iana.org/assignments/icmpv6-parameters/>.
icmpv6-parameters.xhtml#icmpv6-parameters-3>.
[icmpcodes] [icmpcodes]
IANA, "Code Fields", 2018, IANA, "Code Fields",
<https://www.iana.org/assignments/icmp-parameters/ <https://www.iana.org/assignments/icmp-parameters/>.
icmp-parameters.xhtml#icmp-parameters-codes>.
[IEEE802.1Q] [IEEE802.1Q]
IEEE, "IEEE Standard for Local and metropolitan area IEEE, "IEEE Standard for Local and Metropolitan Area
networks -- Bridges and Bridged Networks", Networks--Bridges and Bridged Networks", IEEE
DOI 10.1109/IEEESTD.2014.6991462, 2014. Standard 802.1Q.
[opcodes] IANA, "DNS OpCodes", 2018, [Knot] "Knot DNS", <https://www.knot-dns.cz/>.
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-5>. [lz4] "LZ4", <https://lz4.github.io/lz4/>.
[mmark] Gieben, M., "mmark", commit de69698, May 2019,
<https://github.com/mmarkdown/mmark>.
[NSD] NLnet Labs, "NSD", 2019,
<https://www.nlnetlabs.nl/projects/nsd/about/>.
[opcodes] IANA, "DNS OpCodes",
<https://www.iana.org/assignments/dns-parameters/>.
[packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
2014, <https://github.com/dotse/PacketQ>. commit c9b2e89, February 2019,
<https://github.com/DNS-OARC/PacketQ>.
[pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcap] "PCAP", 2019, <https://www.tcpdump.org/>.
[pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. [pcapng] "pcapng: PCAP next generation file format specification",
Harris, "pcap-ng", 2016, commit 3c35b6a, March 2019,
<https://github.com/pcapng/pcapng>. <https://github.com/pcapng/pcapng>.
[rcodes] IANA, "DNS RCODEs", 2018, [Protocol-Buffers]
<http://www.iana.org/assignments/dns-parameters/ Google LLC, "Protocol Buffers",
dns-parameters.xhtml#dns-parameters-6>. <https://developers.google.com/protocol-buffers/>.
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running [rcodes] IANA, "DNS RCODEs",
Code: The Implementation Status Section", BCP 205, <https://www.iana.org/assignments/dns-parameters/>.
RFC 7942, DOI 10.17487/RFC7942, July 2016,
<https://www.rfc-editor.org/info/rfc7942>. [RFC5077] Salowey, J., Zhou, H., Eronen, P., and H. Tschofenig,
"Transport Layer Security (TLS) Session Resumption without
Server-Side State", RFC 5077, DOI 10.17487/RFC5077,
January 2008, <https://www.rfc-editor.org/info/rfc5077>.
[RFC7413] Cheng, Y., Chu, J., Radhakrishnan, S., and A. Jain, "TCP
Fast Open", RFC 7413, DOI 10.17487/RFC7413, December 2014,
<https://www.rfc-editor.org/info/rfc7413>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259, Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017, DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>. <https://www.rfc-editor.org/info/rfc8259>.
[RFC8427] Hoffman, P., "Representing DNS Messages in JSON", [RFC8427] Hoffman, P., "Representing DNS Messages in JSON",
RFC 8427, DOI 10.17487/RFC8427, July 2018, RFC 8427, DOI 10.17487/RFC8427, July 2018,
<https://www.rfc-editor.org/info/rfc8427>. <https://www.rfc-editor.org/info/rfc8427>.
[rrclasses] [rrclasses]
IANA, "DNS CLASSes", 2018, IANA, "DNS CLASSes",
<http://www.iana.org/assignments/dns-parameters/ <https://www.iana.org/assignments/dns-parameters/>.
dns-parameters.xhtml#dns-parameters-2>.
[rrtypes] IANA, "Resource Record (RR) TYPEs", 2018,
<http://www.iana.org/assignments/dns-parameters/
dns-parameters.xhtml#dns-parameters-4>.
18.3. URIs
[1] https://github.com/dns-stats/compactor/wiki
[2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
[3] https://www.sinodun.com/2017/06/compressing-pcap-files/
[4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu-
trusty-packet-capture-woes/
[5] https://www.icann.org/en/system/files/files/rssac-
040-07aug18-en.pdf
[6] https://github.com/miekg/mmark
[7] https://www.nlnetlabs.nl/projects/nsd/
[8] https://www.knot-dns.cz/
[9] https://avro.apache.org/
[10] https://developers.google.com/protocol-buffers/
[11] http://cbor.io
[12] https://github.com/kubo/snzip [rrtypes] IANA, "Resource Record (RR) TYPEs",
<https://www.iana.org/assignments/dns-parameters/>.
[13] http://google.github.io/snappy/ [RSSAC04] ICANN, "Recommendations on Anonymization Processes for
Source IP Addresses Submitted for Future Analysis",
August 2018, <https://www.icann.org/en/system/files/files/
rssac-040-07aug18-en.pdf>.
[14] http://lz4.github.io/lz4/ [snappy] "snappy", <https://google.github.io/snappy/>.
[15] http://www.gzip.org/ [snzip] "Snzip, a compression/decompression tool based on snappy",
commit 809c6f2, October 2018,
<https://github.com/kubo/snzip>.
[16] http://facebook.github.io/zstd/ [xz] "XZ Utils", <https://tukaani.org/xz/>.
[17] http://tukaani.org/xz/ [zstd] "Zstandard - Real-time data compression algorithm",
<https://facebook.github.io/zstd/>.
Appendix A. CDDL Appendix A. CDDL
This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for This appendix gives a CDDL [RFC8610] specification for C-DNS.
C-DNS.
CDDL does not permit a range of allowed values to be specified for a CDDL does not permit a range of allowed values to be specified for a
bitfield. Where necessary, those values are given as a CDDL group, bitfield. Where necessary, those values are given as a CDDL group,
but the group definition is commented out to prevent CDDL tooling but the group definition is commented out to prevent CDDL tooling
from warning that the group is unused. from warning that the group is unused.
; CDDL specification of the file format for C-DNS, ; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS messages and ; which describes a collection of DNS messages and
; traffic meta-data. ; traffic metadata.
; ;
; The overall structure of a file. ; The overall structure of a file.
; ;
File = [ File = [
file-type-id : "C-DNS", file-type-id : "C-DNS",
file-preamble : FilePreamble, file-preamble : FilePreamble,
file-blocks : [* Block], file-blocks : [* Block],
] ]
; ;
; The file preamble. ; The File Preamble.
; ;
FilePreamble = { FilePreamble = {
major-format-version => 1, major-format-version => 1,
minor-format-version => 0, minor-format-version => 0,
? private-version => uint, ? private-version => uint,
block-parameters => [+ BlockParameters], block-parameters => [+ BlockParameters],
} }
major-format-version = 0 major-format-version = 0
minor-format-version = 1 minor-format-version = 1
private-version = 2 private-version = 2
skipping to change at page 59, line 28 skipping to change at page 59, line 16
max-block-items => uint, max-block-items => uint,
storage-hints => StorageHints, storage-hints => StorageHints,
opcodes => [+ OpcodeRange], opcodes => [+ OpcodeRange],
rr-types => [+ RRTypeRange], rr-types => [+ RRTypeRange],
? storage-flags => StorageFlags, ? storage-flags => StorageFlags,
? client-address-prefix-ipv4 => IPv4PrefixLength, ? client-address-prefix-ipv4 => IPv4PrefixLength,
? client-address-prefix-ipv6 => IPv6PrefixLength, ? client-address-prefix-ipv6 => IPv6PrefixLength,
? server-address-prefix-ipv4 => IPv4PrefixLength, ? server-address-prefix-ipv4 => IPv4PrefixLength,
? server-address-prefix-ipv6 => IPv6PrefixLength, ? server-address-prefix-ipv6 => IPv6PrefixLength,
? sampling-method => tstr, ? sampling-method => tstr,
? anonymisation-method => tstr, ? anonymization-method => tstr,
} }
ticks-per-second = 0 ticks-per-second = 0
max-block-items = 1 max-block-items = 1
storage-hints = 2 storage-hints = 2
opcodes = 3 opcodes = 3
rr-types = 4 rr-types = 4
storage-flags = 5 storage-flags = 5
client-address-prefix-ipv4 = 6 client-address-prefix-ipv4 = 6
client-address-prefix-ipv6 = 7 client-address-prefix-ipv6 = 7
server-address-prefix-ipv4 = 8 server-address-prefix-ipv4 = 8
server-address-prefix-ipv6 = 9 server-address-prefix-ipv6 = 9
sampling-method = 10 sampling-method = 10
anonymisation-method = 11 anonymization-method = 11
; A hint indicates if the collection method will output the ; A hint indicates whether the collection method will always omit
; item or will ignore the item if present. ; the item from the file.
StorageHints = { StorageHints = {
query-response-hints => QueryResponseHints, query-response-hints => QueryResponseHints,
query-response-signature-hints => query-response-signature-hints =>
QueryResponseSignatureHints, QueryResponseSignatureHints,
rr-hints => RRHints, rr-hints => RRHints,
other-data-hints => OtherDataHints, other-data-hints => OtherDataHints,
} }
query-response-hints = 0 query-response-hints = 0
query-response-signature-hints = 1 query-response-signature-hints = 1
rr-hints = 2 rr-hints = 2
skipping to change at page 60, line 22 skipping to change at page 60, line 10
client-port : 2, client-port : 2,
transaction-id : 3, transaction-id : 3,
qr-signature-index : 4, qr-signature-index : 4,
client-hoplimit : 5, client-hoplimit : 5,
response-delay : 6, response-delay : 6,
query-name-index : 7, query-name-index : 7,
query-size : 8, query-size : 8,
response-size : 9, response-size : 9,
response-processing-data : 10, response-processing-data : 10,
query-question-sections : 11, ; Second & subsequent query-question-sections : 11, ; Second & subsequent
; questions ; Questions
query-answer-sections : 12, query-answer-sections : 12,
query-authority-sections : 13, query-authority-sections : 13,
query-additional-sections : 14, query-additional-sections : 14,
response-answer-sections : 15, response-answer-sections : 15,
response-authority-sections : 16, response-authority-sections : 16,
response-additional-sections : 17, response-additional-sections : 17,
) )
QueryResponseHints = uint .bits QueryResponseHintValues QueryResponseHints = uint .bits QueryResponseHintValues
QueryResponseSignatureHintValues = &( QueryResponseSignatureHintValues = &(
server-address : 0, server-address-index : 0,
server-port : 1, server-port : 1,
qr-transport-flags : 2, qr-transport-flags : 2,
qr-type : 3, qr-type : 3,
qr-sig-flags : 4, qr-sig-flags : 4,
query-opcode : 5, query-opcode : 5,
dns-flags : 6, qr-dns-flags : 6,
query-rcode : 7, query-rcode : 7,
query-class-type : 8, query-classtype-index : 8,
query-qdcount : 9, query-qdcount : 9,
query-ancount : 10, query-ancount : 10,
query-arcount : 11, query-nscount : 11,
query-nscount : 12, query-arcount : 12,
query-edns-version : 13, query-edns-version : 13,
query-udp-size : 14, query-udp-size : 14,
query-opt-rdata : 15, query-opt-rdata-index : 15,
response-rcode : 16, response-rcode : 16,
) )
QueryResponseSignatureHints = QueryResponseSignatureHints =
uint .bits QueryResponseSignatureHintValues uint .bits QueryResponseSignatureHintValues
RRHintValues = &( RRHintValues = &(
ttl : 0, ttl : 0,
rdata-index : 1, rdata-index : 1,
) )
RRHints = uint .bits RRHintValues RRHints = uint .bits RRHintValues
OtherDataHintValues = &( OtherDataHintValues = &(
malformed-messages : 0, malformed-messages : 0,
address-event-counts : 1, address-event-counts : 1,
) )
OtherDataHints = uint .bits OtherDataHintValues OtherDataHints = uint .bits OtherDataHintValues
StorageFlagValues = &( StorageFlagValues = &(
anonymised-data : 0, anonymized-data : 0,
sampled-data : 1, sampled-data : 1,
normalized-names : 2, normalized-names : 2,
) )
StorageFlags = uint .bits StorageFlagValues StorageFlags = uint .bits StorageFlagValues
; Hints for later analysis. ; Metadata about data collection
VLANIdRange = 1..4094 VLANIdRange = 1..4094
CollectionParameters = { CollectionParameters = {
? query-timeout => uint, ? query-timeout => uint, ; Milliseconds
? skew-timeout => uint, ? skew-timeout => uint, ; Microseconds
? snaplen => uint, ? snaplen => uint,
? promisc => bool, ? promisc => bool,
? interfaces => [+ tstr], ? interfaces => [+ tstr],
? server-addresses => [+ IPAddress], ? server-addresses => [+ IPAddress],
? vlan-ids => [+ VLANIdRange], ? vlan-ids => [+ VLANIdRange],
? filter => tstr, ? filter => tstr,
? generator-id => tstr, ? generator-id => tstr,
? host-id => tstr, ? host-id => tstr,
} }
query-timeout = 0 query-timeout = 0
skipping to change at page 62, line 23 skipping to change at page 62, line 12
? malformed-messages => [+ MalformedMessage], ? malformed-messages => [+ MalformedMessage],
} }
block-preamble = 0 block-preamble = 0
block-statistics = 1 block-statistics = 1
block-tables = 2 block-tables = 2
query-responses = 3 query-responses = 3
address-event-counts = 4 address-event-counts = 4
malformed-messages = 5 malformed-messages = 5
; ;
; The (mandatory) preamble to a block. ; The (mandatory) preamble to a Block.
; ;
BlockPreamble = { BlockPreamble = {
? earliest-time => Timestamp, ? earliest-time => Timestamp,
? block-parameters-index => uint .default 0, ? block-parameters-index => uint .default 0,
} }
earliest-time = 0 earliest-time = 0
block-parameters-index = 1 block-parameters-index = 1
; Ticks are subsecond intervals. The number of ticks in a second is ; Ticks are sub-second intervals. The number of ticks in a second is
; file/block metadata. Signed and unsigned tick types are defined. ; file/block metadata. Signed and unsigned tick types are defined.
ticks = int ticks = int
uticks = uint uticks = uint
Timestamp = [ Timestamp = [
timestamp-secs : uint, timestamp-secs : uint, ; POSIX time
timestamp-uticks : uticks, timestamp-ticks : uticks,
] ]
; ;
; Statistics about the block contents. ; Statistics about the Block contents.
; ;
BlockStatistics = { BlockStatistics = {
? processed-messages => uint, ? processed-messages => uint,
? qr-data-items => uint, ? qr-data-items => uint,
? unmatched-queries => uint, ? unmatched-queries => uint,
? unmatched-responses => uint, ? unmatched-responses => uint,
? discarded-opcode => uint, ? discarded-opcode => uint,
? malformed-items => uint, ? malformed-items => uint,
} }
processed-messages = 0 processed-messages = 0
qr-data-items = 1 qr-data-items = 1
unmatched-queries = 2 unmatched-queries = 2
unmatched-responses = 3 unmatched-responses = 3
discarded-opcode = 4 discarded-opcode = 4
malformed-items = 5 malformed-items = 5
; ;
; Tables of common data referenced from records in a block. ; Tables of common data referenced from records in a Block.
; ;
BlockTables = { BlockTables = {
? ip-address => [+ IPAddress], ? ip-address => [+ IPAddress],
? classtype => [+ ClassType], ? classtype => [+ ClassType],
? name-rdata => [+ bstr], ; Holds both Names ? name-rdata => [+ bstr], ; Holds both names
; and RDATA ; and RDATA
? qr-sig => [+ QueryResponseSignature], ? qr-sig => [+ QueryResponseSignature],
? QuestionTables, ? QuestionTables,
? RRTables, ? RRTables,
? malformed-message-data => [+ MalformedMessageData], ? malformed-message-data => [+ MalformedMessageData],
} }
ip-address = 0 ip-address = 0
classtype = 1 classtype = 1
name-rdata = 2 name-rdata = 2
qr-sig = 3 qr-sig = 3
qlist = 4 qlist = 4
qrr = 5 qrr = 5
rrlist = 6 rrlist = 6
rr = 7 rr = 7
malformed-message-data = 8 malformed-message-data = 8
IPv4Address = bstr .size 4 IPv4Address = bstr .size (0..4)
IPv6Address = bstr .size 16 IPv6Address = bstr .size (0..16)
IPAddress = IPv4Address / IPv6Address IPAddress = IPv4Address / IPv6Address
ClassType = { ClassType = {
type => uint, type => uint,
class => uint, class => uint,
} }
type = 0 type = 0
class = 1 class = 1
QueryResponseSignature = { QueryResponseSignature = {
? server-address-index => uint, ? server-address-index => uint,
? server-port => uint, ? server-port => uint,
? qr-transport-flags => QueryResponseTransportFlags, ? qr-transport-flags => QueryResponseTransportFlags,
? qr-type => QueryResponseType, ? qr-type => QueryResponseType,
? qr-sig-flags => QueryResponseFlags, ? qr-sig-flags => QueryResponseFlags,
? query-opcode => uint, ? query-opcode => uint,
? qr-dns-flags => DNSFlags, ? qr-dns-flags => DNSFlags,
? query-rcode => uint, ? query-rcode => uint,
? query-classtype-index => uint, ? query-classtype-index => uint,
? query-qd-count => uint, ? query-qdcount => uint,
? query-an-count => uint, ? query-ancount => uint,
? query-ns-count => uint, ? query-nscount => uint,
? query-ar-count => uint, ? query-arcount => uint,
? edns-version => uint, ? query-edns-version => uint,
? udp-buf-size => uint, ? query-udp-size => uint,
? opt-rdata-index => uint, ? query-opt-rdata-index => uint,
? response-rcode => uint, ? response-rcode => uint,
} }
server-address-index = 0 server-address-index = 0
server-port = 1 server-port = 1
qr-transport-flags = 2 qr-transport-flags = 2
qr-type = 3 qr-type = 3
qr-sig-flags = 4 qr-sig-flags = 4
query-opcode = 5 query-opcode = 5
qr-dns-flags = 6 qr-dns-flags = 6
query-rcode = 7 query-rcode = 7
query-classtype-index = 8 query-classtype-index = 8
query-qd-count = 9 query-qdcount = 9
query-an-count = 10 query-ancount = 10
query-ns-count = 12 query-nscount = 11
query-ar-count = 12 query-arcount = 12
edns-version = 13 query-edns-version = 13
udp-buf-size = 14 query-udp-size = 14
opt-rdata-index = 15 query-opt-rdata-index = 15
response-rcode = 16 response-rcode = 16
; Transport gives the values that may appear in bits 1..4 of ; Transport gives the values that may appear in bits 1..4 of
; TransportFlags. There is currently no way to express this in ; TransportFlags. There is currently no way to express this in
; CDDL, so Transport is unused. To avoid confusion when used ; CDDL, so Transport is unused. To avoid confusion when used
; with CDDL tools, it is commented out. ; with CDDL tools, it is commented out.
; ;
; Transport = &( ; Transport = &(
; udp : 0, ; udp : 0,
; tcp : 1, ; tcp : 1,
; tls : 2, ; tls : 2,
; dtls : 3, ; dtls : 3,
; doh : 4, ; https : 4,
; non-standard : 15,
; ) ; )
TransportFlagValues = &( TransportFlagValues = &(
ip-version : 0, ; 0=IPv4, 1=IPv6 ip-version : 0, ; 0=IPv4, 1=IPv6
) / (1..4) ) / (1..4)
TransportFlags = uint .bits TransportFlagValues TransportFlags = uint .bits TransportFlagValues
QueryResponseTransportFlagValues = &( QueryResponseTransportFlagValues = &(
query-trailingdata : 5, query-trailingdata : 5,
) / TransportFlagValues ) / TransportFlagValues
QueryResponseTransportFlags = QueryResponseTransportFlags =
uint .bits QueryResponseTransportFlagValues uint .bits QueryResponseTransportFlagValues
QueryResponseType = &( QueryResponseType = &(
stub : 0, stub : 0,
client : 1, client : 1,
resolver : 2, resolver : 2,
skipping to change at page 65, line 21 skipping to change at page 65, line 16
stub : 0, stub : 0,
client : 1, client : 1,
resolver : 2, resolver : 2,
auth : 3, auth : 3,
forwarder : 4, forwarder : 4,
tool : 5, tool : 5,
) )
QueryResponseFlagValues = &( QueryResponseFlagValues = &(
has-query : 0, has-query : 0,
has-reponse : 1, has-response : 1,
query-has-opt : 2, query-has-opt : 2,
response-has-opt : 3, response-has-opt : 3,
query-has-no-question : 4, query-has-no-question : 4,
response-has-no-question: 5, response-has-no-question: 5,
) )
QueryResponseFlags = uint .bits QueryResponseFlagValues QueryResponseFlags = uint .bits QueryResponseFlagValues
DNSFlagValues = &( DNSFlagValues = &(
query-cd : 0, query-cd : 0,
query-ad : 1, query-ad : 1,
skipping to change at page 66, line 4 skipping to change at page 66, line 9
response-rd: 12, response-rd: 12,
response-tc: 13, response-tc: 13,
response-aa: 14, response-aa: 14,
) )
DNSFlags = uint .bits DNSFlagValues DNSFlags = uint .bits DNSFlagValues
QuestionTables = ( QuestionTables = (
qlist => [+ QuestionList], qlist => [+ QuestionList],
qrr => [+ Question] qrr => [+ Question]
) )
QuestionList = [+ uint] ; Index of Question QuestionList = [+ uint] ; Index of Question
Question = { ; Second and subsequent questions Question = { ; Second and subsequent Questions
name-index => uint, ; Index to a name in the name-index => uint, ; Index to a name in the
; name-rdata table ; name-rdata table
classtype-index => uint, classtype-index => uint,
} }
name-index = 0 name-index = 0
classtype-index = 1 classtype-index = 1
RRTables = ( RRTables = (
rrlist => [+ RRList], rrlist => [+ RRList],
rr => [+ RR] rr => [+ RR]
skipping to change at page 66, line 42 skipping to change at page 67, line 4
MalformedMessageData = { MalformedMessageData = {
? server-address-index => uint, ? server-address-index => uint,
? server-port => uint, ? server-port => uint,
? mm-transport-flags => TransportFlags, ? mm-transport-flags => TransportFlags,
? mm-payload => bstr, ? mm-payload => bstr,
} }
; Other map key values already defined above. ; Other map key values already defined above.
mm-transport-flags = 2 mm-transport-flags = 2
mm-payload = 3 mm-payload = 3
; ;
; A single query/response pair. ; A single Query/Response data item.
; ;
QueryResponse = { QueryResponse = {
? time-offset => uticks, ; Time offset from ? time-offset => uticks, ; Time offset from
; start of block ; start of Block
? client-address-index => uint, ? client-address-index => uint,
? client-port => uint, ? client-port => uint,
? transaction-id => uint, ? transaction-id => uint,
? qr-signature-index => uint, ? qr-signature-index => uint,
? client-hoplimit => uint, ? client-hoplimit => uint,
? response-delay => ticks, ? response-delay => ticks,
? query-name-index => uint, ? query-name-index => uint,
? query-size => uint, ; DNS size of query ? query-size => uint, ; DNS size of Query
? response-size => uint, ; DNS size of response ? response-size => uint, ; DNS size of Response
? response-processing-data => ResponseProcessingData, ? response-processing-data => ResponseProcessingData,
? query-extended => QueryResponseExtended, ? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended, ? response-extended => QueryResponseExtended,
} }
time-offset = 0 time-offset = 0
client-address-index = 1 client-address-index = 1
client-port = 2 client-port = 2
transaction-id = 3 transaction-id = 3
qr-signature-index = 4 qr-signature-index = 4
client-hoplimit = 5 client-hoplimit = 5
skipping to change at page 68, line 7 skipping to change at page 68, line 20
} }
question-index = 0 question-index = 0
answer-index = 1 answer-index = 1
authority-index = 2 authority-index = 2
additional-index = 3 additional-index = 3
; ;
; Address event data. ; Address event data.
; ;
AddressEventCount = { AddressEventCount = {
ae-type => &AddressEventType, ae-type => &AddressEventType,
? ae-code => uint, ? ae-code => uint,
ae-address-index => uint, ae-address-index => uint,
ae-count => uint, ? ae-transport-flags => TransportFlags,
ae-count => uint,
} }
ae-type = 0 ae-type = 0
ae-code = 1 ae-code = 1
ae-address-index = 2 ae-address-index = 2
ae-count = 3 ae-transport-flags = 3
ae-count = 4
AddressEventType = ( AddressEventType = (
tcp-reset : 0, tcp-reset : 0,
icmp-time-exceeded : 1, icmp-time-exceeded : 1,
icmp-dest-unreachable : 2, icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3, icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4, icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5, icmpv6-packet-too-big : 5,
) )
; ;
; Malformed messages. ; Malformed messages.
; ;
MalformedMessage = { MalformedMessage = {
? time-offset => uticks, ; Time offset from ? time-offset => uticks, ; Time offset from
; start of block ; start of Block
? client-address-index => uint, ? client-address-index => uint,
? client-port => uint, ? client-port => uint,
? message-data-index => uint, ? message-data-index => uint,
} }
; Other map key values already defined above. ; Other map key values already defined above.
message-data-index = 3 message-data-index = 3
Appendix B. DNS Name compression example Appendix B. DNS Name Compression Example
The basic algorithm, which follows the guidance in [RFC1035], is The basic algorithm, which follows the guidance in [RFC1035], is
simply to collect each name, and the offset in the packet at which it simply to collect each name, and the offset in the packet at which it
starts, during packet construction. As each name is added, it is starts, during packet construction. As each name is added, it is
offered to each of the collected names in order of collection, offered to each of the collected names in order of collection,
starting from the first name. If labels at the end of the name can starting from the first name. If (1) labels at the end of the name
be replaced with a reference back to part (or all) of the earlier can be replaced with a reference back to part (or all) of the earlier
name, and if the uncompressed part of the name is shorter than any name and (2) the uncompressed part of the name is shorter than any
compression already found, the earlier name is noted as the compression already found, the earlier name is noted as the
compression target for the name. compression target for the name.
The following tables illustrate the process. In an example packet, The following tables illustrate the step-by-step process of adding
the first name is foo.example. names and performing name compression. In an example packet, the
first name added is foo.example, which cannot be compressed.
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
| 1 | foo.example | | | | 1 | foo.example | foo.example | None |
+---+-------------+--------------+--------------------+ +---+-------------+--------------+--------------------+
The next name added is bar.example. This is matched against The next name added is bar.example. This is matched against
foo.example. The example part of this can be used as a compression foo.example. The example part of this can be used as a compression
target, with the remaining uncompressed part of the name being bar. target, with the remaining uncompressed part of the name being bar.
+---+-------------+--------------+-----------------------+ +---+-------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+-----------------------+ +---+-------------+--------------+-----------------------+
| 1 | foo.example | | | | 1 | foo.example | foo.example | None |
| 2 | bar.example | bar | 1 + offset to example | | 2 | bar.example | bar | 1 + offset to example |
+---+-------------+--------------+-----------------------+ +---+-------------+--------------+-----------------------+
The third name added is www.bar.example. This is first matched The third name added is www.bar.example. This is first matched
against foo.example, and as before this is recorded as a compression against foo.example, and as before this is recorded as a compression
target, with the remaining uncompressed part of the name being target, with the remaining uncompressed part of the name being
www.bar. It is then matched against the second name, which again can www.bar. It is then matched against the second name, which again can
be a compression target. Because the remaining uncompressed part of be a compression target. Because the remaining uncompressed part of
the name is www, this is an improved compression, and so it is the name is www, this is an improved compression, and so it is
adopted. adopted.
+---+-----------------+--------------+-----------------------+ +---+-----------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target | | N | Name | Uncompressed | Compression Target |
+---+-----------------+--------------+-----------------------+ +---+-----------------+--------------+-----------------------+
| 1 | foo.example | | | | 1 | foo.example | foo.example | None |
| 2 | bar.example | bar | 1 + offset to example | | 2 | bar.example | bar | 1 + offset to example |
| 3 | www.bar.example | www | 2 | | 3 | www.bar.example | www | 2 |
+---+-----------------+--------------+-----------------------+ +---+-----------------+--------------+-----------------------+
As an optimization, if a name is already perfectly compressed (in As an optimization, if a name is already perfectly compressed (in
other words, the uncompressed part of the name is empty), then no other words, the uncompressed part of the name is empty), then no
further names will be considered for compression. further names will be considered for compression.
B.1. NSD compression algorithm B.1. NSD Compression Algorithm
Using the above basic algorithm the packet lengths of responses Using the above basic algorithm, the packet lengths of Responses
generated by NSD [7] can be matched almost exactly. At the time of generated by the Name Server Daemon (NSD) [NSD] can be matched almost
writing, a tiny number (<.01%) of the reconstructed packets had exactly. At the time of writing, a tiny number (<.01%) of the
incorrect lengths. reconstructed packets had incorrect lengths.
B.2. Knot Authoritative compression algorithm B.2. Knot Authoritative Compression Algorithm
The Knot Authoritative [8] name server uses different compression The Knot Authoritative name server [Knot] uses different compression
behavior, which is the result of internal optimization designed to behavior, which is the result of internal optimization designed to
balance runtime speed with compression size gains. In brief, and balance runtime speed with compression size gains. In brief, and
omitting complications, Knot Authoritative will only consider the omitting complications, Knot Authoritative will only consider the
QNAME and names in the immediately preceding RR section in an RRSET QNAME and names in the immediately preceding RR section in an RRSET
as compression targets. as compression targets.
A set of smart heuristics as described below can be implemented to A set of smart heuristics as described below can be implemented to
mimic this and while not perfect it produces output nearly, but not mimic this, and while not perfect, it produces output nearly, but not
quite, as good a match as with NSD. The heuristics are: quite, as good a match as with NSD. The heuristics are as follows:
1. A match is only perfect if the name is completely compressed AND 1. A match is only perfect if the name is completely compressed AND
the TYPE of the section in which the name occurs matches the TYPE the TYPE of the section in which the name occurs matches the TYPE
of the name used as the compression target. of the name used as the compression target.
2. If the name occurs in RDATA: 2. If the name occurs in RDATA:
* If the compression target name is in a query, then only the * If the compression target name is in a Query, then only the
first RR in an RRSET can use that name as a compression first RR in an RRSET can use that name as a compression
target. target.
* The compression target name MUST be in RDATA. * The compression target name MUST be in RDATA.
* The name section TYPE must match the compression target name * The name section TYPE must match the compression target name
section TYPE. section TYPE.
* The compression target name MUST be in the immediately * The compression target name MUST be in the immediately
preceding RR in the RRSET. preceding RR in the RRSET.
Using this algorithm less than 0.1% of the reconstructed packets had Using this algorithm, less than 0.1% of the reconstructed packets had
incorrect lengths. incorrect lengths.
B.3. Observed differences B.3. Observed Differences
In sample traffic collected on a root name server around 2-4% of In sample traffic collected on a root name server, around 2-4% of
responses generated by Knot had different packet lengths to those Responses generated by Knot had different packet lengths than those
produced by NSD. produced by NSD.
Appendix C. Comparison of Binary Formats Appendix C. Comparison of Binary Formats
Several binary serialisation formats were considered, and for Several binary serialization formats were considered. For
completeness were also compared to JSON. completeness, they were also compared to JSON.
o Apache Avro [9]. Data is stored according to a pre-defined o Apache Avro [Avro]. Data is stored according to a predefined
schema. The schema itself is always included in the data file. schema. The schema itself is always included in the data file.
Data can therefore be stored untagged, for a smaller serialization
Data can therefore be stored untagged, for a smaller serialisation
size, and be written and read by an Avro library. size, and be written and read by an Avro library.
* At the time of writing, Avro libraries are available for C, * At the time of writing, Avro libraries are available for C,
C++, C#, Java, Python, Ruby and PHP. Optionally tools are C++, C#, Java, Python, Ruby, and PHP. Optionally, tools are
available for C++, Java and C# to generate code for encoding available for C++, Java, and C# to generate code for encoding
and decoding. and decoding.
o Google Protocol Buffers [10]. Data is stored according to a pre- o Google Protocol Buffers [Protocol-Buffers]. Data is stored
defined schema. The schema is used by a generator to generate according to a predefined schema. The schema is used by a
code for encoding and decoding the data. Data can therefore be generator to generate code for encoding and decoding the data.
stored untagged, for a smaller serialisation size. The schema is Data can therefore be stored untagged, for a smaller serialization
not stored with the data, so unlike Avro cannot be read with a size. The schema is not stored with the data, so unlike Avro, it
generic library. cannot be read with a generic library.
* Code must be generated for a particular data schema to read and * Code must be generated for a particular data schema to read and
write data using that schema. At the time of writing, the write data using that schema. At the time of writing, the
Google code generator can currently generate code for encoding Google code generator can currently generate code for encoding
and decoding a schema for C++, Go, Java, Python, Ruby, C#, and decoding a schema for C++, Go, Java, Python, Ruby, C#,
Objective-C, Javascript and PHP. Objective-C, JavaScript, and PHP.
o CBOR [11]. Defined in [RFC7049], this serialisation format is o CBOR [RFC7049]. This serialization format is comparable to JSON
comparable to JSON but with a binary representation. It does not but with a binary representation. It does not use a predefined
use a pre-defined schema, so data is always stored tagged. schema, so data is always stored tagged. However, CBOR data
However, CBOR data schemas can be described using CDDL schemas can be described using CDDL [RFC8610], and tools exist to
[I-D.ietf-cbor-cddl] and tools exist to verify data files conform verify that data files conform to the schema.
to the schema.
* CBOR is a simple format, and simple to implement. At the time * CBOR is a simple format and is simple to implement. At the
of writing, the CBOR website lists implementations for 16 time of writing, the CBOR website lists implementations for 16
languages. languages.
Avro and Protocol Buffers both allow storage of untagged data, but Avro and Protocol Buffers both allow storage of untagged data, but
because they rely on the data schema for this, their implementation because they rely on the data schema for this, their implementation
is considerably more complex than CBOR. Using Avro or Protocol is considerably more complex than CBOR. Using Avro or Protocol
Buffers in an unsupported environment would require notably greater Buffers in an unsupported environment would require notably greater
development effort compared to CBOR. development effort compared to CBOR.
A test program was written which reads input from a PCAP file and A test program was written that reads input from a PCAP file and
writes output using one of two basic structures; either a simple writes output using one of two basic structures: either a simple
structure, where each query/response pair is represented in a single structure, where each Query/Response pair is represented in a single
record entry, or the C-DNS block structure. record entry, or the C-DNS block structure.
The resulting output files were then compressed using a variety of The resulting output files were then compressed using a variety of
common general-purpose lossless compression tools to explore the common general-purpose lossless compression tools to explore the
compressibility of the formats. The compression tools employed were: compressibility of the formats. The compression tools employed were:
o snzip [12]. A command line compression tool based on the Google o snzip [snzip]. A command-line compression tool based on the
Snappy [13] library. Google Snappy library [snappy].
o lz4 [14]. The command line compression tool from the reference C o lz4 [lz4]. The command-line compression tool from the reference C
LZ4 implementation. LZ4 implementation.
o gzip [15]. The ubiquitous GNU zip tool. o gzip [gzip]. The ubiquitous GNU zip tool.
o zstd [16]. Compression using the Zstandard algorithm. o zstd [zstd]. Compression using the Zstandard algorithm.
o xz [17]. A popular compression tool noted for high compression. o xz [xz]. A popular compression tool noted for high compression.
In all cases the compression tools were run using their default In all cases, the compression tools were run using their default
settings. settings.
Note that this draft does not mandate the use of compression, nor any Note that this document does not mandate the use of compression, nor
particular compression scheme, but it anticipates that in practice any particular compression scheme, but it anticipates that in
output data will be subject to general-purpose compression, and so practice output data will be subject to general-purpose compression,
this should be taken into consideration. and so this should be taken into consideration.
"test.pcap", a 662Mb capture of sample data from a root instance was "test.pcap", a 662 MB capture of sample data from a root instance,
used for the comparison. The following table shows the formatted was used for the comparison. The following table shows the formatted
size and size after compression (abbreviated to Comp. in the table size and size after compression (abbreviated to Comp. in the table
headers), together with the task resident set size (RSS) and the user headers), together with the task Resident Set Size (RSS) and the user
time taken by the compression. File sizes are in Mb, RSS in kb and time taken by the compression. File sizes are in MB, RSS is in kB,
user time in seconds. and user time is in seconds.
+-------------+-----------+-------+------------+-------+-----------+ +-------------+-----------+-------+------------+-------+-----------+
| Format | File size | Comp. | Comp. size | RSS | User time | | Format | File Size | Comp. | Comp. Size | RSS | User Time |
+-------------+-----------+-------+------------+-------+-----------+ +-------------+-----------+-------+------------+-------+-----------+
| PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 | | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 |
| | | lz4 | 181.58 | 6336 | 1.35 | | | | lz4 | 181.58 | 6336 | 1.35 |
| | | gzip | 153.46 | 1428 | 18.20 | | | | gzip | 153.46 | 1428 | 18.20 |
| | | zstd | 87.07 | 3544 | 4.27 | | | | zstd | 87.07 | 3544 | 4.27 |
| | | xz | 49.09 | 97416 | 160.79 | | | | xz | 49.09 | 97416 | 160.79 |
| | | | | | | | | | | | | |
| JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 | | JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 |
| | | lz4 | 386.42 | 5636 | 5.25 | | | | lz4 | 386.42 | 5636 | 5.25 |
| | | gzip | 271.11 | 1492 | 73.00 | | | | gzip | 271.11 | 1492 | 73.00 |
skipping to change at page 73, line 43 skipping to change at page 74, line 19
| | | | | | | | | | | | | |
| PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 | | PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 |
| | | lz4 | 52.39 | 8304 | 0.24 | | | | lz4 | 52.39 | 8304 | 0.24 |
| | | gzip | 40.19 | 1520 | 3.63 | | | | gzip | 40.19 | 1520 | 3.63 |
| | | zstd | 31.61 | 3576 | 1.40 | | | | zstd | 31.61 | 3576 | 1.40 |
| | | xz | 17.94 | 97440 | 33.99 | | | | xz | 17.94 | 97440 | 33.99 |
+-------------+-----------+-------+------------+-------+-----------+ +-------------+-----------+-------+------------+-------+-----------+
The above results are discussed in the following sections. The above results are discussed in the following sections.
C.1. Comparison with full PCAP files C.1. Comparison with Full PCAP Files
An important first consideration is whether moving away from PCAP An important first consideration is whether moving away from PCAP
offers significant benefits. offers significant benefits.
The simple binary formats are typically larger than PCAP, even though The simple binary formats are typically larger than PCAP, even though
they omit some information such as Ethernet MAC addresses. But not they omit some information such as Ethernet Media Access Control
only do they require less CPU to compress than PCAP, the resulting (MAC) addresses. But not only do they require less CPU to compress
compressed files are smaller than compressed PCAP. than PCAP, the resulting compressed files are smaller than compressed
PCAP.
C.2. Simple versus block coding C.2. Simple versus Block Coding
The intention of the block coding is to perform data de-duplication The intention of the block coding is to perform data deduplication on
on query/response records within the block. The simple and block Query/Response records within the block. The simple and block
formats above store exactly the same information for each query/ formats shown above store exactly the same information for each
response record. This information is parsed from the DNS traffic in Query/Response record. This information is parsed from the DNS
the input PCAP file, and in all cases each field has an identifier traffic in the input PCAP file, and in all cases each field has an
and the field data is typed. identifier and the field data is typed.
The data de-duplication on the block formats show an order of The data deduplication on the block formats show an order-of-
magnitude reduction in the size of the format file size against the magnitude reduction in the size of the format file size against the
simple formats. As would be expected, the compression tools are able simple formats. As would be expected, the compression tools are able
to find and exploit a lot of this duplication, but as the de- to find and exploit a lot of this duplication, but as the
duplication process uses knowledge of DNS traffic, it is able to deduplication process uses knowledge of DNS traffic, it is able to
retain a size advantage. This advantage reduces as stronger retain a size advantage. This advantage reduces as stronger
compression is applied, as again would be expected, but even with the compression is applied, as again would be expected, but even with the
strongest compression applied the block formatted data remains around strongest compression applied the block-formatted data remains around
75% of the size of the simple format and its compression requires 75% of the size of the simple format and its compression requires
roughly a third of the CPU time. roughly a third of the CPU time.
C.3. Binary versus text formats C.3. Binary versus Text Formats
Text data formats offer many advantages over binary formats, Text data formats offer many advantages over binary formats,
particularly in the areas of ad-hoc data inspection and extraction. particularly in the areas of ad hoc data inspection and extraction.
It was therefore felt worthwhile to carry out a direct comparison, It was therefore felt worthwhile to carry out a direct comparison,
implementing JSON versions of the simple and block formats. implementing JSON versions of the simple and block formats.
Concentrating on JSON block format, the format files produced are a Concentrating on JSON block format, the format files produced are a
significant fraction of an order of magnitude larger than binary significant fraction of an order of magnitude larger than binary
formats. The impact on file size after compression is as might be formats. The impact on file size after compression is as might be
expected from that starting point; the stronger compression produces expected from that starting point; the stronger compression produces
files that are 150% of the size of similarly compressed binary files that are 150% of the size of similarly compressed binary format
format, and require over 4x more CPU to compress. and require over 4x more CPU to compress.
C.4. Performance C.4. Performance
Concentrating again on the block formats, all three produce format Concentrating again on the block formats, all three produce format
files that are close to an order of magnitude smaller that the files that are close to an order of magnitude smaller than the
original "test.pcap" file. CBOR produces the largest files and Avro original "test.pcap" file. CBOR produces the largest files and Avro
the smallest, 20% smaller than CBOR. the smallest, 20% smaller than CBOR.
However, once compression is taken into account, the size difference However, once compression is taken into account, the size difference
narrows. At medium compression (with gzip), the size difference is narrows. At medium compression (with gzip), the size difference is
4%. Using strong compression (with xz) the difference reduces to 2%, 4%. Using strong compression (with xz), the difference reduces to
with Avro the largest and Protocol Buffers the smallest, although 2%, with Avro the largest and Protocol Buffers the smallest, although
CBOR and Protocol Buffers require slightly more compression CPU. CBOR and Protocol Buffers require slightly more compression CPU.
The measurements presented above do not include data on the CPU The measurements presented above do not include data on the CPU
required to generate the format files. Measurements indicate that required to generate the format files. Measurements indicate that
writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It
appears, therefore, that Avro's advantage in compression CPU usage is appears, therefore, that Avro's advantage in compression CPU usage is
probably offset by a larger CPU requirement in writing Avro. probably offset by a larger CPU requirement in writing Avro.
C.5. Conclusions C.5. Conclusions
The above assessments lead us to the choice of a binary format file The above assessments lead us to the choice of a binary format file
using blocking. using blocking.
As noted previously, this draft anticipates that output data will be As noted previously, this document anticipates that output data will
subject to compression. There is no compelling case for one be subject to compression. There is no compelling case for one
particular binary serialisation format in terms of either final file particular binary serialization format in terms of either final file
size or machine resources consumed, so the choice must be largely size or machine resources consumed, so the choice must be largely
based on other factors. CBOR was therefore chosen as the binary based on other factors. CBOR was therefore chosen as the binary
serialisation format for the reasons listed in Section 5. serialization format for the reasons listed in Section 5.
C.6. Block size choice C.6. Block Size Choice
Given the choice of a CBOR format using blocking, the question arises Given the choice of a CBOR format using blocking, the question arises
of what an appropriate default value for the maximum number of query/ of what an appropriate default value for the maximum number of
response pairs in a block should be. This has two components; what Query/Response pairs in a block should be. This has two components:
is the impact on performance of using different block sizes in the
format file, and what is the impact on the size of the format file 1. What is the impact on performance of using different block sizes
before and after compression. in the format file?
2. What is the impact on the size of the format file before and
after compression?
The following table addresses the performance question, showing the The following table addresses the performance question, showing the
impact on the performance of a C++ program converting "test.pcap" to impact on the performance of a C++ program converting "test.pcap"
C-DNS. File size is in Mb, resident set size (RSS) in kb. to C-DNS. File sizes are in MB, RSS is in kB, and user time is
in seconds.
+------------+-----------+--------+-----------+ +------------+-----------+--------+-----------+
| Block size | File size | RSS | User time | | Block Size | File Size | RSS | User Time |
+------------+-----------+--------+-----------+ +------------+-----------+--------+-----------+
| 1000 | 133.46 | 612.27 | 15.25 | | 1,000 | 133.46 | 612.27 | 15.25 |
| 5000 | 89.85 | 676.82 | 14.99 | | 5,000 | 89.85 | 676.82 | 14.99 |
| 10000 | 76.87 | 752.40 | 14.53 | | 10,000 | 76.87 | 752.40 | 14.53 |
| 20000 | 67.86 | 750.75 | 14.49 | | 20,000 | 67.86 | 750.75 | 14.49 |
| 40000 | 61.88 | 736.30 | 14.29 | | 40,000 | 61.88 | 736.30 | 14.29 |
| 80000 | 58.08 | 694.16 | 14.28 | | 80,000 | 58.08 | 694.16 | 14.28 |
| 160000 | 55.94 | 733.84 | 14.44 | | 160,000 | 55.94 | 733.84 | 14.44 |
| 320000 | 54.41 | 799.20 | 13.97 | | 320,000 | 54.41 | 799.20 | 13.97 |
+------------+-----------+--------+-----------+ +------------+-----------+--------+-----------+
Increasing block size, therefore, tends to increase maximum RSS a Therefore, increasing block size tends to increase maximum RSS a
little, with no significant effect (if anything a small reduction) on little, with no significant effect (if anything, a small reduction)
CPU consumption. on CPU consumption.
The following table demonstrates the effect of increasing block size The following table demonstrates the effect of increasing block size
on output file size for different compressions. on output file size for different compressions.
+------------+--------+-------+-------+-------+-------+-------+ +------------+--------+-------+-------+-------+-------+-------+
| Block size | None | snzip | lz4 | gzip | zstd | xz | | Block Size | None | snzip | lz4 | gzip | zstd | xz |
+------------+--------+-------+-------+-------+-------+-------+ +------------+--------+-------+-------+-------+-------+-------+
| 1000 | 133.46 | 90.52 | 90.03 | 74.65 | 44.78 | 25.63 | | 1,000 | 133.46 | 90.52 | 90.03 | 74.65 | 44.78 | 25.63 |
| 5000 | 89.85 | 59.69 | 59.43 | 46.99 | 37.33 | 22.34 | | 5,000 | 89.85 | 59.69 | 59.43 | 46.99 | 37.33 | 22.34 |
| 10000 | 76.87 | 50.39 | 50.28 | 38.94 | 33.62 | 21.09 | | 10,000 | 76.87 | 50.39 | 50.28 | 38.94 | 33.62 | 21.09 |
| 20000 | 67.86 | 43.91 | 43.90 | 33.24 | 32.62 | 20.16 | | 20,000 | 67.86 | 43.91 | 43.90 | 33.24 | 32.62 | 20.16 |
| 40000 | 61.88 | 39.63 | 39.69 | 29.44 | 28.72 | 19.52 | | 40,000 | 61.88 | 39.63 | 39.69 | 29.44 | 28.72 | 19.52 |
| 80000 | 58.08 | 36.93 | 37.01 | 27.05 | 26.25 | 19.00 | | 80,000 | 58.08 | 36.93 | 37.01 | 27.05 | 26.25 | 19.00 |
| 160000 | 55.94 | 35.10 | 35.06 | 25.44 | 24.56 | 19.63 | | 160,000 | 55.94 | 35.10 | 35.06 | 25.44 | 24.56 | 19.63 |
| 320000 | 54.41 | 33.87 | 33.74 | 24.36 | 23.44 | 18.66 | | 320,000 | 54.41 | 33.87 | 33.74 | 24.36 | 23.44 | 18.66 |
+------------+--------+-------+-------+-------+-------+-------+ +------------+--------+-------+-------+-------+-------+-------+
There is obviously scope for tuning the default block size to the There is obviously scope for tuning the default block size to the
compression being employed, traffic characteristics, frequency of compression being employed, traffic characteristics, frequency of
output file rollover etc. Using a strong compression scheme, block output file rollover, etc. Using a strong compression scheme, block
sizes over 10,000 query/response pairs would seem to offer limited sizes over 10,000 Query/Response pairs would seem to offer limited
improvements. improvements.
Appendix D. Data Fields for Traffic Regeneration
D.1. Recommended Fields for Traffic Regeneration
This section specifies the data fields that would need to be captured
in order to perform the fullest PCAP traffic reconstruction for
well-formed DNS messages that is possible with C-DNS.
o All data fields in the QueryResponse type except response-
processing-data.
o All data fields in the QueryResponseSignature type except qr-type.
o All data fields in the RR TYPE.
D.2. Issues with Small Data Captures
At the other extreme, an interesting corner case arises when opting
to perform captures with a smaller data set than that recommended
above. The following list specifies a subset of the above data
fields; if only these data fields are captured, then even a minimal
traffic reconstruction is problematic because there is not enough
information to determine if the Query/Response data item contained
just a Query, just a Response, or a Query/Response pair.
o The following data fields from the QueryResponse type:
* time-offset
* client-address-index
* client-port
* transaction-id
* query-name-index
o The following data fields from the QueryResponseSignature type:
* server-address-index
* server-port
* qr-transport-flags
* query-classtype-index
In this case, simply also capturing the qr-sig-flags will provide
enough information to perform a minimal traffic reconstruction
(assuming that suitable defaults for the remaining fields are
provided). Additionally, capturing response-delay, query-opcode, and
response-rcode will avoid having to rely on potentially misleading
defaults for these values and should result in a PCAP that represents
the basics of the real traffic flow.
Acknowledgements
The authors wish to thank CZ.NIC -- in particular, Tomas Gavenciak --
for many useful discussions on binary formats, compression, and
packet matching. Thanks also to Jan Vcelak and Wouter Wijngaards for
discussions on name compression, and Paul Hoffman for a detailed
review of this document and the C-DNS CDDL.
Thanks also to Robert Edmonds, Jerry Lundstrom, Richard Gibson,
Stephane Bortzmeyer, and many other members of DNSOP for review.
Also, thanks to Miek Gieben for [mmark].
Authors' Addresses Authors' Addresses
John Dickinson John Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom United Kingdom
Email: jad@sinodun.com Email: jad@sinodun.com
Jim Hague Jim Hague
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom United Kingdom
Email: jim@sinodun.com Email: jim@sinodun.com
Sara Dickinson Sara Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom United Kingdom
Email: sara@sinodun.com Email: sara@sinodun.com
Terry Manderson Terry Manderson
ICANN ICANN
12025 Waterfront Drive 12025 Waterfront Drive
Suite 300 Suite 300
Los Angeles CA 90094-2536 Los Angeles, CA 90094-2536
United States of America
Email: terry.manderson@icann.org Email: terry.manderson@icann.org
John Bond John Bond
ICANN Wikimedia Foundation, Inc.
12025 Waterfront Drive 1 Montgomery Street
Suite 300 Suite 1600
Los Angeles CA 90094-2536 San Francisco, CA 94104
United States of America
Email: john.bond@icann.org Email: ietf-wikimedia@johnbond.org
 End of changes. 520 change blocks. 
1637 lines changed or deleted 1636 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/