draft-ietf-dnsop-dns-capture-format-04.txt   draft-ietf-dnsop-dns-capture-format-05.txt 
dnsop J. Dickinson dnsop J. Dickinson
Internet-Draft J. Hague Internet-Draft J. Hague
Intended status: Standards Track S. Dickinson Intended status: Standards Track S. Dickinson
Expires: July 7, 2018 Sinodun IT Expires: August 26, 2018 Sinodun IT
T. Manderson T. Manderson
J. Bond J. Bond
ICANN ICANN
January 3, 2018 February 22, 2018
C-DNS: A DNS Packet Capture Format C-DNS: A DNS Packet Capture Format
draft-ietf-dnsop-dns-capture-format-04 draft-ietf-dnsop-dns-capture-format-05
Abstract Abstract
This document describes a data representation for collections of DNS This document describes a data representation for collections of DNS
messages. The format is designed for efficient storage and messages. The format is designed for efficient storage and
transmission of large packet captures of DNS traffic; it attempts to transmission of large packet captures of DNS traffic; it attempts to
minimize the size of such packet capture files but retain the full minimize the size of such packet capture files but retain the full
DNS message contents along with the most useful transport metadata. DNS message contents along with the most useful transport metadata.
It is intended to assist with the development of DNS traffic It is intended to assist with the development of DNS traffic
monitoring applications. monitoring applications.
skipping to change at page 1, line 40 skipping to change at page 1, line 40
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 7, 2018. This Internet-Draft will expire on August 26, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 5 3. Data collection use cases . . . . . . . . . . . . . . . . . . 5
4. Design Considerations . . . . . . . . . . . . . . . . . . . . 7 4. Design considerations . . . . . . . . . . . . . . . . . . . . 7
5. Conceptual Overview . . . . . . . . . . . . . . . . . . . . . 8 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8
6. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8 6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9
7. The C-DNS format . . . . . . . . . . . . . . . . . . . . . . 9 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 10
7.1. CDDL definition . . . . . . . . . . . . . . . . . . . . . 9 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 10
7.2. Format overview . . . . . . . . . . . . . . . . . . . . . 9 6.2.1. Optional data items . . . . . . . . . . . . . . . . . 10
7.3. File header contents . . . . . . . . . . . . . . . . . . 10 6.2.2. Optional RRs and OPCODES . . . . . . . . . . . . . . 11
7.4. File preamble contents . . . . . . . . . . . . . . . . . 10 6.2.3. Sampling and anonymisation . . . . . . . . . . . . . 12
7.5. Configuration contents . . . . . . . . . . . . . . . . . 11 6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 12
7.6. Block contents . . . . . . . . . . . . . . . . . . . . . 13 7. C-DNS format detailed description . . . . . . . . . . . . . . 12
7.7. Block preamble map . . . . . . . . . . . . . . . . . . . 13 7.1. Map quantities and indexes . . . . . . . . . . . . . . . 12
7.8. Block statistics . . . . . . . . . . . . . . . . . . . . 14 7.2. Tabular representation . . . . . . . . . . . . . . . . . 12
7.9. Block table map . . . . . . . . . . . . . . . . . . . . . 14 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 13
7.10. IP address table . . . . . . . . . . . . . . . . . . . . 15 7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 14
7.11. Class/Type table . . . . . . . . . . . . . . . . . . . . 15 7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 14
7.12. Name/RDATA table . . . . . . . . . . . . . . . . . . . . 16 7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 18
7.13. Query Signature table . . . . . . . . . . . . . . . . . . 16 7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.14. Question table . . . . . . . . . . . . . . . . . . . . . 19 7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 20
7.15. Resource Record (RR) table . . . . . . . . . . . . . . . 19 7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 21
7.16. Question list table . . . . . . . . . . . . . . . . . . . 19 7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 22
7.17. Resource Record list table . . . . . . . . . . . . . . . 20 7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 27
7.18. Query/Response data . . . . . . . . . . . . . . . . . . . 20 7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 29
7.19. Address Event counts . . . . . . . . . . . . . . . . . . 23 7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 29
7.20. Malformed packet records . . . . . . . . . . . . . . . . 23 7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 30
8. Malformed Packets . . . . . . . . . . . . . . . . . . . . . . 24 7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 31
9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 25 8. Malformed messages . . . . . . . . . . . . . . . . . . . . . 32
9.1. Name Compression . . . . . . . . . . . . . . . . . . . . 26 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 33
10. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 26 9.1. Name compression . . . . . . . . . . . . . . . . . . . . 34
10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 27 10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 35
10.2. Message identifiers . . . . . . . . . . . . . . . . . . 27 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 35
10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 27 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 36
10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 28 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 36
10.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 28 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 36
10.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 28 10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 36
10.5. Algorithm Limitations . . . . . . . . . . . . . . . . . 28 10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 36
10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 28 10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 37
10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 29 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 37
10.8. Post Processing . . . . . . . . . . . . . . . . . . . . 29 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 37
11. Implementation Status . . . . . . . . . . . . . . . . . . . . 29 10.8. Post processing . . . . . . . . . . . . . . . . . . . . 37
11.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 30 11. Implementation guidance . . . . . . . . . . . . . . . . . . . 37
12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 38
13. Security Considerations . . . . . . . . . . . . . . . . . . . 30 11.2. Trailing data in TCP . . . . . . . . . . . . . . . . . . 38
14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 38
15. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 31 12. Implementation status . . . . . . . . . . . . . . . . . . . . 38
16. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 39
16.1. Normative References . . . . . . . . . . . . . . . . . . 32 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 39
16.2. Informative References . . . . . . . . . . . . . . . . . 32 14. Security considerations . . . . . . . . . . . . . . . . . . . 39
16.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 34 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 39
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 35 16. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Appendix B. DNS Name compression example . . . . . . . . . . . . 41 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 42 17.1. Normative References . . . . . . . . . . . . . . . . . . 42
B.2. Knot Authoritative compression algorithm . . . . . . . . 43 17.2. Informative References . . . . . . . . . . . . . . . . . 42
B.3. Observed differences . . . . . . . . . . . . . . . . . . 43 17.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 43 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 44
C.1. Comparison with full PCAP files . . . . . . . . . . . . . 46 Appendix B. DNS Name compression example . . . . . . . . . . . . 54
C.2. Simple versus block coding . . . . . . . . . . . . . . . 47 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 55
C.3. Binary versus text formats . . . . . . . . . . . . . . . 47 B.2. Knot Authoritative compression algorithm . . . . . . . . 55
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 47 B.3. Observed differences . . . . . . . . . . . . . . . . . . 56
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 48 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 56
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 48 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 59
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 49 C.2. Simple versus block coding . . . . . . . . . . . . . . . 59
C.3. Binary versus text formats . . . . . . . . . . . . . . . 59
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 60
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 60
C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 60
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 61
1. Introduction 1. Introduction
There has long been a need to collect DNS queries and responses on There has long been a need to collect DNS queries and responses on
authoritative and recursive name servers for monitoring and analysis. authoritative and recursive name servers for monitoring and analysis.
This data is used in a number of ways including traffic monitoring, This data is used in a number of ways including traffic monitoring,
analyzing network attacks and "day in the life" (DITL) [ditl] analyzing network attacks and "day in the life" (DITL) [ditl]
analysis. analysis.
A wide variety of tools already exist that facilitate the collection A wide variety of tools already exist that facilitate the collection
skipping to change at page 4, line 28 skipping to change at page 4, line 33
This document contains: This document contains:
o A discussion of the some common use cases in which such DNS data o A discussion of the some common use cases in which such DNS data
is collected Section 3 is collected Section 3
o A discussion of the major design considerations in developing an o A discussion of the major design considerations in developing an
efficient data representation for collections of DNS messages efficient data representation for collections of DNS messages
Section 4 Section 4
o A conceptual overview of the C-DNS format Section 5
o A description of why CBOR [RFC7049] was chosen for this format o A description of why CBOR [RFC7049] was chosen for this format
Section 6 Section 5
o A conceptual overview of the C-DNS format Section 6
o The definition of the C-DNS format for the collection of DNS o The definition of the C-DNS format for the collection of DNS
messages Section 7. messages Section 7.
o Notes on converting C-DNS data to PCAP format Section 9 o Notes on converting C-DNS data to PCAP format Section 9
o Some high level implementation considerations for applications o Some high level implementation considerations for applications
designed to produce C-DNS Section 10 designed to produce C-DNS Section 10
2. Terminology 2. Terminology
skipping to change at page 5, line 11 skipping to change at page 5, line 16
are UDP, but may be constructed from a TCP packet. "Message", unless are UDP, but may be constructed from a TCP packet. "Message", unless
otherwise qualified, refers to a DNS payload extracted from a UDP or otherwise qualified, refers to a DNS payload extracted from a UDP or
TCP data stream. TCP data stream.
The parts of DNS messages are named as they are in [RFC1035]. In The parts of DNS messages are named as they are in [RFC1035]. In
specific, the DNS message has five sections: Header, Question, specific, the DNS message has five sections: Header, Question,
Answer, Authority, and Additional. Answer, Authority, and Additional.
Pairs of DNS messages are called a Query and a Response. Pairs of DNS messages are called a Query and a Response.
3. Data Collection Use Cases 3. Data collection use cases
In an ideal world, it would be optimal to collect full packet In an ideal world, it would be optimal to collect full packet
captures of all packets going in or out of a name server. However, captures of all packets going in or out of a name server. However,
there are several design choices or other limitations that are common there are several design choices or other limitations that are common
to many DNS installations and operators. to many DNS installations and operators.
o DNS servers are hosted in a variety of situations o DNS servers are hosted in a variety of situations
* Self-hosted servers * Self-hosted servers
skipping to change at page 7, line 5 skipping to change at page 7, line 7
leads to designing a format that requires a relatively low overhead leads to designing a format that requires a relatively low overhead
to produce and minimizes the requirement for further potentially to produce and minimizes the requirement for further potentially
costly compression. costly compression.
However, it was also essential that interoperability with less However, it was also essential that interoperability with less
restricted infrastructure was maintained. In particular, it is restricted infrastructure was maintained. In particular, it is
highly desirable that the collection format should facilitate the re- highly desirable that the collection format should facilitate the re-
creation of common formats (such as PCAP) that are as close to the creation of common formats (such as PCAP) that are as close to the
original as is realistic given the restrictions above. original as is realistic given the restrictions above.
4. Design Considerations 4. Design considerations
This section presents some of the major design considerations used in This section presents some of the major design considerations used in
the development of the C-DNS format. the development of the C-DNS format.
1. The basic unit of data is a combined DNS Query and the associated 1. The basic unit of data is a combined DNS Query and the associated
Response (a "Q/R data item"). The same structure will be used Response (a "Q/R data item"). The same structure will be used
for unmatched Queries and Responses. Queries without Responses for unmatched Queries and Responses. Queries without Responses
will be captured omitting the response data. Responses without will be captured omitting the response data. Responses without
queries will be captured omitting the Query data (but using the queries will be captured omitting the Query data (but using the
Question section from the response, if present, as an identifying Question section from the response, if present, as an identifying
QNAME). QNAME).
* Rationale: A Query and Response represents the basic level of * Rationale: A Query and Response represents the basic level of
a clients interaction with the server. Also, combining the a clients interaction with the server. Also, combining the
Query and Response into one item often reduces storage Query and Response into one item often reduces storage
requirements due to commonality in the data of the two requirements due to commonality in the data of the two
messages. messages.
2. Each Q/R data item will comprise a default Q/R data description 2. All top level fields in each Q/R data item will be optional.
and a set of optional sections. Inclusion of optional sections
shall be configurable.
* Rationale: Different users will have different requirements * Rationale: Different users will have different requirements
for data to be available for analysis. Users with minimal for data to be available for analysis. Users with minimal
requirements should not have to pay the cost of recording full requirements should not have to pay the cost of recording full
data, however this will limit the ability to reconstruct data, however this will limit the ability to perform certain
packet captures. For example, omitting the resource records kinds of data analysis and also reconstruct packet captures.
from a Response will reduce the files size, and in principle For example, omitting the resource records from a Response
responses can be synthesized if there is enough context. will reduce the C-DNS file size, and in principle responses
can be synthesized if there is enough context.
3. Multiple Q/R data items will be collected into blocks in the 3. Multiple Q/R data items will be collected into blocks in the
format. Common data in a block will be abstracted and referenced format. Common data in a block will be abstracted and referenced
from individual Q/R data items by indexing. The maximum number from individual Q/R data items by indexing. The maximum number
of Q/R data items in a block will be configurable. of Q/R data items in a block will be configurable.
* Rationale: This blocking and indexing provides a significant * Rationale: This blocking and indexing provides a significant
reduction in the volume of file data generated. Although this reduction in the volume of file data generated. Although this
introduces complexity, it provides compression of the data introduces complexity, it provides compression of the data
that makes use of knowledge of the DNS message structure. that makes use of knowledge of the DNS message structure.
skipping to change at page 8, line 9 skipping to change at page 8, line 9
* It is anticipated that the files produced can be subject to * It is anticipated that the files produced can be subject to
further compression using general purpose compression tools. further compression using general purpose compression tools.
Measurements show that blocking significantly reduces the CPU Measurements show that blocking significantly reduces the CPU
required to perform such strong compression. See required to perform such strong compression. See
Appendix C.2. Appendix C.2.
* [TODO: Further discussion of commonality between DNS messages * [TODO: Further discussion of commonality between DNS messages
e.g. common query signatures, a finite set of valid responses e.g. common query signatures, a finite set of valid responses
from authoritatives] from authoritatives]
4. Metadata about other packets received can optionally be included 4. Traffic metadata can optionally be included in each block.
in each block. For example, counts of malformed DNS packets and Specifically, counts of some types of non-DNS packets (e.g.
non-DNS packets (e.g. ICMP, TCP resets) sent to the server may ICMP, TCP resets) sent to the server may be of interest.
be of interest.
5. The wire format content of malformed DNS packets can optionally 5. The wire format content of malformed DNS messages can optionally
be recorded. be recorded.
* Rationale: Any structured capture format that does not capture * Rationale: Any structured capture format that does not capture
the DNS payload byte for byte will be limited to some extent the DNS payload byte for byte will be limited to some extent
in that it cannot represent "malformed" DNS packets (see in that it cannot represent "malformed" DNS messages (see
Section 8). Only those packets that can be transformed Section 8). Only those messages that can be fully parsed and
reasonably into the structured format can be represented by transformed into the structured format can be fully
the format. However this can result in rather misleading represented. Therefore it can greatly aid downstream analysis
statistics. For example, a malformed query which cannot be to have the wire format of the malformed DNS messages
represented in the C-DNS format will lead to the (well formed) available directly in the C-DNS file. Note, however, this can
DNS responses with error code FORMERR appearing as result in rather misleading statistics. For example, a
'unmatched'. Therefore it can greatly aid downstream analysis malformed query which cannot be represented in the C-DNS
to have the wire format of the malformed DNS packets available format will lead to the (well formed) DNS responses with error
directly in the C-DNS file. code FORMERR appearing as 'unmatched'.
5. Conceptual Overview
The following figures show purely schematic representations of the
C-DNS format to convey the high-level structure of the C-DNS format.
Section 7 provides a detailed discussion of the CBOR representation
and individual elements.
Figure showing the C-DNS format (PNG) [1]
Figure showing the C-DNS format (SVG) [2]
Figure showing the Q/R data item and Block tables format (PNG) [3]
Figure showing the Q/R data item and Block tables format (SVG) [4]
6. Choice of CBOR 5. Choice of CBOR
This document presents a detailed format description using CBOR, the This document presents a detailed format description using CBOR, the
Concise Binary Object Representation defined in [RFC7049]. Concise Binary Object Representation defined in [RFC7049].
The choice of CBOR was made taking a number of factors into account. The choice of CBOR was made taking a number of factors into account.
o CBOR is a binary representation, and thus is economical in storage o CBOR is a binary representation, and thus is economical in storage
space. space.
o Other binary representations were investigated, and whilst all had o Other binary representations were investigated, and whilst all had
skipping to change at page 9, line 25 skipping to change at page 9, line 9
industry. industry.
o CBOR is a simple format, and can easily be implemented from o CBOR is a simple format, and can easily be implemented from
scratch if necessary. More complex formats require library scratch if necessary. More complex formats require library
support which may present problems on unusual platforms. support which may present problems on unusual platforms.
o CBOR can also be easily converted to text formats such as JSON o CBOR can also be easily converted to text formats such as JSON
([RFC7159]) for debugging and other human inspection requirements. ([RFC7159]) for debugging and other human inspection requirements.
o CBOR data schemas can be described using CDDL o CBOR data schemas can be described using CDDL
[I-D.greevenbosch-appsawg-cbor-cddl]. [I-D.ietf-cbor-cddl].
7. The C-DNS format 6. C-DNS format conceptual overview
7.1. CDDL definition The following figures show purely schematic representations of the
C-DNS format to convey the high-level structure of the C-DNS format.
Section 7 provides a detailed discussion of the CBOR representation
and individual elements.
The CDDL definition for the C-DNS format is given in Appendix A. Figure showing the C-DNS format (PNG) [1]
7.2. Format overview Figure showing the C-DNS format (SVG) [2]
A C-DNS file begins with a file header containing a file type Figure showing the Query/Response data item and Block Tables format
identifier and a preamble. The preamble contains information on the (PNG) [3]
collection settings.
The file header is followed by a series of data blocks. Figure showing the Query/Response item and Block Tables format (SVG)
[4]
A block consists of a block header, containing various tables of A C-DNS file begins with a file header containing a File Type
common data, and some statistics for the traffic received over the Identifier and a File Preamble. The File Preamble contains
block. The block header is then followed by a list of the Q/R data information on the file Format Version and an array of Block
items detailing the queries and responses received during processing Parameters items (the contents of which include Collection and
of the block input. The list of Q/R data items is in turn followed Storage Parameters used for one or more blocks).
by a list of per-client counts of particular IP events that occurred
during collection of the block data. The file header is followed by a series of data Blocks.
A Block consists of a Block Preamble item, some Block Statistics for
the traffic stored within the Block and then various arrays of common
data collectively called the Block Tables. This is then followed by
an array of the Query/Response data items detailing the queries and
responses stored within the Block. The array of Query/Response data
items is in turn followed by the Address/Event Counts data items (an
array of per-client counts of particular IP events) and then
Malformed Message data items (an array of malformed messages that
stored in the Block).
The exact nature of the DNS data will affect what block size is the The exact nature of the DNS data will affect what block size is the
best fit, however sample data for a root server indicated that block best fit, however sample data for a root server indicated that block
sizes up to 10,000 Q/R data items give good results. See sizes up to 10,000 Q/R data items give good results. See
Appendix C.6 for more details. Appendix C.6 for more details.
If no field type is specified, then the field is unsigned. 6.1. Block Parameters
In all quantities that contain bit flags, bit 0 indicates the least The details of the Block Parameters items are not shown in the
significant bit. An item described as an index is the index of the diagrams but are discussed here for context.
Q/R data item in the referenced table. Indexes are 1-based. An
index value of 0 is reserved to mean "not present".
All map keys are unsigned integers with values specified in the CDDL An array of Block Parameters items is stored in the File Preamble
(string keys would significantly bloat the file size). (with a minimum of one item at index 0); a Block Parameters item
consists of a collection of Storage and Collection Parameters that
applies to any given Block. An array is used in order to support use
cases such as wanting to merge C-DNS files from different sources.
The Block Preamble item then contains an optional index for the Block
Parameters item that applies for that Block; if not present the index
defaults to 0. Hence, in effect, a global Block Parameters item is
defined which can then be overridden per Block.
7.3. File header contents 6.2. Storage Parameters
The file header contains the following: The Block Parameters item includes a Storage Parameters item - this
contains information about the specific data fields stored in the
C-DNS file.
+---------------+---------------+-----------------------------------+ These parameters include:
| Field | Type | Description |
+---------------+---------------+-----------------------------------+
| file-type-id | Text string | String "C-DNS" identifying the |
| | | file type. |
| | | |
| file-preamble | Map of items | Collection information for the |
| | | whole file. |
| | | |
| file-blocks | Array of | The data blocks. |
| | Blocks | |
+---------------+---------------+-----------------------------------+
7.4. File preamble contents o The sub-second timing resolution used by the data.
The file preamble contains the following: o Information (hints) on which optional data items can be expected
to appear in the data. See Section 6.2.1.
+----------------------+----------+---------------------------------+ o Recorded OPCODES and RR types. See Section 6.2.2.
| Field | Type | Description |
+----------------------+----------+---------------------------------+
| major-format-version | Unsigned | Unsigned integer '1'. The major |
| | | version of format used in file. |
| | | |
| minor-format-version | Unsigned | Unsigned integer '0'. The minor |
| | | version of format used in file. |
| | | |
| private-version | Unsigned | Version indicator available for |
| | | private use by applications. |
| | | Optional. |
| | | |
| configuration | Map of | The collection configuration. |
| | items | Optional. |
| | | |
| generator-id | Text | String identifying the |
| | string | collection program. Optional. |
| | | |
| host-id | Text | String identifying the |
| | string | collecting host. Empty if |
| | | converting an existing packet |
| | | capture file. Optional. |
+----------------------+----------+---------------------------------+
7.5. Configuration contents o Flags indicating whether the data is sampled or anonymised. See
Section 6.2.3.
The collection configuration contains the following items. All are o Client and server IPv4 and IPv6 address prefixes. See
optional. Section 6.2.4
+--------------------+----------+-----------------------------------+ 6.2.1. Optional data items
| Field | Type | Description |
+--------------------+----------+-----------------------------------+
| query-timeout | Unsigned | To be matched with a query, a |
| | | response must arrive within this |
| | | number of seconds. |
| | | |
| skew-timeout | Unsigned | The network stack may report a |
| | | response before the corresponding |
| | | query. A response is not |
| | | considered to be missing a query |
| | | until after this many micro- |
| | | seconds. |
| | | |
| snaplen | Unsigned | Collect up to this many bytes per |
| | | packet. |
| | | |
| promisc | Unsigned | 1 if promiscuous mode was enabled |
| | | on the interface, 0 otherwise. |
| | | |
| interfaces | Array of | Identifiers of the interfaces |
| | text | used for collection. |
| | strings | |
| | | |
| server-addresses | Array of | Server collection IP addresses. |
| | byte | Hint for downstream analysers; |
| | strings | does not affect collection. |
| | | |
| vlan-ids | Array of | Identifiers of VLANs selected for |
| | unsigned | collection. |
| | | |
| filter | Text | 'tcpdump' [pcap] style filter for |
| | string | input. |
| | | |
| query-options | Unsigned | Bit flags indicating sections in |
| | | Query messages to be collected. |
| | | Bit 0. Collect second and |
| | | subsequent Questions in the |
| | | Question section. |
| | | Bit 1. Collect Answer sections. |
| | | Bit 2. Collect Authority |
| | | sections. |
| | | Bit 3. Collection Additional |
| | | sections. |
| | | |
| response-options | Unsigned | Bit flags indicating sections in |
| | | Response messages to be |
| | | collected. |
| | | Bit 0. Collect second and |
| | | subsequent Questions in the |
| | | Question section. |
| | | Bit 1. Collect Answer sections. |
| | | Bit 2. Collect Authority |
| | | sections. |
| | | Bit 3. Collection Additional |
| | | sections. |
| | | |
| accept-rr-types | Array of | A set of RR type names [rrtypes]. |
| | text | If not empty, only the nominated |
| | strings | RR types are collected. |
| | | |
| ignore-rr-types | Array of | A set of RR type names [rrtypes]. |
| | text | If not empty, all RR types are |
| | strings | collected except those listed. If |
| | | present, this item must be empty |
| | | if a non-empty list of Accept RR |
| | | types is present. |
| | | |
| max-block-qr-items | Unsigned | Maximum number of Q/R data items |
| | | in a block. |
| | | |
| collect-malformed | Unsigned | 1 if malformed packet contents |
| | | are collected, 0 otherwise. |
+--------------------+----------+-----------------------------------+
7.6. Block contents To enable applications to store data to their precise requirements in
as space-efficient manner as possible, all fields in the following
arrays are optional:
Each block contains the following: o Query/Response
+-----------------------+--------------+----------------------------+ o Query Signature
| Field | Type | Description |
+-----------------------+--------------+----------------------------+
| preamble | Map of items | Overall information for |
| | | the block. |
| | | |
| statistics | Map of | Statistics about the |
| | statistics | block. Optional. |
| | | |
| tables | Map of | The tables containing data |
| | tables | referenced by individual |
| | | Q/R data items. |
| | | |
| queries | Array of Q/R | Details of individual Q/R |
| | data items | data items. |
| | | |
| address-event-counts | Array of | Per client counts of ICMP |
| | Address | messages and TCP resets. |
| | Event counts | Optional. |
| | | |
| malformed-packet-data | Array of | Wire contents of malformed |
| | malformed | packets. Optional. |
| | packets | |
+-----------------------+--------------+----------------------------+
7.7. Block preamble map o Malformed messages
In other words, an application can choose to omit any data item that
is not required for its use case. In addition, implementations may
be configured to not record all RRs, or only record messages with
certain OPCODES.
The block preamble map contains overall information for the block. This does, however, mean that a consumer of a C-DNS file faces two
problems:
+---------------+----------+----------------------------------------+ 1. How can it quickly determine whether a file contains the data
| Field | Type | Description | items it requires to complete a particular task (e.g.
+---------------+----------+----------------------------------------+ reconstructing query traffic or performing a specific piece of
| earliest-time | Array of | A timestamp for the earliest record in | data analysis)?
| | unsigned | the block. The timestamp is specified |
| | | as a CBOR array with two or three |
| | | elements. The first two elements are |
| | | as in Posix struct timeval. The first |
| | | element is an unsigned integer time_t |
| | | and the second is an unsigned integer |
| | | number of microseconds. The third, if |
| | | present, is an unsigned integer number |
| | | of picoseconds. The microsecond and |
| | | picosecond items always have a value |
| | | between 0 and 999,999. |
+---------------+----------+----------------------------------------+
7.8. Block statistics 2. How can it determine if a data item is not present because it was
The block statistics section contains some basic statistical * explicitly not recorded, or
information about the block. All are optional.
+---------------------+----------+----------------------------------+ * either was not present in the original data stream or the data
| Field | Type | Description | item was not available to the collecting application?
+---------------------+----------+----------------------------------+
| total-packets | Unsigned | Total number of packets | For example, an application capturing C-DNS data from within a
| | | processed from the input traffic | nameserver implementation is unlikely to be able to record the Client
| | | stream during collection of the | Hoplimit. Or, if there is no query ARCount recorded and no query OPT
| | | block data. | RDATA recorded, is that because no query contained an OPT RR, or
| total-pairs | Unsigned | Total number of Q/R data items | because that data was not stored?
| | | in the block. |
| unmatched-queries | Unsigned | Number of unmatched queries in | The Storage Parameters therefore also contains a Storage Hints item
| | | the block. | which specifies whether the encoder of the file recorded each data
| unmatched-responses | Unsigned | Number of unmatched responses in | item if it was present. An application decoding that file can then
| | | the block. | use these to quickly determine whether the input data is rich enough
| malformed-packets | Unsigned | Number of malformed packets | for its needs.
| | | found in input for the block. |
+---------------------+----------+----------------------------------+ QUESTION: Should the items within certain tables also be optional
e.g. within the RR table should all of Name index, ClassType, TTL and
RDATA be optional?
6.2.2. Optional RRs and OPCODES
Also included in the Storage Parameters is an explicit array of the
RR types and OPCODES that were recorded. Using an explicit array
removes any ambiguity about whether the OPCODE/RR type was not
recognised by the collecting implementation or whether it was
specifically configured not to record it.
For the case of unrecognised OPCODES the message may be parsable (for
example, if it has a format similar enough to the one described in
[RFC1035]) or it may not. See Section 8 for further discussion of
storing partially parsed messages.
6.2.3. Sampling and anonymisation
The format contains flags that can be used to indicate if the data is
either anonymised or produced from sample data.
QUESTION: Should fields be added to indicate the sampling/
anonymisation method used? If so, it is proposed to use a text
string and RECOMMEND it contain a URI pointing to a resource
describing the method used.
QUESTION: Should there be another flag to indicate that names have
been normalised (e.g. converted to uniform case)?
6.2.4. IP Address storage
The format contains fields to indicate if only IP prefixes were
stored. If IP address prefixes are given, only the prefix bits of
addresses are stored. For example, if a client IPv4 prefix of 16 is
specified, a client address of 192.0.2.1 will be stored as 0xc000
(192.0), reducing address storage space requirements.
7. C-DNS format detailed description
The CDDL definition for the C-DNS format is given in Appendix A.
7.1. Map quantities and indexes
All map keys are integers with values specified in the CDDL. String
keys would significantly bloat the file size.
All key values specified are positive integers under 24, so their
CBOR representation is a single byte.
Implementations may choose to add additional implementation-specific Implementations may choose to add additional implementation-specific
fields to the statistics. entries to any map. Negative integer map keys are reserved for these
values. Key values from -1 to -24 also have a single byte CBOR
representation, so such implementation-specific extensions are not at
any space efficiency disadvantage.
7.9. Block table map An item described as an index is the index of the data item in the
referenced array. Indexes are 0-based.
The block table map contains the block tables. Each element, or 7.2. Tabular representation
table, is an array. The following tables detail the contents of each
block table.
The Present column in the following tables indicates the The following sections present the C-DNS specification in tabular
circumstances when an optional field will be present. A Q/R data format with a detailed description of each item.
item may be:
o A Query plus a Response. In all quantities that contain bit flags, bit 0 indicates the least
significant bit, i.e. flag "n" in quantity "q" is on if "(q & (1 <<
n)) != 0".
o A Query without a Response. For the sake of readability, all type and field names defined in the
CDDL definition are shown in double quotes. Type names are by
convention camel case (e.g. "BlockTable"), field names are lower-
case with hyphens (e.g. "block-tables").
o A Response without a Query. For the sake of brevity, the following conventions are used in the
tables:
Also: o The column O marks whether items in a map are optional.
o A Query and/or a Response may contain an OPT section. * O - Optional. The item may be omitted.
o A Question may or may not be present. If the Query is available, * M - Mandatory. The item must be present.
the Question section of the Query is used. If no Query is
available, the Question section of the Response is used. Unless
otherwise noted, a Question refers to the first Question in the
Question section.
So, for example, a field listed with a Present value of QUERY is o The column T gives the CBOR data type of the item.
present whenever the Q/R data item contains a Query. If the pair
contains a Response only, the field will not be present.
7.10. IP address table * U - Unsigned integer
The table "ip-address" holds all client and server IP addresses in * I - Signed integer
the block. Each item in the table is a single IP address.
+------------+--------+---------------------------------------------+ * B - Byte string
| Field | Type | Description |
+------------+--------+---------------------------------------------+
| ip-address | Byte | The IP address, in network byte order. The |
| | string | string is 4 bytes long for an IPv4 address, |
| | | 16 bytes long for an IPv6 address. |
+------------+--------+---------------------------------------------+
7.11. Class/Type table * T - Text string
The table "classtype" holds pairs of RR CLASS and TYPE values. Each * M - Map
item in the table is a CBOR map.
+-------+----------+--------------+ * A - Array
| Field | Type | Description |
+-------+----------+--------------+
| type | Unsigned | TYPE value. |
| | | |
| class | Unsigned | CLASS value. |
+-------+----------+--------------+
7.12. Name/RDATA table In the case of maps and arrays, more information on the type of each
value, include the CDDL definition name if applicable, is given in
the description.
The table "name-rdata" holds the contents of all NAME or RDATA items 7.3. "File"
in the block. Each item in the table is the content of a single NAME
or RDATA.
Note that NAMEs, and labels within RDATA contents, are full domain A C-DNS file has an outer structure "File", a map that contains the
names or labels; no DNS style name compression is used on the following:
individual names/labels within the format.
+------------+-------------+----------------------------------------+ +---------------+---+---+-------------------------------------------+
| Field | Type | Description | | Field | O | T | Description |
+------------+-------------+----------------------------------------+ +---------------+---+---+-------------------------------------------+
| name-rdata | Byte string | The NAME or RDATA contents | | file-type-id | M | T | String "C-DNS" identifying the file type. |
| | | (uncompressed). | | | | | |
+------------+-------------+----------------------------------------+ | file-preamble | M | M | Version and parameter information for the |
| | | | whole file. Map of type "FilePreamble", |
| | | | see Section 7.4. |
| | | | |
| file-blocks | M | A | Array of items of type "Block", see |
| | | | Section 7.5. The array may be empty if |
| | | | the file contains no data. |
+---------------+---+---+-------------------------------------------+
7.13. Query Signature table 7.4. "FilePreamble"
The table "query-sig" holds elements of the Q/R data item that are Information about data in the file. A map containing the following:
often common between multiple individual Q/R data items. Each item
in the table is a CBOR map. Each item in the map has an unsigned
value and an unsigned integer key.
The following abbreviations are used in the Present (P) column +----------------------+---+---+------------------------------------+
| Field | O | T | Description |
+----------------------+---+---+------------------------------------+
| major-format-version | M | U | Unsigned integer '1'. The major |
| | | | version of format used in file. |
| | | | |
| minor-format-version | M | U | Unsigned integer '0'. The minor |
| | | | version of format used in file. |
| | | | |
| private-version | O | U | Version indicator available for |
| | | | private use by applications. |
| | | | |
| block-parameters | M | A | Array of items of type |
| | | | "BlockParameters", see Section |
| | | | 7.4.1. The array must contain at |
| | | | least one entry. (The "block- |
| | | | parameters-index" item in each |
| | | | "BlockPreamble" indicates which |
| | | | array entry applies to that |
| | | | "Block".) |
+----------------------+---+---+------------------------------------+
o Q = QUERY 7.4.1. "BlockParameters"
o A = Always Parameters relating to data storage and collection which apply to one
or more items of type "Block". An array containing the following:
o QT = QUESTION +-----------------------+---+---+-----------------------------------+
| Field | O | T | Description |
+-----------------------+---+---+-----------------------------------+
| storage-parameters | M | M | Parameters relating to data |
| | | | storage in a "Block" item. Map |
| | | | of type "StorageParameters", see |
| | | | Section 7.4.1.1. |
| | | | |
| collection-parameters | O | M | Parameters relating to collection |
| | | | of the data in a "Block" item. |
| | | | Map of type |
| | | | "CollectionParameters", see |
| | | | Section 7.4.2. |
+-----------------------+---+---+-----------------------------------+
o QO = QUERY, OPT 7.4.1.1. "StorageParameters"
o QR = QUERY & RESPONSE Parameters relating to how data is stored in the items of type
"Block". A map containing the following:
o R = RESPONSE +------------------+---+---+----------------------------------------+
| Field | O | T | Description |
+------------------+---+---+----------------------------------------+
| ticks-per-second | M | U | Sub-second timing is recorded in |
| | | | ticks. This specifies the number of |
| | | | ticks in a second. |
| | | | |
| max-block-items | M | U | The maximum number of items stored in |
| | | | any of the arrays in a "Block" item |
| | | | (Q/R items, address event counts or |
| | | | malformed messages). An indication to |
| | | | a decoder of the resources needed to |
| | | | process the file. |
| | | | |
| storage-hints | M | M | Collection of hints as to which fields |
| | | | are present in the arrays that have |
| | | | optional fields. Map of type |
| | | | "StorageHints", see Section 7.4.1.1.1. |
| | | | |
| opcodes | M | A | Array of OPCODES [opcodes] (unsigned |
| | | | integers) recorded by the collection |
| | | | application. |
| | | | |
| rr-types | M | A | Array of RR types [rrtypes] (unsigned |
| | | | integers) recorded by the collection |
| | | | application. |
| | | | |
| storage-flags | O | U | Bit flags indicating attributes of |
| | | | stored data. |
| | | | Bit 0. The data has been anonymised. |
| | | | Bit 1. The data is sampled data. |
| | | | |
| client-address | O | U | IPv4 client address prefix length. If |
| -prefix-ipv4 | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| client-address | O | U | IPv6 client address prefix length. If |
| -prefix-ipv6 | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| server-address | O | U | IPv4 server address prefix length. If |
| -prefix-ipv4 | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| server-address | O | U | IPv6 server address prefix length. If |
| -prefix-ipv6 | | | specified, only the address prefix |
| | | | bits are stored. |
+------------------+---+---+----------------------------------------+
+-----------------------+----+--------------------------------------+ 7.4.1.1.1. "StorageHints"
| Field | P | Description |
+-----------------------+----+--------------------------------------+
| server-address-index | A | The index in the IP address table of |
| | | the server IP address. |
| | | |
| server-port | A | The server port. |
| | | |
| transport-flags | A | Bit flags describing the transport |
| | | used to service the query. Bit 0 is |
| | | the least significant bit. |
| | | Bit 0. Transport type. 0 = UDP, 1 = |
| | | TCP. |
| | | Bit 1. IP type. 0 = IPv4, 1 = IPv6. |
| | | Bit 2. Trailing bytes in query |
| | | payload. The DNS query message in |
| | | the UDP payload was followed by some |
| | | additional bytes, which were |
| | | discarded. |
| | | |
| qr-sig-flags | A | Bit flags indicating information |
| | | present in this Q/R data item. Bit 0 |
| | | is the least significant bit. |
| | | Bit 0. 1 if a Query is present. |
| | | Bit 1. 1 if a Response is present. |
| | | Bit 2. 1 if one or more Question is |
| | | present. |
| | | Bit 3. 1 if a Query is present and |
| | | it has an OPT Resource Record. |
| | | Bit 4. 1 if a Response is present |
| | | and it has an OPT Resource Record. |
| | | Bit 5. 1 if a Response is present |
| | | but has no Question. |
| | | |
| query-opcode | Q | Query OPCODE. Optional. |
| | | |
| qr-dns-flags | A | Bit flags with values from the Query |
| | | and Response DNS flags. Bit 0 is the |
| | | least significant bit. Flag values |
| | | are 0 if the Query or Response is |
| | | not present. |
| | | Bit 0. Query Checking Disabled (CD). |
| | | Bit 1. Query Authenticated Data |
| | | (AD). |
| | | Bit 2. Query reserved (Z). |
| | | Bit 3. Query Recursion Available |
| | | (RA). |
| | | Bit 4. Query Recursion Desired (RD). |
| | | Bit 5. Query TrunCation (TC). |
| | | Bit 6. Query Authoritative Answer |
| | | (AA). |
| | | Bit 7. Query DNSSEC answer OK (DO). |
| | | Bit 8. Response Checking Disabled |
| | | (CD). |
| | | Bit 9. Response Authenticated Data |
| | | (AD). |
| | | Bit 10. Response reserved (Z). |
| | | Bit 11. Response Recursion Available |
| | | (RA). |
| | | Bit 12. Response Recursion Desired |
| | | (RD). |
| | | Bit 13. Response TrunCation (TC). |
| | | Bit 14. Response Authoritative |
| | | Answer (AA). |
| | | |
| query-rcode | Q | Query RCODE. If the Query contains |
| | | OPT, this value incorporates any |
| | | EXTENDED_RCODE_VALUE. Optional. |
| | | |
| query-classtype-index | QT | The index in the Class/Type table of |
| | | the CLASS and TYPE of the first |
| | | Question. Optional. |
| | | |
| query-qd-count | QT | The QDCOUNT in the Query, or |
| | | Response if no Query present. |
| | | Optional. |
| | | |
| query-an-count | Q | Query ANCOUNT. Optional. |
| | | |
| query-ar-count | Q | Query ARCOUNT. Optional. |
| | | |
| query-ns-count | Q | Query NSCOUNT. Optional. |
| | | |
| edns-version | QO | The Query EDNS version. Optional. |
| | | |
| udp-buf-size | QO | The Query EDNS sender's UDP payload |
| | | size. Optional. |
| | | |
| opt-rdata-index | QO | The index in the NAME/RDATA table of |
| | | the OPT RDATA. Optional. |
| | | |
| response-rcode | R | Response RCODE. If the Response |
| | | contains OPT, this value |
| | | incorporates any |
| | | EXTENDED_RCODE_VALUE. Optional. |
+-----------------------+----+--------------------------------------+
7.14. Question table An indicator of which fields the collecting application stores in the
arrays with optional fields. A map containing the following:
The table "qrr" holds details on individual Questions in a Question +------------------+---+---+----------------------------------------+
section. Each item in the table is a CBOR map containing a single | Field | O | T | Description |
Question. Each item in the map has an unsigned value and an unsigned +------------------+---+---+----------------------------------------+
integer key. This data is optionally collected. | query-response | M | U | Hints indicating which "QueryResponse" |
| -hints | | | fields are stored, see section Section |
| | | | 7.6. If the field is stored the bit is |
| | | | set. |
| | | | Bit 0. time-offset |
| | | | Bit 1. client-address-index |
| | | | Bit 2. client-port |
| | | | Bit 3. transaction-id |
| | | | Bit 4. qr-signature-index |
| | | | Bit 5. client-hoplimit |
| | | | Bit 6. response-delay |
| | | | Bit 7. query-name-index |
| | | | Bit 8. query-size |
| | | | Bit 9. response-size |
| | | | Bit 10. response-processing-data |
| | | | Bit 11. query-question-sections |
| | | | Bit 12. query-answer-sections |
| | | | Bit 13. query-authority-sections |
| | | | Bit 14. query-additional-sections |
| | | | Bit 15. response-answer-sections |
| | | | Bit 16. response-authority-sections |
| | | | Bit 17. response-additional-sections |
| | | | |
| query-response | M | U | Hints indicating which |
| -signature-hints | | | "QueryResponseSignature" fields are |
| | | | stored, see section Section 7.5.3.2. |
| | | | If the field is stored the bit is set. |
| | | | Bit 0. server-address |
| | | | Bit 1. server-port |
| | | | Bit 2. qr-transport-flags |
| | | | Bit 3. qr-type |
| | | | Bit 4. qr-sig-flags |
| | | | Bit 5. query-opcode |
| | | | Bit 6. dns-flags |
| | | | Bit 7. query-rcode |
| | | | Bit 8. query-class-type |
| | | | Bit 9. query-qdcount |
| | | | Bit 10. query-ancount |
| | | | Bit 11. query-nscount |
| | | | Bit 12. query-arcount |
| | | | Bit 13. query-edns-version |
| | | | Bit 14. query-udp-size |
| | | | Bit 15. query-opt-rdata |
| | | | Bit 16. response-rcode |
| | | | |
| other-data-hints | M | U | Hints indicating which other data |
| | | | types are stored. If the data type is |
| | | | stored the bit is set. |
| | | | Bit 0. malformed-messages |
| | | | Bit 1. address-event-counts |
+------------------+---+---+----------------------------------------+
+-----------------+-------------------------------------------------+ TODO: For completeness the other-data-hints need to cover optional
| Field | Description | fields in malformed message data maps.
+-----------------+-------------------------------------------------+
| name-index | The index in the NAME/RDATA table of the QNAME. |
| | |
| classtype-index | The index in the Class/Type table of the CLASS |
| | and TYPE of the Question. |
+-----------------+-------------------------------------------------+
7.15. Resource Record (RR) table 7.4.2. "CollectionParameters"
The table "rr" holds details on individual Resource Records in RR Parameters relating to how data in the file was collected.
sections. Each item in the table is a CBOR map containing a single
Resource Record. This data is optionally collected.
+-----------------+-------------------------------------------------+ These parameters have no default. If they do not appear, nothing can
| Field | Description | be inferred about their value.
+-----------------+-------------------------------------------------+
| name-index | The index in the NAME/RDATA table of the NAME. |
| | |
| classtype-index | The index in the Class/Type table of the CLASS |
| | and TYPE of the RR. |
| | |
| ttl | The RR Time to Live. |
| | |
| rdata-index | The index in the NAME/RDATA table of the RR |
| | RDATA. |
+-----------------+-------------------------------------------------+
7.16. Question list table A map containing the following items:
The table "qlist" holds a list of second and subsequent individual +------------------+---+---+----------------------------------------+
Questions in a Question section. Each item in the table is a CBOR | Field | O | T | Description |
unsigned integer. This data is optionally collected. +------------------+---+---+----------------------------------------+
| query-timeout | O | U | To be matched with a query, a response |
| | | | must arrive within this number of |
| | | | seconds. |
| | | | |
| skew-timeout | O | U | The network stack may report a |
| | | | response before the corresponding |
| | | | query. A response is not considered to |
| | | | be missing a query until after this |
| | | | many micro-seconds. |
| | | | |
| snaplen | O | U | Collect up to this many bytes per |
| | | | packet. |
| | | | |
| promisc | O | U | 1 if promiscuous mode was enabled on |
| | | | the interface, 0 otherwise. |
| | | | |
| interfaces | O | A | Array of identifiers (of type text |
| | | | string) of the interfaces used for |
| | | | collection. |
| | | | |
| server-addresses | O | A | Array of server collection IP |
| | | | addresses (of type byte string). Hint |
| | | | for downstream analysers; does not |
| | | | affect collection. |
| | | | |
| vlan-ids | O | A | Array of identifiers (of type unsigned |
| | | | integer) of VLANs selected for |
| | | | collection. |
| | | | |
| filter | O | T | "tcpdump" [pcap] style filter for |
| | | | input. |
| | | | |
| generator-id | O | T | String identifying the collection |
| | | | method. |
| | | | |
| host-id | O | T | String identifying the collecting |
| | | | host. Empty if converting an existing |
| | | | packet capture file. |
+------------------+---+---+----------------------------------------+
+----------+--------------------------------------------------------+ 7.5. "Block"
| Field | Description |
+----------+--------------------------------------------------------+
| question | The index in the Question table of the individual |
| | Question. |
+----------+--------------------------------------------------------+
7.17. Resource Record list table Container for data with common collection and and storage parameters.
A map containing the following:
The table "rrlist" holds a list of individual Resource Records in a +--------------------+---+---+--------------------------------------+
Answer, Authority or Additional section. Each item in the table is a | Field | O | T | Description |
CBOR unsigned integer. This data is optionally collected. +--------------------+---+---+--------------------------------------+
| block-preamble | M | M | Overall information for the "Block" |
| | | | item. Map of type "BlockPreamble", |
| | | | see Section 7.5.1. |
| | | | |
| block-statistics | O | M | Statistics about the "Block" item. |
| | | | Map of type "BlockStatistics", see |
| | | | Section 7.5.2. |
| | | | |
| block-tables | O | M | The arrays containing data |
| | | | referenced by individual |
| | | | "QueryResponse" or |
| | | | "MalformedMessage" items. Map of |
| | | | type "BlockTables", see Section |
| | | | 7.5.3. |
| | | | |
| query-responses | O | A | Details of individual DNS Q/R data |
| | | | items. Array of items of type |
| | | | "QueryResponse", see Section 7.6. If |
| | | | present, the array must not be |
| | | | empty. |
| | | | |
| address-event | O | A | Per client counts of ICMP messages |
| -counts | | | and TCP resets. Array of items of |
| | | | type "AddressEventCount", see |
| | | | Section 7.7. If present, the array |
| | | | must not be empty. |
| | | | |
| malformed-messages | O | A | Details of malformed DNS messages. |
| | | | Array of items of type |
| | | | "MalformedMessage", see Section 7.8. |
| | | | If present, the array must not be |
| | | | empty. |
+--------------------+---+---+--------------------------------------+
+-------+-----------------------------------------------------------+ 7.5.1. "BlockPreamble"
| Field | Description |
+-------+-----------------------------------------------------------+
| rr | The index in the Resource Record table of the individual |
| | Resource Record. |
+-------+-----------------------------------------------------------+
7.18. Query/Response data Overall information for a "Block" item. A map containing the
following:
The block Q/R data is a CBOR array of individual Q/R data items. +------------------+---+---+----------------------------------------+
Each item in the array is a CBOR map containing details on the | Field | O | T | Description |
individual Q/R data item. +------------------+---+---+----------------------------------------+
| earliest-time | O | A | A timestamp (2 unsigned integers, |
| | | | "Timestamp") for the earliest record |
| | | | in the "Block" item. The first integer |
| | | | is the number of seconds since the |
| | | | Posix epoch ("time_t"). The second |
| | | | integer is the number of ticks since |
| | | | the start of the second. This |
| | | | timestamp can only be omitted if all |
| | | | block items containing a time offset |
| | | | from the start of the block also omit |
| | | | the timestamp. |
| | | | |
| block-parameters | O | U | The index of the item in the "block- |
| -index | | | parameters" array (in the "file- |
| | | | premable" item) applicable to this |
| | | | block. If not present, index 0 is |
| | | | used. See Section 7.4.1. |
+------------------+---+---+----------------------------------------+
Note that there is no requirement that the elements of the Q/R array 7.5.2. "BlockStatistics"
are presented in strict chronological order.
The following abbreviations are used in the Present (P) column Basic statistical information about a "Block" item. A map containing
the following:
o Q = QUERY +---------------------+---+---+-------------------------------------+
| Field | O | T | Description |
+---------------------+---+---+-------------------------------------+
| total-messages | O | U | Total number of DNS messages |
| | | | processed from the input traffic |
| | | | stream during collection of data in |
| | | | this "Block" item. |
| | | | |
| total-pairs | O | U | Total number of Q/R data items in |
| | | | this "Block" item. |
| | | | |
| unmatched-queries | O | U | Number of unmatched queries in this |
| | | | "Block" item. |
| | | | |
| unmatched-responses | O | U | Number of unmatched responses in |
| | | | this "Block" item. |
| | | | |
| malformed-messages | O | U | Number of malformed messages found |
| | | | in input for this "Block" item. |
+---------------------+---+---+-------------------------------------+
o A = Always 7.5.3. "BlockTables"
o QT = QUESTION Arrays containing data referenced by individual "QueryResponse" or
"MalformedMessage" items in this "Block". Each element is an array
which, if present, must not be empty.
o QO = QUERY, OPT An item in the "qlist" array contains indexes to values in the "qrr"
array. Therefore, if "qlist" is present, "qrr" must also be present.
Similarly, if "rrlist" is present, "rr" must also be present.
o QR = QUERY & RESPONSE The map contains the following items:
o R = RESPONSE +-------------------+---+---+---------------------------------------+
| Field | O | T | Description |
+-------------------+---+---+---------------------------------------+
| ip-address | O | A | Array of IP addresses, in network |
| | | | byte order (of type byte string). If |
| | | | client or server address prefixes are |
| | | | set, only the address prefix bits are |
| | | | stored. Each string is therefore up |
| | | | to 4 bytes long for an IPv4 address, |
| | | | or up to 16 bytes long for an IPv6 |
| | | | address. See Section 7.4.1.1. |
| | | | |
| classtype | O | A | Array of RR class and type |
| | | | information. Type is "ClassType", see |
| | | | Section 7.5.3.1. |
| | | | |
| name-rdata | O | A | Array where each entry is the |
| | | | contents of a single NAME or RDATA |
| | | | (of type byte string). Note that |
| | | | NAMEs, and labels within RDATA |
| | | | contents, are full domain names or |
| | | | labels; no DNS style name compression |
| | | | is used on the individual |
| | | | names/labels within the format. |
| | | | |
| qr-sig | O | A | Array Q/R data item signatures. Type |
| | | | is "QueryResponseSignature", see |
| | | | Section 7.5.3.2. |
| | | | |
| qlist | O | A | Array of type "QuestionList". A |
| | | | "QuestionList" is an array of |
| | | | unsigned integers, indexes to |
| | | | "Question" items in the "qrr" array. |
| | | | |
| qrr | O | A | Array of type "Question". Each entry |
| | | | is the contents of a single question, |
| | | | where a question is the second or |
| | | | subsequent question in a query. See |
| | | | Section 7.5.3.3. |
| | | | |
| rrlist | O | A | Array of type "RRList". An "RRList" |
| | | | is an array of unsigned integers, |
| | | | indexes to "RR" items in the "rr" |
| | | | array. |
| | | | |
| rr | O | A | Array of type "RR". Each entry is the |
| | | | contents of a single RR. See Section |
| | | | 7.5.3.4. |
| | | | |
| malformed-message | O | A | Array of the contents of malformed |
| -data | | | messages. Array of type |
| | | | "MalformedMessageData", see Section |
| | | | 7.5.3.5. |
+-------------------+---+---+---------------------------------------+
Each item in the map has an unsigned value (with the exception of 7.5.3.1. "ClassType"
those listed below) and an unsigned integer key.
o query-extended and response-extended which are of type Extended RR class and type information. A map containing the following:
Information.
o delay-useconds and delay-pseconds which are integers (The delay +-------+---+---+--------------+
can be negative if the network stack/capture library returns them | Field | O | T | Description |
out of order.) +-------+---+---+--------------+
| type | M | U | TYPE value. |
| | | | |
| class | M | U | CLASS value. |
+-------+---+---+--------------+
+-----------------------+----+--------------------------------------+ 7.5.3.2. "QueryResponseSignature"
| Field | P | Description |
+-----------------------+----+--------------------------------------+
| time-useconds | A | Q/R timestamp as an offset in |
| | | microseconds from the Block preamble |
| | | Timestamp. The timestamp is the |
| | | timestamp of the Query, or the |
| | | Response if there is no Query. |
| | | |
| time-pseconds | A | Picosecond component of the |
| | | timestamp. Optional. |
| | | |
| client-address-index | A | The index in the IP address table of |
| | | the client IP address. |
| | | |
| client-port | A | The client port. |
| | | |
| transaction-id | A | DNS transaction identifier. |
| | | |
| query-signature-index | A | The index of the Query Signature |
| | | table record for this data item. |
| | | |
| client-hoplimit | Q | The IPv4 TTL or IPv6 Hoplimit from |
| | | the Query packet. Optional. |
| | | |
| delay-useconds | QR | The time difference between Query |
| | | and Response, in microseconds. Only |
| | | present if there is a query and a |
| | | response. |
| | | |
| delay-pseconds | QR | Picosecond component of the time |
| | | different between Query and |
| | | Response. If delay-useconds is non- |
| | | zero then delay-pseconds (if |
| | | present) MUST be of the same sign as |
| | | delay-useconds, or be 0. Optional. |
| | | |
| query-name-index | QT | The index in the NAME/RDATA table of |
| | | the QNAME for the first Question. |
| | | Optional. |
| | | |
| query-size | R | DNS query message size (see below). |
| | | Optional. |
| | | |
| response-size | R | DNS query message size (see below). |
| | | Optional. |
| | | |
| query-extended | Q | Extended Query information. This |
| | | item is only present if collection |
| | | of extra Query information is |
| | | configured. Optional. |
| | | |
| response-extended | R | Extended Response information. This |
| | | item is only present if collection |
| | | of extra Response information is |
| | | configured. Optional. |
+-----------------------+----+--------------------------------------+
An implementation must always collect basic Q/R information. It may Elements of a Q/R data item that are often common between multiple
be configured to collect details on Question, Answer, Authority and individual Q/R data items. A map containing the following:
Additional sections of the Query, the Response or both. Note that
only the second and subsequent Questions of any Question section are
collected (the details of the first are in the basic information),
and that OPT Records are not collected in the Additional section.
The query-size and response-size fields hold the DNS message size. +--------------------+---+---+--------------------------------------+
For UDP this is the size of the UDP payload that contained the DNS | Field | O | T | Description |
message and will therefore include any trailing bytes if present. +--------------------+---+---+--------------------------------------+
Trailing bytes with queries are routinely observed in traffic to | server-address | O | U | The index in the item in the "ip- |
authoritative servers and this value allows a calculation of how many | -index | | | address" array of the server IP |
trailing bytes were present. For TCP it is the size of the DNS | | | | address. See Section 7.5.3. |
message as specified in the two-byte message length header. | | | | |
| server-port | O | U | The server port. |
| | | | |
| qr-transport-flags | O | U | Bit flags describing the transport |
| | | | used to service the query. |
| | | | Bit 0. IP version. 0 = IPv4, 1 = |
| | | | IPv6 |
| | | | Bit 1-4. Transport. 0 = UDP, 1 = |
| | | | TCP, 2 = TLS, 3 = DTLS. |
| | | | Bit 5. Trailing bytes in query |
| | | | payload. The DNS query message in |
| | | | the UDP or TCP payload was followed |
| | | | by some additional bytes, which were |
| | | | discarded. |
| | | | |
| qr-type | O | U | Type of Query/Response transaction. |
| | | | 0 = Stub. A query from a stub |
| | | | resolver. |
| | | | 1 = Client. An incoming query to a |
| | | | recursive resolver. |
| | | | 2 = Resolver. A query sent from a |
| | | | recursive resolver to an authorative |
| | | | resolver. |
| | | | 3 = Authorative. A query to an |
| | | | authorative resolver. |
| | | | 4 = Forwarder. A query sent from a |
| | | | recursive resolver to an upstream |
| | | | recursive resolver. |
| | | | 5 = Tool. A query sent to a server |
| | | | by a server tool. |
| | | | |
| qr-sig-flags | O | U | Bit flags indicating information |
| | | | present in this Q/R data item. |
| | | | Bit 0. 1 if a Query is present. |
| | | | Bit 1. 1 if a Response is present. |
| | | | Bit 2. 1 if one or more Question is |
| | | | present. |
| | | | Bit 3. 1 if a Query is present and |
| | | | it has an OPT Resource Record. |
| | | | Bit 4. 1 if a Response is present |
| | | | and it has an OPT Resource Record. |
| | | | Bit 5. 1 if a Response is present |
| | | | but has no Question. |
| | | | |
| query-opcode | O | U | Query OPCODE. |
| | | | |
| qr-dns-flags | O | U | Bit flags with values from the Query |
| | | | and Response DNS flags. Flag values |
| | | | are 0 if the Query or Response is |
| | | | not present. |
| | | | Bit 0. Query Checking Disabled (CD). |
| | | | Bit 1. Query Authenticated Data |
| | | | (AD). |
| | | | Bit 2. Query reserved (Z). |
| | | | Bit 3. Query Recursion Available |
| | | | (RA). |
| | | | Bit 4. Query Recursion Desired (RD). |
| | | | Bit 5. Query TrunCation (TC). |
| | | | Bit 6. Query Authoritative Answer |
| | | | (AA). |
| | | | Bit 7. Query DNSSEC answer OK (DO). |
| | | | Bit 8. Response Checking Disabled |
| | | | (CD). |
| | | | Bit 9. Response Authenticated Data |
| | | | (AD). |
| | | | Bit 10. Response reserved (Z). |
| | | | Bit 11. Response Recursion Available |
| | | | (RA). |
| | | | Bit 12. Response Recursion Desired |
| | | | (RD). |
| | | | Bit 13. Response TrunCation (TC). |
| | | | Bit 14. Response Authoritative |
| | | | Answer (AA). |
| | | | |
| query-rcode | O | U | Query RCODE. If the Query contains |
| | | | OPT, this value incorporates any |
| | | | EXTENDED_RCODE_VALUE. |
| | | | |
| query-classtype | O | U | The index to the item in the the |
| -index | | | "classtype" array of the CLASS and |
| | | | TYPE of the first Question. See |
| | | | Section 7.5.3. |
| | | | |
| query-qd-count | O | U | The QDCOUNT in the Query, or |
| | | | Response if no Query present. |
| | | | |
| query-an-count | O | U | Query ANCOUNT. |
| | | | |
| query-ns-count | O | U | Query NSCOUNT. |
| | | | |
| query-ar-count | O | U | Query ARCOUNT. |
| | | | |
| edns-version | O | U | The Query EDNS version. |
| | | | |
| udp-buf-size | O | U | The Query EDNS sender's UDP payload |
| | | | size. |
| | | | |
| opt-rdata-index | O | U | The index in the "name-rdata" array |
| | | | of the OPT RDATA. See Section 7.5.3. |
| | | | |
| response-rcode | O | U | Response RCODE. If the Response |
| | | | contains OPT, this value |
| | | | incorporates any |
| | | | EXTENDED_RCODE_VALUE. |
+--------------------+---+---+--------------------------------------+
The Extended information is a CBOR map as follows. Each item in the QUESTION: Currently we collect OPT RDATA as a blob as this is
map is present only if collection of the relevant details is consistent with and re-uses the generic mechanism for RDATA storage.
configured. Each item in the map has an unsigned value and an Should we break individual EDNS(0) options into Option code and data
unsigned integer key. and store the data separately in a new array within the Block type?
This would potentially allow exploitation of option data commonality.
+------------------+------------------------------------------------+ QUESTION: No EDNS(0) option currently includes a name, however if one
| Field | Description | were to include a name and permit name compression then both these
+------------------+------------------------------------------------+ mechanisms would fail.
| question-index | The index in the Questions list table of the |
| | entry listing any second and subsequent |
| | Questions in the Question section for the |
| | Query or Response. |
| | |
| answer-index | The index in the RR list table of the entry |
| | listing the Answer Resource Record sections |
| | for the Query or Response. |
| | |
| authority-index | The index in the RR list table of the entry |
| | listing the Authority Resource Record sections |
| | for the Query or Response. |
| | |
| additional-index | The index in the RR list table of the entry |
| | listing the Additional Resource Record |
| | sections for the Query or Response. |
+------------------+------------------------------------------------+
7.19. Address Event counts 7.5.3.3. "Question"
This table holds counts of various IP related events relating to Details on individual Questions in a Question section. A map
traffic with individual client addresses. containing the following:
+------------------+----------+-------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | Type | Description | | Field | O | T | Description |
+------------------+----------+-------------------------------------+ +-----------------+---+---+-----------------------------------------+
| ae-type | Unsigned | The type of event. The following | | name-index | M | U | The index in the "name-rdata" array of |
| | | events types are currently defined: | | | | | the QNAME. See Section 7.5.3. |
| | | 0. TCP reset. | | | | | |
| | | 1. ICMP time exceeded. | | classtype-index | M | U | The index in the "classtype" array of |
| | | 2. ICMP destination unreachable. | | | | | the CLASS and TYPE of the Question. See |
| | | 3. ICMPv6 time exceeded. | | | | | Section 7.5.3. |
| | | 4. ICMPv6 destination unreachable. | +-----------------+---+---+-----------------------------------------+
| | | 5. ICMPv6 packet too big. |
| | | |
| ae-code | Unsigned | A code relating to the event. |
| | | Optional. |
| | | |
| ae-address-index | Unsigned | The index in the IP address table |
| | | of the client address. |
| | | |
| ae-count | Unsigned | The number of occurrences of this |
| | | event during the block collection |
| | | period. |
+------------------+----------+-------------------------------------+
7.20. Malformed packet records 7.5.3.4. "RR"
This optional table records the original wire format content of Details on individual Resource Records in RR sections. A map
malformed packets (see Section 8). containing the following:
+----------------+--------+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| Field | Type | Description | | Field | O | T | Description |
+----------------+--------+-----------------------------------------+ +-----------------+---+---+-----------------------------------------+
| time-useconds | A | Packet timestamp as an offset in | | name-index | M | U | The index in the "name-rdata" array of |
| | | microseconds from the Block preamble | | | | | the NAME. See Section 7.5.3. |
| | | Timestamp. | | | | | |
| | | | | classtype-index | M | U | The index in the "classtype" array of |
| time-pseconds | A | Picosecond component of the timestamp. | | | | | the CLASS and TYPE of the RR. See |
| | | Optional. | | | | | Section 7.5.3. |
| | | | | | | | |
| packet-content | Byte | The packet content in wire format. | | ttl | M | U | The RR Time to Live. |
| | string | | | | | | |
+----------------+--------+-----------------------------------------+ | rdata-index | M | U | The index in the "name-rdata" array of |
| | | | the RR RDATA. See Section 7.5.3. |
+-----------------+---+---+-----------------------------------------+
8. Malformed Packets 7.5.3.5. "MalformedMessageData"
Details on malformed message items in this "Block" item. A map
containing the following:
+--------------------+---+---+--------------------------------------+
| Field | O | T | Description |
+--------------------+---+---+--------------------------------------+
| server-address | O | U | The index in the "ip-address" array |
| -index | | | of the server IP address. See |
| | | | Section 7.5.3. |
| | | | |
| server-port | O | U | The server port. |
| | | | |
| mm-transport-flags | O | U | Bit flags describing the transport |
| | | | used to service the query. Bit 0 is |
| | | | the least significant bit. |
| | | | Bit 0. IP version. 0 = IPv4, 1 = |
| | | | IPv6 |
| | | | Bit 1-4. Transport. 0 = UDP, 1 = |
| | | | TCP, 2 = TLS, 3 = DTLS. |
| | | | |
| mm-payload | O | B | The payload (raw bytes) of the DNS |
| | | | message. |
+--------------------+---+---+--------------------------------------+
7.6. "QueryResponse"
Details on individual Q/R data items.
Note that there is no requirement that the elements of the "query-
responses" array are presented in strict chronological order.
A map containing the following items:
+----------------------+---+---+------------------------------------+
| Field | O | T | Description |
+----------------------+---+---+------------------------------------+
| time-offset | O | U | Q/R timestamp as an offset in |
| | | | ticks from "earliest-time". The |
| | | | timestamp is the timestamp of the |
| | | | Query, or the Response if there is |
| | | | no Query. |
| | | | |
| client-address-index | O | U | The index in the "ip-address" |
| | | | array of the client IP address. |
| | | | See Section 7.5.3. |
| | | | |
| client-port | O | U | The client port. |
| | | | |
| transaction-id | O | U | DNS transaction identifier. |
| | | | |
| qr-signature-index | O | U | The index in the "qr-sig" array of |
| | | | the "QueryResponseSignature" item. |
| | | | See Section 7.5.3. |
| | | | |
| client-hoplimit | O | U | The IPv4 TTL or IPv6 Hoplimit from |
| | | | the Query packet. |
| | | | |
| response-delay | O | I | The time difference between Query |
| | | | and Response, in ticks. Only |
| | | | present if there is a query and a |
| | | | response. The delay can be |
| | | | negative if the network |
| | | | stack/capture library returns |
| | | | packets out of order. |
| | | | |
| query-name-index | O | U | The index in the "name-rdata" |
| | | | array of the item containing the |
| | | | QNAME for the first Question. See |
| | | | Section 7.5.3. |
| | | | |
| query-size | O | U | DNS query message size (see |
| | | | below). |
| | | | |
| response-size | O | U | DNS query message size (see |
| | | | below). |
| | | | |
| response-processing | O | M | Data on response processing. Map |
| -data | | | of type "ResponseProcessingData", |
| | | | see Section 7.6.1. |
| | | | |
| query-extended | O | M | Extended Query data. Map of type |
| | | | "QueryResponseExtended", see |
| | | | Section 7.6.2. |
| | | | |
| response-extended | O | M | Extended Response data. Map of |
| | | | type "QueryResponseExtended", see |
| | | | Section 7.6.2. |
+----------------------+---+---+------------------------------------+
The "query-size" and "response-size" fields hold the DNS message
size. For UDP this is the size of the UDP payload that contained the
DNS message. For TCP it is the size of the DNS message as specified
in the two-byte message length header. Trailing bytes with queries
are routinely observed in traffic to authoritative servers and this
value allows a calculation of how many trailing bytes were present.
7.6.1. "ResponseProcessingData"
Information on the server processing that produced the response. A
map containing the following:
+------------------+---+---+----------------------------------------+
| Field | O | T | Description |
+------------------+---+---+----------------------------------------+
| bailiwick-index | O | U | The index in the "name-rdata" array of |
| | | | the owner name for the response |
| | | | bailiwick. See Section 7.5.3. |
| | | | |
| processing-flags | O | U | Flags relating to response processing. |
| | | | Bit 0. 1 if the response came from |
| | | | cache. |
+------------------+---+---+----------------------------------------+
QUESTION: Should this be an item in the "QueryResponseSignature"?
7.6.2. "QueryResponseExtended"
Extended data on the Q/R data item.
Each item in the map is present only if collection of the relevant
details is configured.
A map containing the following items:
+------------------+---+---+----------------------------------------+
| Field | O | T | Description |
+------------------+---+---+----------------------------------------+
| question-index | O | U | The index in the "qlist" array of the |
| | | | entry listing any second and |
| | | | subsequent Questions in the Question |
| | | | section for the Query or Response. See |
| | | | Section 7.5.3. |
| | | | |
| answer-index | O | U | The index in the "rrlist" array of the |
| | | | entry listing the Answer Resource |
| | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. |
| | | | |
| authority-index | O | U | The index in the "rrlist" array of the |
| | | | entry listing the Authority Resource |
| | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. |
| | | | |
| additional-index | O | U | The index in the "rrlist" array of the |
| | | | entry listing the Additional Resource |
| | | | Record sections for the Query or |
| | | | Response. See Section 7.5.3. |
+------------------+---+---+----------------------------------------+
7.7. "AddressEventCount"
Counts of various IP related events relating to traffic with
individual client addresses. A map containing the following:
+------------------+---+---+----------------------------------------+
| Field | O | T | Description |
+------------------+---+---+----------------------------------------+
| ae-type | M | U | The type of event. The following |
| | | | events types are currently defined: |
| | | | 0. TCP reset. |
| | | | 1. ICMP time exceeded. |
| | | | 2. ICMP destination unreachable. |
| | | | 3. ICMPv6 time exceeded. |
| | | | 4. ICMPv6 destination unreachable. |
| | | | 5. ICMPv6 packet too big. |
| | | | |
| ae-code | O | U | A code relating to the event. |
| | | | |
| ae-address-index | M | U | The index in the "ip-address" array of |
| | | | the client address. See Section 7.5.3. |
| | | | |
| ae-count | M | U | The number of occurrences of this |
| | | | event during the block collection |
| | | | period. |
+------------------+---+---+----------------------------------------+
7.8. "MalformedMessage"
Details of malformed messages. See Section 8. A map containing the
following:
+----------------------+---+---+------------------------------------+
| Field | O | T | Description |
+----------------------+---+---+------------------------------------+
| time-offset | O | U | Message timestamp as an offset in |
| | | | ticks from "earliest-time". |
| | | | |
| client-address-index | O | U | The index in the "ip-address" |
| | | | array of the client IP address. |
| | | | See Section 7.5.3. |
| | | | |
| client-port | O | U | The client port. |
| | | | |
| message-data-index | O | U | The index in the "malformed- |
| | | | message-data" array of the message |
| | | | data for this message. See Section |
| | | | 7.5.3. |
+----------------------+---+---+------------------------------------+
8. Malformed messages
In the context of generating a C-DNS file it is assumed that only In the context of generating a C-DNS file it is assumed that only
those packets which can be parsed to produce a well-formed DNS those DNS messages which can be parsed to produce a well-formed DNS
message are stored in the C-DNS format. This means as a minimum: message are stored in the C-DNS format and that all other messages
will be recorded (if at all) as malformed messages.
o The packet has a well-formed 12 bytes DNS Header Parsing a well-formed message means as a minimum:
o The packet has a well-formed 12 byte DNS Header
o The section counts are consistent with the section contents o The section counts are consistent with the section contents
o All of the resource records can be parsed o All of the resource records can be parsed
In principle, packets that do not meet these criteria could be In principle, packets that do not meet these criteria could be
classified into two categories: classified into two categories:
o Partially malformed: those packets which can be decoded o Partially malformed: those packets which can be decoded
sufficiently to extract sufficiently to extract
* a DNS header (and therefore a DNS transaction ID) * a well-formed 12 byte DNS header (and therefore a DNS
transaction ID)
* a QDCOUNT
* the first Question in the Question section if QDCOUNT is * the first Question in the Question section if QDCOUNT is
greater than 0 greater than 0
but suffer other issues while parsing. This is the minimum but suffer other issues while parsing. This is the minimum
information required to attempt Query/Response matching as information required to attempt Query/Response matching as described
described in Section 10.1 in Section 10.1.
o Completely malformed: those packets that cannot be decoded to this o Completely malformed: those packets that cannot be decoded to this
extent. extent.
An open question is whether there is value in attempting to process An open question is whether there is value in attempting to process
partially malformed packets in an analogous manner to well formed partially malformed messages in an analogous manner to well formed
packets in terms of attempting to match them with the corresponding messages in terms of attempting to match them with the corresponding
query or response. This could be done by creating 'placeholder' query or response. This could be done by creating 'placeholder'
records during Query/Response matching with just the information records during Query/Response matching with just the information
extracted as above. If the packet were then matched the resulting extracted as above. If the packet were then matched the resulting
C-DNS Q/R data item would include a flag to indicate a malformed C-DNS Q/R data item would include flags to indicate a malformed query
record (in addition to capturing the wire format of the packet). or response or both record (in addition to capturing the wire format
of the packet).
An advantage of this would be that it would result in more meaningful An advantage of this would be that it would result in more meaningful
statistics about matched packets because, for example, some partially statistics about matched packets because, for example, some partially
malformed queries could be matched to responses. However it would malformed queries could be matched to responses. However it would
only apply to those queries where the first Question is well formed. only apply to those queries where the first Question is well formed.
It could also simplify the downstream analysis of C-DNS files and the It could also simplify the downstream analysis of C-DNS files and the
reconstruction of packet streams from C-DNS. reconstruction of packet streams from C-DNS.
A disadvantage is that this adds complexity to the Query/Response A disadvantage is that this adds complexity to the Query/Response
matching and data representation, could potentially lead to false matching and data representation, could potentially lead to false
matches and some additional statistics would be required (e.g. counts matches and some additional statistics would be required (e.g. counts
for matched-partially-malformed, unmatched-partially-malformed, for matched-partially-malformed, unmatched-partially-malformed,
completely-malformed). completely-malformed).
NOTE: Note that within these definitions a message that contained an
unrecognised OPCODE or RR code would be treated as malformed. It may
be the case that the OPCODE/RR is not recognised just because the
implementation does not support it yet, rather than it not being
standardized. For the case of unrecognised OPCODES the message may
be parsable (for example, if it has a format similar enough to the
one described in [RFC1035]) or it may not. Similarly for
unrecognised RR types the RDATA can still be stored, but the
collector will not be able to process it to remove, for example, name
compression pointers.
QUESTION: There has been no feedback to date requesting further work
on the processing partially malformed messages. The editors are
inclined not to include it in this version. It could be the subject
of a future extension.
9. C-DNS to PCAP 9. C-DNS to PCAP
It is possible to re-construct PCAP files from the C-DNS format in a It is possible to re-construct PCAP files from the C-DNS format in a
lossy fashion. Some of the issues with reconstructing both the DNS lossy fashion. Some of the issues with reconstructing both the DNS
payload and the full packet stream are outlined here. payload and the full packet stream are outlined here.
The reconstruction depends on whether or not all the optional The reconstruction depends on whether or not all the optional
sections of both the query and response were captured in the C-DNS sections of both the query and response were captured in the C-DNS
file. Clearly, if they were not all captured, the reconstruction file. Clearly, if they were not all captured, the reconstruction
will be imperfect. will be imperfect.
skipping to change at page 26, line 5 skipping to change at page 34, line 21
o Malformed DNS messages if the wire format is not recorded o Malformed DNS messages if the wire format is not recorded
o Any Non-DNS messages that were in the original packet stream e.g. o Any Non-DNS messages that were in the original packet stream e.g.
ICMP ICMP
Simple assumptions can be made on the reconstruction: fragmented and Simple assumptions can be made on the reconstruction: fragmented and
DNS-over-TCP messages can be reconstructed into single packets and a DNS-over-TCP messages can be reconstructed into single packets and a
single TCP session can be constructed for each TCP packet. single TCP session can be constructed for each TCP packet.
Additionally, if malformed packets and Non-DNS packets are captured Additionally, if malformed messages and Non-DNS packets are captured
separately, they can be merged with packet captures reconstructed separately, they can be merged with packet captures reconstructed
from C-DNS to produce a more complete packet stream. from C-DNS to produce a more complete packet stream.
9.1. Name Compression 9.1. Name compression
All the names stored in the C-DNS format are full domain names; no All the names stored in the C-DNS format are full domain names; no
DNS style name compression is used on the individual names within the DNS style name compression is used on the individual names within the
format. Therefore when reconstructing a packet, name compression format. Therefore when reconstructing a packet, name compression
must be used in order to reproduce the on the wire representation of must be used in order to reproduce the on the wire representation of
the packet. the packet.
[RFC1035] name compression works by substituting trailing sections of [RFC1035] name compression works by substituting trailing sections of
a name with a reference back to the occurrence of those sections a name with a reference back to the occurrence of those sections
earlier in the message. Not all name server software uses the same earlier in the message. Not all name server software uses the same
skipping to change at page 26, line 46 skipping to change at page 35, line 13
algorithm in turn to see if it reproduces the original length, algorithm in turn to see if it reproduces the original length,
stopping at the first match. This would not guarantee the correct stopping at the first match. This would not guarantee the correct
algorithm has been used as it is possible to match the length whilst algorithm has been used as it is possible to match the length whilst
still not matching the on the wire bytes but, without further still not matching the on the wire bytes but, without further
information added to the C-DNS data, this is the best that can be information added to the C-DNS data, this is the best that can be
achieved. achieved.
Appendix B presents an example of two different compression Appendix B presents an example of two different compression
algorithms used by well-known name server software. algorithms used by well-known name server software.
10. Data Collection 10. Data collection
This section describes a non-normative proposed algorithm for the This section describes a non-normative proposed algorithm for the
processing of a captured stream of DNS queries and responses and processing of a captured stream of DNS queries and responses and
matching queries/responses where possible. matching queries/responses where possible.
For the purposes of this discussion, it is assumed that the input has For the purposes of this discussion, it is assumed that the input has
been pre-processed such that: been pre-processed such that:
1. All IP fragmentation reassembly, TCP stream reassembly, and so 1. All IP fragmentation reassembly, TCP stream reassembly, and so
on, has already been performed on, has already been performed
skipping to change at page 28, line 16 skipping to change at page 36, line 32
6. DNS Message ID 6. DNS Message ID
10.2.2. Secondary ID (optional) 10.2.2. Secondary ID (optional)
If present, the first Question in the Question section is used as a If present, the first Question in the Question section is used as a
secondary ID for each message. Note that there may be well formed secondary ID for each message. Note that there may be well formed
DNS queries that have a QDCOUNT of 0, and some responses may have a DNS queries that have a QDCOUNT of 0, and some responses may have a
QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP).
In this case the secondary ID is not used in matching. In this case the secondary ID is not used in matching.
10.3. Algorithm Parameters 10.3. Algorithm parameters
1. Query timeout 1. Query timeout
2. Skew timeout 2. Skew timeout
10.4. Algorithm Requirements 10.4. Algorithm requirements
The algorithm is designed to handle the following input data: The algorithm is designed to handle the following input data:
1. Multiple queries with the same Primary ID (but different 1. Multiple queries with the same Primary ID (but different
Secondary ID) arriving before any responses for these queries are Secondary ID) arriving before any responses for these queries are
seen. seen.
2. Multiple queries with the same Primary and Secondary ID arriving 2. Multiple queries with the same Primary and Secondary ID arriving
before any responses for these queries are seen. before any responses for these queries are seen.
3. Queries for which no later response can be found within the 3. Queries for which no later response can be found within the
specified timeout. specified timeout.
4. Responses for which no previous query can be found within the 4. Responses for which no previous query can be found within the
specified timeout. specified timeout.
10.5. Algorithm Limitations 10.5. Algorithm limitations
For cases 1 and 2 listed in the above requirements, it is not For cases 1 and 2 listed in the above requirements, it is not
possible to unambiguously match queries with responses. This possible to unambiguously match queries with responses. This
algorithm chooses to match to the earliest query with the correct algorithm chooses to match to the earliest query with the correct
Primary and Secondary ID. Primary and Secondary ID.
10.6. Workspace 10.6. Workspace
A FIFO structure is used to hold the Q/R data items during A FIFO structure is used to hold the Q/R data items during
processing. processing.
skipping to change at page 29, line 20 skipping to change at page 37, line 35
1. A matched pair of query and response messages 1. A matched pair of query and response messages
2. A query message with no response 2. A query message with no response
3. A response message with no query 3. A response message with no query
The timestamp of a list item is that of the query for cases 1 and 2 The timestamp of a list item is that of the query for cases 1 and 2
and that of the response for case 3. and that of the response for case 3.
10.8. Post Processing 10.8. Post processing
When ending capture, all remaining entries in the Q/R data item FIFO When ending capture, all remaining entries in the Q/R data item FIFO
should be treated as timed out queries. should be treated as timed out queries.
11. Implementation Status 11. Implementation guidance
Whilst this document makes no specific recommendations with respect
to Canonical CBOR (see Section 3.9 of [RFC7049]) the following
guidance may be of use to implementors.
Adherence to the first two rules given in Section 3.9 of [RFC7049]
will minimise file sizes.
Adherence to the second two rules given in Section 3.9 of [RFC7049]
for all maps and arrays would unacceptably constrain implementations,
for example, in the use case of real-time data collection in
constrained environments.
NOTE: With this clarification to the use of Canonical CBOR, we could
consider re-ordering fields in maps to improve readability.
11.1. Optional data
When decoding data some items required for a particular function the
consumer wishes to perform may be missing. Consumers should consider
providing configurable default values to be used in place of the
missing values in their output.
11.2. Trailing data in TCP
TODO: Clarify the impact of processing wire captures which includes
trailing data in TCP. What will appear as trailing data, what will
appear as malformed messages?
11.3. Limiting collection of RDATA
Implementations should consider providing a configurable maximum
RDATA size for capture , for example, to avoid memory issues when
confronted with large XFR records.
12. Implementation status
[Note to RFC Editor: please remove this section and reference to [Note to RFC Editor: please remove this section and reference to
[RFC7942] prior to publication.] [RFC7942] prior to publication.]
This section records the status of known implementations of the This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942]. Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to The description of implementations in this section is intended to
assist the IETF in its decision processes in progressing drafts to assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation RFCs. Please note that the listing of any individual implementation
skipping to change at page 30, line 5 skipping to change at page 39, line 5
features. Readers are advised to note that other implementations may features. Readers are advised to note that other implementations may
exist. exist.
According to [RFC7942], "this will allow reviewers and working groups According to [RFC7942], "this will allow reviewers and working groups
to assign due consideration to documents that have the benefit of to assign due consideration to documents that have the benefit of
running code, which may serve as evidence of valuable experimentation running code, which may serve as evidence of valuable experimentation
and feedback that have made the implemented protocols more mature. and feedback that have made the implemented protocols more mature.
It is up to the individual working groups to use this information as It is up to the individual working groups to use this information as
they see fit". they see fit".
11.1. DNS-STATS Compactor 12.1. DNS-STATS Compactor
ICANN/Sinodun IT have developed an open source implementation called ICANN/Sinodun IT have developed an open source implementation called
DNS-STATS Compactor. The Compactor is a suite of tools which can DNS-STATS Compactor. The Compactor is a suite of tools which can
capture DNS traffic (from either a network interface or a PCAP file) capture DNS traffic (from either a network interface or a PCAP file)
and store it in the Compacted-DNS (C-DNS) file format. PCAP files and store it in the Compacted-DNS (C-DNS) file format. PCAP files
for the captured traffic can also be reconstructed. See Compactor for the captured traffic can also be reconstructed. See Compactor
[7]. [7].
This implementation: This implementation:
o is mature but has only been deployed for testing in a single o is mature but has only been deployed for testing in a single
environment so is not yet classified as production ready. environment so is not yet classified as production ready.
o covers the whole of the specification described in the -03 draft o covers the whole of the specification described in the -03 draft
with the exception of support for malformed packets (Section 8) with the exception of support for malformed messages (Section 8)
and pico second time resolution. (Note: this implementation does and pico second time resolution. (Note: this implementation does
allow malformed packets to be dumped to a PCAP file). allow malformed messages to be dumped to a PCAP file).
o is released under the Mozilla Public License Version 2.0. o is released under the Mozilla Public License Version 2.0.
o has a users mailing list available, see dns-stats-users [8]. o has a users mailing list available, see dns-stats-users [8].
There is also some discussion of issues encountered during There is also some discussion of issues encountered during
development available at Compressing Pcap Files [9] and Packet development available at Compressing Pcap Files [9] and Packet
Capture [10]. Capture [10].
This information was last updated on 29th of June 2017. This information was last updated on 29th of June 2017.
12. IANA Considerations 13. IANA considerations
None None
13. Security Considerations 14. Security considerations
Any control interface MUST perform authentication and encryption. Any control interface MUST perform authentication and encryption.
Any data upload MUST be authenticated and encrypted. Any data upload MUST be authenticated and encrypted.
14. Acknowledgements 15. Acknowledgements
The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for
many useful discussions on binary formats, compression and packet many useful discussions on binary formats, compression and packet
matching. Also Jan Vcelak and Wouter Wijngaards for discussions on matching. Also Jan Vcelak and Wouter Wijngaards for discussions on
name compression and Paul Hoffman for a detailed review of the name compression and Paul Hoffman for a detailed review of the
document and the C-DNS CDDL. document and the C-DNS CDDL.
Thanks also to Robert Edmonds and Jerry Lundstroem for review. Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson,
Stephane Bortzmeyer and many other members of DNSOP for review.
Also, Miek Gieben for mmark [11] Also, Miek Gieben for mmark [11]
15. Changelog 16. Changelog
draft-ietf-dnsop-dns-capture-format-05
o Make all data items in Q/R, QuerySignature and Malformed Message
arrays optional
o Re-structure the FilePreamble and ConfigurationParameters into
BlockParameters
o BlockParameters has separate Storage and Collection Parameters
o Storage Parameters includes information on what optional fields
are present, and flags specifying anonymisation or sampling
o Addresses can now be stored as prefixes.
o Switch to using a variable sub-second timing granularity
o Add response bailiwick and query response type
o Add specifics of how to record malformed messages
o Add implementation guidance
o Improve terminology and naming consistency
draft-ietf-dnsop-dns-capture-format-04 draft-ietf-dnsop-dns-capture-format-04
o Correct query-d0 to query-do in CDDL o Correct query-d0 to query-do in CDDL
o Clarify that map keys are unsigned integers o Clarify that map keys are unsigned integers
o Add Type to Class/type table o Add Type to Class/Type table
o Clarify storage format in section 7.12 o Clarify storage format in section 7.12
draft-ietf-dnsop-dns-capture-format-03 draft-ietf-dnsop-dns-capture-format-03
o Added an Implementation Status section o Added an Implementation Status section
draft-ietf-dnsop-dns-capture-format-02 draft-ietf-dnsop-dns-capture-format-02
o Update qr_data_format.png to match CDDL o Update qr_data_format.png to match CDDL
o Editorial clarifications and improvements o Editorial clarifications and improvements
draft-ietf-dnsop-dns-capture-format-01 draft-ietf-dnsop-dns-capture-format-01
o Many editorial improvements by Paul Hoffman o Many editorial improvements by Paul Hoffman
o Included discussion of malformed packet handling o Included discussion of malformed message handling
o Improved Appendix C on Comparison of Binary Formats o Improved Appendix C on Comparison of Binary Formats
o Now using C-DNS field names in the tables in section 8 o Now using C-DNS field names in the tables in section 8
o A handful of new fields included (CDDL updated) o A handful of new fields included (CDDL updated)
o Timestamps now include optional picoseconds o Timestamps now include optional picoseconds
o Added details of block statistics o Added details of block statistics
skipping to change at page 32, line 16 skipping to change at page 41, line 44
o Changed DNS-STAT to C-DNS in CDDL o Changed DNS-STAT to C-DNS in CDDL
o Set the format version in the CDDL o Set the format version in the CDDL
o Added a TODO: Add block statistics o Added a TODO: Add block statistics
o Added a TODO: Add extend to support pico/nano. Also do this for o Added a TODO: Add extend to support pico/nano. Also do this for
Time offset and Response delay Time offset and Response delay
o Added a TODO: Need to develop optional representation of malformed o Added a TODO: Need to develop optional representation of malformed
packets within C-DNS and what this means for packet matching. messages within C-DNS and what this means for packet matching.
This may influence which fields are optional in the rest of the This may influence which fields are optional in the rest of the
representation. representation.
o Added section on design goals to Introduction o Added section on design goals to Introduction
o Added a TODO: Can Class be optimised? Should a class of IN be o Added a TODO: Can Class be optimised? Should a class of IN be
inferred if not present? inferred if not present?
draft-dickinson-dnsop-dns-capture-format-00
o Initial commit o Initial commit
16. References 17. References
16.1. Normative References 17.1. Normative References
[RFC1035] Mockapetris, P., "Domain names - implementation and [RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>. November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, <https://www.rfc- DOI 10.17487/RFC2119, March 1997, <https://www.rfc-
editor.org/info/rfc2119>. editor.org/info/rfc2119>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <https://www.rfc-editor.org/info/rfc7049>. October 2013, <https://www.rfc-editor.org/info/rfc7049>.
16.2. Informative References 17.2. Informative References
[ditl] DNS-OARC, "DITL", 2016, <https://www.dns- [ditl] DNS-OARC, "DITL", 2016, <https://www.dns-
oarc.net/oarc/data/ditl>. oarc.net/oarc/data/ditl>.
[dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/ [dnscap] DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
dnscap>. dnscap>.
[dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>. [dnstap] dnstap.info, "dnstap", 2016, <http://dnstap.info/>.
[dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
<https://www.dns-oarc.net/tools/dsc>. <https://www.dns-oarc.net/tools/dsc>.
[I-D.daley-dnsxml] [I-D.daley-dnsxml]
Daley, J., Morris, S., and J. Dickinson, "dnsxml - A Daley, J., Morris, S., and J. Dickinson, "dnsxml - A
standard XML representation of DNS data", draft-daley- standard XML representation of DNS data", draft-daley-
dnsxml-00 (work in progress), July 2013. dnsxml-00 (work in progress), July 2013.
[I-D.greevenbosch-appsawg-cbor-cddl]
Birkholz, H., Vigano, C., and C. Bormann, "Concise data
definition language (CDDL): a notational convention to
express CBOR data structures", draft-greevenbosch-appsawg-
cbor-cddl-11 (work in progress), July 2017.
[I-D.hoffman-dns-in-json] [I-D.hoffman-dns-in-json]
Hoffman, P., "Representing DNS Messages in JSON", draft- Hoffman, P., "Representing DNS Messages in JSON", draft-
hoffman-dns-in-json-13 (work in progress), October 2017. hoffman-dns-in-json-13 (work in progress), October 2017.
[I-D.ietf-cbor-cddl]
Birkholz, H., Vigano, C., and C. Bormann, "Concise data
definition language (CDDL): a notational convention to
express CBOR data structures", draft-ietf-cbor-cddl-01
(work in progress), January 2018.
[opcodes] IANA, "OPCODES", 2016, <http://www.iana.org/assignments/
dns-parameters/dns-parameters.xhtml#dns-parameters-5>.
[packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ",
2014, <https://github.com/dotse/PacketQ>. 2014, <https://github.com/dotse/PacketQ>.
[pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>. [pcap] tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.
[pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
Harris, "pcap-ng", 2016, <https://github.com/pcapng/ Harris, "pcap-ng", 2016, <https://github.com/pcapng/
pcapng>. pcapng>.
[RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data [RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
skipping to change at page 34, line 5 skipping to change at page 43, line 29
2014, <https://www.rfc-editor.org/info/rfc7159>. 2014, <https://www.rfc-editor.org/info/rfc7159>.
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205, Code: The Implementation Status Section", BCP 205,
RFC 7942, DOI 10.17487/RFC7942, July 2016, RFC 7942, DOI 10.17487/RFC7942, July 2016,
<https://www.rfc-editor.org/info/rfc7942>. <https://www.rfc-editor.org/info/rfc7942>.
[rrtypes] IANA, "RR types", 2016, <http://www.iana.org/assignments/ [rrtypes] IANA, "RR types", 2016, <http://www.iana.org/assignments/
dns-parameters/dns-parameters.xhtml#dns-parameters-4>. dns-parameters/dns-parameters.xhtml#dns-parameters-4>.
16.3. URIs 17.3. URIs
[1] https://github.com/dns-stats/draft-dns-capture- [1] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/cdns_format.png format/blob/master/draft-05/cdns_format.png
[2] https://github.com/dns-stats/draft-dns-capture- [2] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/cdns_format.svg format/blob/master/draft-05/cdns_format.svg
[3] https://github.com/dns-stats/draft-dns-capture- [3] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/qr_data_format.png format/blob/master/draft-05/qr_data_format.png
[4] https://github.com/dns-stats/draft-dns-capture- [4] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/qr_data_format.svg format/blob/master/draft-05/qr_data_format.svg
[5] https://github.com/dns-stats/draft-dns-capture- [5] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/packet_matching.png format/blob/master/draft-05/packet_matching.png
[6] https://github.com/dns-stats/draft-dns-capture- [6] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/draft-04/packet_matching.svg format/blob/master/draft-05/packet_matching.svg
[7] https://github.com/dns-stats/compactor/wiki [7] https://github.com/dns-stats/compactor/wiki
[8] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users [8] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
[9] https://www.sinodun.com/2017/06/compressing-pcap-files/ [9] https://www.sinodun.com/2017/06/compressing-pcap-files/
[10] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu- [10] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu-
trusty-packet-capture-woes/ trusty-packet-capture-woes/
skipping to change at page 35, line 17 skipping to change at page 44, line 42
[22] http://tukaani.org/xz/ [22] http://tukaani.org/xz/
[23] https://github.com/dns-stats/draft-dns-capture- [23] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.png format/blob/master/file-size-versus-block-size.png
[24] https://github.com/dns-stats/draft-dns-capture- [24] https://github.com/dns-stats/draft-dns-capture-
format/blob/master/file-size-versus-block-size.svg format/blob/master/file-size-versus-block-size.svg
Appendix A. CDDL Appendix A. CDDL
; CDDL specification of the file format for C-DNS, ; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS messages and ; which describes a collection of DNS messages and
; traffic meta-data. ; traffic meta-data.
File = [ ;
file-type-id : tstr, ; = "C-DNS" ; The overall structure of a file.
file-preamble : FilePreamble, ;
file-blocks : [* Block], File = [
] file-type-id : tstr .regexp "C-DNS",
file-preamble : FilePreamble,
file-blocks : [* Block],
FilePreamble = { ]
major-format-version => uint, ; = 1
minor-format-version => uint, ; = 0 ;
? private-version => uint, ; The file preamble.
? configuration => Configuration, ;
? generator-id => tstr, FilePreamble = {
? host-id => tstr, major-format-version => uint .eq 1,
minor-format-version => uint .eq 0,
? private-version => uint,
block-parameters => [+ BlockParameters],
}
major-format-version = 0
minor-format-version = 1
private-version = 2
block-parameters = 3
BlockParameters = {
storage-parameters => StorageParameters,
? collection-parameters => CollectionParameters,
}
storage-parameters = 0
collection-parameters = 1
StorageParameters = {
ticks-per-second => uint,
max-block-items => uint,
storage-hints => StorageHints,
opcodes => [+ uint],
rr-types => [+ uint],
? storage-flags => StorageFlags,
? client-address-prefix-ipv4 => uint,
? client-address-prefix-ipv6 => uint,
? server-address-prefix-ipv4 => uint,
? server-address-prefix-ipv6 => uint,
} }
ticks-per-second = 0
max-block-items = 1
storage-hints = 2
opcodes = 3
rr-types = 4
storage-flags = 5
client-address-prefix-ipv4 = 6
client-address-prefix-ipv6 = 7
server-address-prefix-ipv4 = 8
server-address-prefix-ipv6 = 9
major-format-version = 0 ; A hint indicates if the collection method will output the
minor-format-version = 1 ; item or will ignore the item if present.
private-version = 2
configuration = 3
generator-id = 4
host-id = 5
Configuration = { StorageHints = {
query-response-hints => QueryResponseHints,
query-response-signature-hints => QueryResponseSignatureHints,
other-data-hints => OtherDataHints,
}
query-response-hints = 0
query-response-signature-hints = 1
other-data-hints = 2
QueryResponseHintValues = &(
time-offset : 0,
client-address-index : 1,
client-port : 2,
transaction-id : 3,
qr-signature-index : 4,
client-hoplimit : 5,
response-delay : 6,
query-name-index : 7,
query-size : 8,
response-size : 9,
response-processing-data : 10,
query-question-sections : 11, ; Second & subsequent questions
query-answer-sections : 12,
query-authority-sections : 13,
query-additional-sections : 14,
response-answer-sections : 15,
response-authority-sections : 16,
response-additional-sections : 17,
)
QueryResponseHints = uint .bits QueryResponseHintValues
QueryResponseSignatureHintValues =&(
server-address : 0,
server-port : 1,
qr-transport-flags : 2,
qr-type : 3,
qr-sig-flags : 4,
query-opcode : 5,
dns-flags : 6,
query-rcode : 7,
query-class-type : 8,
query-qdcount : 9,
query-ancount : 10,
query-arcount : 11,
query-nscount : 12,
query-edns-version : 13,
query-udp-size : 14,
query-opt-rdata : 15,
response-rcode : 16,
)
QueryResponseSignatureHints = uint .bits QueryResponseSignatureHintValues
OtherDataHintValues = &(
malformed-messages : 0,
address-event-counts : 1,
)
OtherDataHints = uint .bits OtherDataHintValues
StorageFlagValues = &(
anonymised-data : 0,
sampled-data : 1,
)
StorageFlags = uint .bits StorageFlagValues
CollectionParameters = {
? query-timeout => uint, ? query-timeout => uint,
? skew-timeout => uint, ? skew-timeout => uint,
? snaplen => uint, ? snaplen => uint,
? promisc => uint, ? promisc => uint,
? interfaces => [* tstr], ? interfaces => [+ tstr],
? server-addresses => [* IPAddress], ; Hint for later analysis ? server-addresses => [+ IPAddress], ; Hint for later analysis
? vlan-ids => [* uint], ? vlan-ids => [+ uint],
? filter => tstr, ? filter => tstr,
? query-options => QRCollectionSections, ? generator-id => tstr,
? response-options => QRCollectionSections, ? host-id => tstr,
? accept-rr-types => [* uint],
? ignore-rr-types => [* uint],
? max-block-qr-items => uint,
? collect-malformed => uint,
} }
QRCollectionSectionValues = &(
question : 0, ; Second & subsequent questions
answer : 1,
authority : 2,
additional: 3,
)
QRCollectionSections = uint .bits QRCollectionSectionValues
query-timeout = 0 query-timeout = 0
skew-timeout = 1 skew-timeout = 1
snaplen = 2 snaplen = 2
promisc = 3 promisc = 3
interfaces = 4 interfaces = 4
vlan-ids = 5 server-addresses = 5
filter = 6 vlan-ids = 6
query-options = 7 filter = 7
response-options = 8 generator-id = 8
accept-rr-types = 9 host-id = 9
ignore-rr-types = 10
server-addresses = 11
max-block-qr-items = 12
collect-malformed = 13
Block = { ;
preamble => BlockPreamble, ; Data in the file is stored in Blocks.
? statistics => BlockStatistics, ;
tables => BlockTables, Block = {
queries => [* QueryResponse], block-preamble => BlockPreamble,
? address-event-counts => [* AddressEventCount], ? block-statistics => BlockStatistics, ; Much of this could be derived
? malformed-packet-data => [* MalformedPacket], ? block-tables => BlockTables,
} ? query-responses => [+ QueryResponse],
? address-event-counts => [+ AddressEventCount],
? malformed-messages => [+ MalformedMessage],
}
block-preamble = 0
block-statistics = 1
block-tables = 2
query-responses = 3
address-event-counts = 4
malformed-messages = 5
preamble = 0 ;
statistics = 1 ; The (mandatory) preamble to a block.
tables = 2 ;
queries = 3 BlockPreamble = {
address-event-counts = 4 ? earliest-time => Timestamp,
malformed-packet-data = 5 ? block-parameters-index => uint .default 0,
}
earliest-time = 0
block-parameters-index = 1
BlockPreamble = { ; Ticks are subsecond intervals. The number of ticks in a second is file/block
earliest-time => Timeval ; metadata. Signed and unsigned tick types are defined.
ticks = int
uticks = uint
} Timestamp = [
timestamp-secs : uint,
timestamp-uticks : uticks,
]
earliest-time = 1 ;
; Statistics about the block contents.
;
BlockStatistics = {
? total-messages => uint,
? total-pairs => uint,
? total-unmatched-queries => uint,
? total-unmatched-responses => uint,
? total-malformed-messages => uint,
}
total-messages = 0
total-pairs = 1
total-unmatched-queries = 2
total-unmatched-responses = 3
total-malformed-messages = 4
Timeval = [ ;
seconds : uint, ; Tables of common data referenced from records in a block.
microseconds : uint, ;
? picoseconds : uint, BlockTables = {
] ? ip-address => [+ IPAddress],
? classtype => [+ ClassType],
? name-rdata => [+ bstr], ; Holds both Name RDATA and RDATA
? qr-sig => [+ QueryResponseSignature],
? QuestionTables,
? RRTables,
? malformed-message-data => [+ MalformedMessageData],
}
ip-address = 0
classtype = 1
name-rdata = 2
qr-sig = 3
qlist = 4
qrr = 5
rrlist = 6
rr = 7
malformed-message-data = 8
BlockStatistics = { IPv4Address = bstr .size 4
? total-packets => uint, IPv6Address = bstr .size 16
? total-pairs => uint, IPAddress = IPv4Address / IPv6Address
? unmatched-queries => uint,
? unmatched-responses => uint,
? malformed-packets => uint,
}
total-packets = 0 ClassType = {
total-pairs = 1 type => uint,
unmatched-queries = 2 class => uint,
unmatched-responses = 3 }
malformed-packets = 4 type = 0
class = 1
BlockTables = { QueryResponseSignature = {
ip-address => [* IPAddress], ? server-address-index => uint,
classtype => [* ClassType], ? server-port => uint,
name-rdata => [* bstr], ; Holds both Name RDATA and RDATA ? qr-transport-flags => QueryResponseTransportFlags,
query-sig => [* QuerySignature] ? qr-type => QueryResponseType,
? qlist => [* QuestionList], ? qr-sig-flags => QueryResponseFlags,
? qrr => [* Question], ? query-opcode => uint,
? rrlist => [* RRList], ? qr-dns-flags => DNSFlags,
? rr => [* RR], ? query-rcode => uint,
} ? query-classtype-index => uint,
? query-qd-count => uint,
? query-an-count => uint,
? query-ns-count => uint,
? query-ar-count => uint,
? edns-version => uint,
? udp-buf-size => uint,
? opt-rdata-index => uint,
? response-rcode => uint,
ip-address = 0 }
classtype = 1 server-address-index = 0
name-rdata = 2 server-port = 1
query-sig = 3 qr-transport-flags = 2
qlist = 4 qr-type = 3
qrr = 5 qr-sig-flags = 4
rrlist = 6 query-opcode = 5
rr = 7 qr-dns-flags = 6
query-rcode = 7
query-classtype-index = 8
query-qd-count = 9
query-an-count = 10
query-ns-count = 12
query-ar-count = 12
edns-version = 13
udp-buf-size = 14
opt-rdata-index = 15
response-rcode = 16
QueryResponse = { Transport = &(
time-useconds => uint, ; Time offset from start of block udp : 0,
? time-pseconds => uint, ; in microseconds and picoseconds tcp : 1,
client-address-index => uint, tls : 2,
client-port => uint, dtls : 3,
transaction-id => uint, )
query-signature-index => uint,
? client-hoplimit => uint,
? delay-useconds => int,
? delay-pseconds => int, ; Has same sign as delay-useconds
? query-name-index => uint,
? query-size => uint, ; DNS size of query
? response-size => uint, ; DNS size of response
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
time-useconds = 0 TransportFlagValues = &(
time-pseconds = 1 ip-version : 0, ; 0=IPv4, 1=IPv6
client-address-index = 2 ; Transport value bits 1-4
client-port = 3 ) / (1..4)
transaction-id = 4 TransportFlags = uint .bits TransportFlagValues
query-signature-index = 5
client-hoplimit = 6
delay-useconds = 7
delay-pseconds = 8
query-name-index = 9
query-size = 10
response-size = 11
query-extended = 12
response-extended = 13
ClassType = { QueryResponseTransportFlagValues = &(
type => uint, query-trailingdata : 5,
class => uint, ) / TransportFlagValues
} QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues
type = 0 QueryResponseType = &(
class = 1 stub : 0,
client : 1,
resolver : 2,
auth : 3,
forwarder : 4,
tool : 5,
)
QueryResponseFlagValues = &(
has-query : 0,
has-reponse : 1,
query-has-question : 2,
query-has-opt : 3,
response-has-opt : 4,
response-has-no-question: 5,
)
QueryResponseFlags = uint .bits QueryResponseFlagValues
DNSFlagValues = &( DNSFlagValues = &(
query-cd : 0, query-cd : 0,
query-ad : 1, query-ad : 1,
query-z : 2, query-z : 2,
query-ra : 3, query-ra : 3,
query-rd : 4, query-rd : 4,
query-tc : 5, query-tc : 5,
query-aa : 6, query-aa : 6,
query-do : 7, query-do : 7,
response-cd: 8, response-cd: 8,
response-ad: 9, response-ad: 9,
response-z : 10, response-z : 10,
response-ra: 11, response-ra: 11,
response-rd: 12, response-rd: 12,
response-tc: 13, response-tc: 13,
response-aa: 14, response-aa: 14,
) )
DNSFlags = uint .bits DNSFlagValues DNSFlags = uint .bits DNSFlagValues
QueryResponseFlagValues = &( QuestionTables = (
has-query : 0, qlist => [+ QuestionList],
has-reponse : 1, qrr => [+ Question]
query-has-question : 2, )
query-has-opt : 3,
response-has-opt : 4,
response-has-no-question: 5,
)
QueryResponseFlags = uint .bits QueryResponseFlagValues
TransportFlagValues = &(
tcp : 0,
ipv6 : 1,
query-trailingdata: 2,
)
TransportFlags = uint .bits TransportFlagValues
QuerySignature = {
server-address-index => uint,
server-port => uint,
transport-flags => TransportFlags,
qr-sig-flags => QueryResponseFlags,
? query-opcode => uint,
qr-dns-flags => DNSFlags,
? query-rcode => uint,
? query-classtype-index => uint,
? query-qd-count => uint,
? query-an-count => uint,
? query-ar-count => uint,
? query-ns-count => uint,
? edns-version => uint,
? udp-buf-size => uint,
? opt-rdata-index => uint,
? response-rcode => uint,
}
server-address-index = 0
server-port = 1
transport-flags = 2
qr-sig-flags = 3
query-opcode = 4
qr-dns-flags = 5
query-rcode = 6
query-classtype-index = 7
query-qd-count = 8
query-an-count = 9
query-ar-count = 10
query-ns-count = 11
edns-version = 12
udp-buf-size = 13
opt-rdata-index = 14
response-rcode = 15
QuestionList = [ QuestionList = [+ uint] ; Index of Question
* uint, ; Index of Question
]
Question = { ; Second and subsequent questions Question = { ; Second and subsequent questions
name-index => uint, ; Index to a name in the name-rdata table name-index => uint, ; Index to a name in the name-rdata table
classtype-index => uint, classtype-index => uint,
} }
name-index = 0 name-index = 0
classtype-index = 1 classtype-index = 1
RRList = [ RRTables = (
* uint, ; Index of RR rrlist => [+ RRList],
] rr => [+ RR]
)
RRList = [+ uint] ; Index of RR
RR = { RR = {
name-index => uint, ; Index to a name in the name-rdata table name-index => uint, ; Index to a name in the name-rdata table
classtype-index => uint, classtype-index => uint,
ttl => uint, ttl => uint,
rdata-index => uint, ; Index to RDATA in the name-rdata table rdata-index => uint, ; Index to RDATA in the name-rdata table
} }
; Other map key values already defined above.
ttl = 2 ttl = 2
rdata-index = 3 rdata-index = 3
QueryResponseExtended = { MalformedMessageData = {
? question-index => uint, ; Index of QuestionList ? server-address-index => uint,
? answer-index => uint, ; Index of RRList ? server-port => uint,
? authority-index => uint, ? mm-transport-flags => TransportFlags,
? additional-index => uint, ? mm-payload => bstr,
} }
; Other map key values already defined above.
question-index = 0 mm-transport-flags = 2
answer-index = 1 mm-payload = 3
authority-index = 2
additional-index = 3
AddressEventCount = { ;
ae-type => &AddressEventType, ; A single query/response pair.
? ae-code => uint, ;
ae-address-index => uint, QueryResponse = {
ae-count => uint, ? time-offset => uticks, ; Time offset from start of block
} ? client-address-index => uint,
? client-port => uint,
? transaction-id => uint,
? qr-signature-index => uint,
? client-hoplimit => uint,
? response-delay => ticks,
? query-name-index => uint,
? query-size => uint, ; DNS size of query
? response-size => uint, ; DNS size of response
? response-processing-data => ResponseProcessingData,
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
time-offset = 0
client-address-index = 1
client-port = 2
transaction-id = 3
qr-signature-index = 4
client-hoplimit = 5
response-delay = 6
query-name-index = 7
query-size = 8
response-size = 9
response-processing-data = 10
query-extended = 11
response-extended = 12
ae-type = 0 ResponseProcessingData = {
ae-code = 1 ? bailiwick-index => uint,
ae-address-index = 2 ? processing-flags => ResponseProcessingFlags,
ae-count = 3 }
bailiwick-index = 0
processing-flags = 1
AddressEventType = ( ResponseProcessingFlagValues = &(
tcp-reset : 0, from-cache : 0,
icmp-time-exceeded : 1,
icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5,
) )
ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues
MalformedPacket = { QueryResponseExtended = {
time-useconds => uint, ; Time offset from start of block ? question-index => uint, ; Index of QuestionList
? time-pseconds => uint, ; in microseconds and picoseconds ? answer-index => uint, ; Index of RRList
packet-content => bstr, ; Raw packet contents ? authority-index => uint,
} ? additional-index => uint,
}
question-index = 0
answer-index = 1
authority-index = 2
additional-index = 3
time-useconds = 0 ;
time-pseconds = 1 ; Address event data.
packet-content = 2 ;
AddressEventCount = {
ae-type => &AddressEventType,
? ae-code => uint,
ae-address-index => uint,
ae-count => uint,
}
ae-type = 0
ae-code = 1
ae-address-index = 2
ae-count = 3
IPv4Address = bstr .size 4 AddressEventType = (
IPv6Address = bstr .size 16 tcp-reset : 0,
IPAddress = IPv4Address / IPv6Address icmp-time-exceeded : 1,
icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5,
)
;
; Malformed messages.
;
MalformedMessage = {
? time-offset => uticks, ; Time offset from start of block
? client-address-index => uint,
? client-port => uint,
? message-data-index => uint,
}
; Other map key values already defined above.
message-data-index = 3
Appendix B. DNS Name compression example Appendix B. DNS Name compression example
The basic algorithm, which follows the guidance in [RFC1035], is The basic algorithm, which follows the guidance in [RFC1035], is
simply to collect each name, and the offset in the packet at which it simply to collect each name, and the offset in the packet at which it
starts, during packet construction. As each name is added, it is starts, during packet construction. As each name is added, it is
offered to each of the collected names in order of collection, offered to each of the collected names in order of collection,
starting from the first name. If labels at the end of the name can starting from the first name. If labels at the end of the name can
be replaced with a reference back to part (or all) of the earlier be replaced with a reference back to part (or all) of the earlier
name, and if the uncompressed part of the name is shorter than any name, and if the uncompressed part of the name is shorter than any
skipping to change at page 44, line 30 skipping to change at page 57, line 4
* Code must be generated for a particular data schema to to read * Code must be generated for a particular data schema to to read
and write data using that schema. At the time of writing, the and write data using that schema. At the time of writing, the
Google code generator can currently generate code for encoding Google code generator can currently generate code for encoding
and decoding a schema for C++, Go, Java, Python, Ruby, C#, and decoding a schema for C++, Go, Java, Python, Ruby, C#,
Objective-C, Javascript and PHP. Objective-C, Javascript and PHP.
o CBOR [16]. Defined in [RFC7049], this serialisation format is o CBOR [16]. Defined in [RFC7049], this serialisation format is
comparable to JSON but with a binary representation. It does not comparable to JSON but with a binary representation. It does not
use a pre-defined schema, so data is always stored tagged. use a pre-defined schema, so data is always stored tagged.
However, CBOR data schemas can be described using CDDL However, CBOR data schemas can be described using CDDL
[I-D.greevenbosch-appsawg-cbor-cddl] and tools exist to verify
data files conform to the schema. [I-D.ietf-cbor-cddl] and tools exist to verify data files conform
to the schema.
* CBOR is a simple format, and simple to implement. At the time * CBOR is a simple format, and simple to implement. At the time
of writing, the CBOR website lists implementations for 16 of writing, the CBOR website lists implementations for 16
languages. languages.
Avro and Protocol Buffers both allow storage of untagged data, but Avro and Protocol Buffers both allow storage of untagged data, but
because they rely on the data schema for this, their implementation because they rely on the data schema for this, their implementation
is considerably more complex than CBOR. Using Avro or Protocol is considerably more complex than CBOR. Using Avro or Protocol
Buffers in an unsupported environment would require notably greater Buffers in an unsupported environment would require notably greater
development effort compared to CBOR. development effort compared to CBOR.
skipping to change at page 48, line 21 skipping to change at page 60, line 41
C.5. Conclusions C.5. Conclusions
The above assessments lead us to the choice of a binary format file The above assessments lead us to the choice of a binary format file
using blocking. using blocking.
As noted previously, this draft anticipates that output data will be As noted previously, this draft anticipates that output data will be
subject to compression. There is no compelling case for one subject to compression. There is no compelling case for one
particular binary serialisation format in terms of either final file particular binary serialisation format in terms of either final file
size or machine resources consumed, so the choice must be largely size or machine resources consumed, so the choice must be largely
based on other factors. CBOR was therefore chosen as the binary based on other factors. CBOR was therefore chosen as the binary
serialisation format for the reasons listed in Section 6. serialisation format for the reasons listed in Section 5.
C.6. Block size choice C.6. Block size choice
Given the choice of a CBOR format using blocking, the question arises Given the choice of a CBOR format using blocking, the question arises
of what an appropriate default value for the maximum number of query/ of what an appropriate default value for the maximum number of query/
response pairs in a block should be. This has two components; what response pairs in a block should be. This has two components; what
is the impact on performance of using different block sizes in the is the impact on performance of using different block sizes in the
format file, and what is the impact on the size of the format file format file, and what is the impact on the size of the format file
before and after compression. before and after compression.
skipping to change at page 49, line 25 skipping to change at page 61, line 46
block sizes over 10,000 query/response pairs would seem to offer block sizes over 10,000 query/response pairs would seem to offer
limited improvements. limited improvements.
Authors' Addresses Authors' Addresses
John Dickinson John Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom
Email: jad@sinodun.com Email: jad@sinodun.com
Jim Hague Jim Hague
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom
Email: jim@sinodun.com Email: jim@sinodun.com
Sara Dickinson Sara Dickinson
Sinodun IT Sinodun IT
Magdalen Centre Magdalen Centre
Oxford Science Park Oxford Science Park
Oxford OX4 4GA Oxford OX4 4GA
United Kingdom
Email: sara@sinodun.com Email: sara@sinodun.com
Terry Manderson Terry Manderson
ICANN ICANN
12025 Waterfront Drive 12025 Waterfront Drive
Suite 300 Suite 300
Los Angeles CA 90094-2536 Los Angeles CA 90094-2536
Email: terry.manderson@icann.org Email: terry.manderson@icann.org
John Bond John Bond
ICANN ICANN
 End of changes. 201 change blocks. 
976 lines changed or deleted 1518 lines changed or added

This html diff was produced by rfcdiff 1.46. The latest version is available from http://tools.ietf.org/tools/rfcdiff/