dnsop                                                       J. Dickinson
Internet-Draft                                                  J. Hague
Intended status: Standards Track                            S. Dickinson
Expires: June 9, August 25, 2017                                      Sinodun IT
                                                            T. Manderson
                                                                 J. Bond
                                                                   ICANN
                                                        December 6, 2016
                                                       February 21, 2017

                   C-DNS: A DNS Packet Capture Format
                 draft-ietf-dnsop-dns-capture-format-00
                 draft-ietf-dnsop-dns-capture-format-01

Abstract

   This document describes a data representation for collections of DNS
   messages.  The format is designed for efficient storage and
   transmission of large packet captures of DNS traffic; it attempts to
   minimize the size of such packet capture files but retain the full
   DNS message contents along with the most useful transport meta data. metadata.
   It is intended to assist with the development of DNS traffic
   monitoring applications.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on June 9, August 25, 2017.

Copyright Notice

   Copyright (c) 2016 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Requirements  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Data Collection Use Cases . . . . . . . . . . . . . . . . . .   4   5
   4.  Design Considerations . . . . . . . . . . . . . . . . . . . .   6
   5.  C-DNS conceptual overview  Conceptual Overview . . . . . . . . . . . . . . . . . . . . .   8
   6.  Choice of CBOR  . . . . . . . . . . . . . . . . . . . . . . .   8
   7.  The C-DNS CBOR format  . . . . . . . . . . . . . . . . . . . . . .   8   9
     7.1.  CDDL definition . . . . . . . . . . . . . . . . . . . . .   8   9
     7.2.  Format overview . . . . . . . . . . . . . . . . . . . . .   9
     7.3.  File header contents  . . . . . . . . . . . . . . . . . .   9  10
     7.4.  File preamble contents  . . . . . . . . . . . . . . . . .  10
     7.5.  Configuration contents  . . . . . . . . . . . . . . . . .  10  11
     7.6.  Block contents  . . . . . . . . . . . . . . . . . . . . .  11  12
     7.7.  Block preamble map  . . . . . . . . . . . . . . . . . . .  12  13
     7.8.  Block statistics  . . . . . . . . . . . . . . . . . . . .  12  14
     7.9.  Block table map . . . . . . . . . . . . . . . . . . . . .  13  14
     7.10. IP address table  . . . . . . . . . . . . . . . . . . . .  13  15
     7.11. Class/Type table  . . . . . . . . . . . . . . . . . . . .  13  15
     7.12. Name/RDATA table  . . . . . . . . . . . . . . . . . . . .  14  15
     7.13. Query Signature table . . . . . . . . . . . . . . . . . .  14  16
     7.14. Question table  . . . . . . . . . . . . . . . . . . . . .  16  18
     7.15. Resource Record (RR) table  . . . . . . . . . . . . . . .  17  18
     7.16. Question list table . . . . . . . . . . . . . . . . . . .  17  19
     7.17. Resource Record list table  . . . . . . . . . . . . . . .  17  19
     7.18. Query/Response data . . . . . . . . . . . . . . . . . . .  18  19
     7.19. Address Event counts  . . . . . . . . . . . . . . . . . .  20  22
     7.20. Malformed packet records  . . . . . . . . . . . . . . . .  23
   8.  Malformed Packets . . . . . . . . . . . . . . . . . . . . . .  23
   9.  C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . .  21
     8.1.  25
     9.1.  Name Compression  . . . . . . . . . . . . . . . . . . . .  22
   9.  26
   10. Data Collection . . . . . . . . . . . . . . . . . . . . . . .  23
     9.1.  26
     10.1.  Matching algorithm . . . . . . . . . . . . . . . . . . .  23
     9.2.  27
     10.2.  Message identifiers  . . . . . . . . . . . . . . . . . . .  24
       9.2.1.  27
       10.2.1.  Primary ID (required)  . . . . . . . . . . . . . . . .  24
       9.2.2.  27
       10.2.2.  Secondary ID (optional)  . . . . . . . . . . . . . . .  24
     9.3.  28
     10.3.  Algorithm Parameters . . . . . . . . . . . . . . . . . .  24
     9.4.  28
     10.4.  Algorithm Requirements . . . . . . . . . . . . . . . . .  24
     9.5.  28
     10.5.  Algorithm Limitations  . . . . . . . . . . . . . . . . . .  25
     9.6.  28
     10.6.  Workspace  . . . . . . . . . . . . . . . . . . . . . . . .  25
     9.7.  28
     10.7.  Output . . . . . . . . . . . . . . . . . . . . . . . . .  25
     9.8.  28
     10.8.  Post Processing  . . . . . . . . . . . . . . . . . . . . .  25

   10.  29
   11. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  25
   11.  29
   12. Security Considerations . . . . . . . . . . . . . . . . . . .  25
   12.  29
   13. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  25
   13.  29
   14. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . .  26
   14.  29
   15. References  . . . . . . . . . . . . . . . . . . . . . . . . .  26
     14.1.  30
     15.1.  Normative References . . . . . . . . . . . . . . . . . .  26
     14.2.  30
     15.2.  Informative References . . . . . . . . . . . . . . . . .  27
     14.3.  31
     15.3.  URIs . . . . . . . . . . . . . . . . . . . . . . . . . .  28  32
   Appendix A.  CDDL . . . . . . . . . . . . . . . . . . . . . . . .  28  33
   Appendix B.  DNS Name compression example . . . . . . . . . . . .  33  39
     B.1.  NSD compression algorithm . . . . . . . . . . . . . . . .  34  40
     B.2.  Knot Authoritative compression algorithm  . . . . . . . .  35  41
     B.3.  Observed differences  . . . . . . . . . . . . . . . . . .  35  41
   Appendix C.  Comparison of Binary Formats . . . . . . . . . . . .  35
   Appendix D.  Sample data on the C-DNS format  41
     C.1.  Comparison with full PCAP files . . . . . . . . . .  36
     D.1.  Comparison to full PCAPS . . .  44
     C.2.  Simple versus block coding  . . . . . . . . . . . . . .  36
     D.2.  Block size choices .  45
     C.3.  Binary versus text formats  . . . . . . . . . . . . . . .  45
     C.4.  Performance . . . . . . . . . . . . . . . . . . .  36
     D.3.  Blocking vs more simple output . . . .  45
     C.5.  Conclusions . . . . . . . . .  36 . . . . . . . . . . . . . .  46
     C.6.  Block size choice . . . . . . . . . . . . . . . . . . . .  46
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  37  47

1.  Introduction

   There has long been a need to collect DNS queries and responses on
   authoritative and recursive name servers for monitoring and analysis.
   This data is used in a number of ways including traffic monitoring,
   analyzing network attacks and DITL [ditl]. "day in the life" (DITL) [ditl]
   analysis.

   A wide variety of tools already exist to that facilitate the collection
   of DNS traffic data. data, such as DSC [dsc], packetq [packetq], dnscap
   [dnscap] and dnstap [dnstap].  However, there is no standard exchange
   format for large DNS packet captures and captures.  The PCAP [pcap] or PCAP-NG
   [pcapng] formats are typically used in practice.  Such practice for packet captures,
   but these file formats can contain much a great deal of additional
   information that is not directly pertinent to DNS traffic analysis
   which
   and thus unnecessarily increases the capture file size.

   There has also been work on using other text based formats to describe DNS
   packets such as [I-D.daley-dnsxml], [I-D.hoffman-dns-in-json] [I-D.hoffman-dns-in-json], but
   these are largely aimed at producing convenient representations of
   single messages.

   Many DNS operators may receive 100's hundreds of thousands of queries per
   second on a single name server instance so a mechanism to minimize
   the storage size (and therefore upload overhead) of the data
   collected is highly desirable.

   This documents

   The format described in this document, C-DNS (Compacted-DNS),
   focusses on the problem of capturing and storing large packet capture
   files of DNS traffic. with the following goals in mind:

   o  Minimize the file size for storage and transmission

   o  Minimizing the overhead of producing the packet capture file and
      the cost of any further (general purpose) compression of the file
      to minimise the size

   This document contains contains:

   o  A discussion of the some common use cases in which such DNS data
      is collected.  See collected Section 3. 3

   o  A discussion of the major design considerations in developing an
      efficient data representation for collections of DNS messages.
      See messages
      Section 4. 4

   o  A definition conceptual overview of a the C-DNS format Section 5

   o  A description of why CBOR [RFC7049] representation of a collection was chosen for this format
      Section 6

   o  The definition of
      DNS messages.  This will be referred to as the C-DNS format
      (Compacted-DNS).  See for the collection of DNS
      messages Section 7.

   o  Notes on converting C-DNS back data to PCAP format.  See format Section 8. 9

   o  Some high level implementation considerations for applications
      designed to produce C-DNS, e.g. a query response matching
      algorithm.  See C-DNS Section 9. 10

2.  Requirements  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

   The parts of DNS messages are named as they are in [RFC1035].  In
   specific, the DNS message has five sections: Header, Question,
   Answer, Authority, and Additional.

   Pairs of DNS messages are called a Query and a Response.

3.  Data Collection Use Cases

   In an ideal world world, it would be optimal to collect full packet
   captures of all packets going in or out of a name server.  However,
   there are several design choices or other limitations that are common
   to many DNS installations and operators.

   o  Servers  DNS servers are hosted in a variety of situations

      *  Operator self hosted  Self-hosted servers

      *  Third party hosting (including multiple third parties)

      *  Third party hardware (including multiple third parties)

   o  Data is collected under different conditions

      *  On well provisioned well-provisioned servers running in a steady state. state

      *  On heavily loaded servers

      *  On virtualized servers

      *  On servers that are under DoS attack

      *  On servers that are unwitting intermediaries in DoS attacks

   o  Traffic can be collected via a variety of mechanisms

      *  On the same hardware as the name server itself

      *  Using a network tap on an adjacent host to listen in from another server to DNS
         traffic

      *  Using port mirroring to listen in from another server host

   o  The capabilities of data collection (and upload) networks vary

      *  Out-of-band networks with the same capacity as the in-band
         network

      *  Out-of-band networks with less capacity than the in-band
         network

      *  Everything being on the in-band network

   Clearly,

   Thus, there is a wide range of use cases from very limited data
   collection environments (third party hardware, servers that are under
   attack, packet capture on the name server itself and no out-of-band
   network) to 'limitless' "limitless" environments (self hosted, well provisioned
   servers, using a network tap or port mirroring with an out-of-band
   networks with the same capacity as the in-band network).  In the
   former, it is unfeasible infeasible to reliably collect full PCAPS packet captures,
   especially if the server is under attack.  In the latter case,
   collection of full
   PCAPs packet captures may be reasonable.

   As a result of these restrictions restrictions, the C-DNS data format discussed below was designed
   with the most limited use case in mind such that that:

   o  Data  data collection will occur on the same hardware as the name server
      itself.
      itself

   o  Collected  collected data will be stored on the same hardware as the name
      server itself, at least temporarily. temporarily

   o  Collected  collected data being returned to some central analysis system will
      use the same network interface as the DNS queries and responses. responses

   o  There are  there can be multiple third party servers involved.

   and therefore involved

   Because of these considerations, a major factor in the design of the
   format is minimal storage size of the capture files is a major
   factor. files.

   Another significant consideration for any application that records
   DNS traffic is that the running of the name server software and the
   transmission of DNS queries and responses is are the most important job jobs
   of a name server. server; capturing data is not.  Any data collection system
   co-located with the name server will need needs to be intelligent enough to
   carefully manage its CPU, disk, memory and network utilization.  Hence this use case benefits from  This
   leads to designing a format that has requires a relatively low overhead
   to produce and minimizes the requirement for further potentially
   costly compression.

   However, it was also essential that interoperability with less
   restricted infrastructure was maintained.  In particular particular, it is
   highly desirable that the resulting collection format should facilitate the
   re-creation re-
   creation of common formats (such as PCAPs) PCAP) that are as close to the
   original as is realistic given the restrictions above.

4.  Design Considerations

   This section presents some of the major design considerations used in
   the development of the C-DNS format.

   o

   1.  The basic unit of data is a combined DNS Query and the associated
       Response (a 'Q/R "Q/R data item'). item").  The same structure will be used
       for unmatched queries Queries and responses. Responses.  Queries without responses Responses
       will be captured omitting the Response response data.  Responses without
       queries will be captured omitting the Query data (but using the Query
       Question section from the Response, response, if present, as an identifying
       QNAME).

       *  Rationale: A Query and Response represents the basic level of
          a clients interaction with the server.  Also, combining the
          Query and Response into one item lowers often reduces storage
          requirements due to commonality in the data in most cases.

   o of the two
          messages.

   2.  Each Q/R data item will comprise a default Q/R data description
       and a set of optional sections.  Inclusion of optional sections
       shall be configurable.

       *  Rationale: Different users will have different requirements
          for data to be available for analysis.  Users with minimal
          requirements should not have to pay the cost of recording full
          data, however this will limit the ability to reconstruct PCAPS.
          packet captures.  For example example, omitting the
   Resource Records resource records
          from a Response will reduce the files size, and in principle
          responses can be synthesized if there is enough context.

   o

   3.  Multiple Q/R data items will be collected into blocks in the
       format.  Common data in a block will be abstracted and referenced
       from individual Q/R data items by indexing.  The maximum number
       of Q/R data items in a block will be configurable.

       *  Rationale: This blocking and indexing provides a significant
          reduction in the volume of file data generated.  Whilst introducing
   complexity  Although this
          introduces complexity, it provides compression of the data
          that makes use of knowledge of the DNS packet structure.

   [TODO: Further discussion on commonality between DNS packets e.g.

   o  common query signatures

   o  for the authoritative case there are a finite set of valid
      responses and much commonality in NXDOMAIN responses]

       *  It is anticipated that the files produced will can be subject to
          further compression using general purpose compression tools.
          Measurements show that blocking significantly reduces the CPU
          required to perform such strong compression.  See
          Appendix D.

   o  Meta-data C.2.

       *  [TODO: Further discussion of commonality between DNS packets
          e.g. common query signatures, a finite set of valid responses
          from authoritatives]

   4.  Metadata about other packets received should also can optionally be included
       in each block.  For example example, counts of malformed DNS packets and non-
      DNS
       non-DNS packets (e.g.  ICMP, TCP resets) sent to the server are may
       be of interest.

   It should

   5.  The wire format content of malformed DNS packets can optionally
       be noted that any recorded.

       *  Rationale: Any structured capture format that does not capture
          the DNS payload byte for byte will likely be limited to some extent
          in that it cannot represent 'malformed' "malformed" DNS packets. packets (see
          Section 8).  Only those packets that can be transformed
          reasonably into the structured format can be represented by it.  So if
          the format.  However this can result in rather misleading
          statistics.  For example, a query is malformed this query which cannot be
          represented in the C-DNS format will lead to the (well formed)
          DNS responses with error code FORMERR appearing as
          'unmatched'.

   [TODO: Need further discussion of well-formed vs malformed packets
   and how name servers view this definition.]

   [TODO: Need  Therefore it can greatly aid downstream analysis
          to develop optional representation have the wire format of the malformed DNS packets
   within CBOR and what this means for packet matching.  This may
   influence which fields are optional available
          directly in the rest of the
   representation.]

   Packets such as those described above can be separately recorded in a
   PCAP file for later analysis.

5. C-DNS conceptual overview file.

5.  Conceptual Overview

   The following figures show purely schematic representations of the
   C-DNS format to convey the high-level structure of the C-DNS format.
   Section 7 provides a detailed discussion of the CBOR representation
   and individual elements.

   Figure showing the C-DNS format (PNG) [1]

   Figure showing the C-DNS format (SVG) [2]

   Figure showing the Q/R data item and Block tables format (PNG) [3]

   Figure showing the Q/R data item and Block tables format (SVG) [4]

6.  Choice of CBOR

   This document presents a detailed format description using CBOR, the
   Concise Binary Object Representation defined in [RFC7049].

   The choice of CBOR was made taking a number of factors into account.

   o  CBOR is a binary representation, and so thus is economical in storage
      space.

   o  Other similar binary representations were investigated, and whilst all had
      attractive features, none had a significant advantage over CBOR.
      See Appendix C and Appendix D - for some discussion of this.

   o  CBOR is an IETF Standard standard and familiar to IETF participants, and
      being participants.  It is
      based on the successful JSON text format, now-common ideas of lists and objects, and thus
      requires very little familiarization for those in the wider
      industry.

   o  CBOR is a simple format, and can easily be implemented from
      scratch if necessary.  More complex formats require library
      support which may present problems on unusual platforms.

   o  CBOR can also be easily converted to text formats such as JSON
      ([RFC7159]) for debugging and other human inspection requirements.

   o  CBOR data schemas can be described using CDDL
      [I-D.greevenbosch-appsawg-cbor-cddl].

7.  The C-DNS CBOR format

7.1.  CDDL definition

   The CDDL definition for the C-DNS format is given in Appendix A.

7.2.  Format overview

   A C-DNS file begins with a file header containing a file type
   identifier and a preamble.  The preamble contains information on the
   collection settings.

   This

   The file header is followed by a series of data blocks.

   A block consists of a block header, containing various tables of
   common data, and some statistics for the traffic received over the
   block.  The block header is then followed by a list of the Q/R pairs data
   items detailing the queries and responses received during processing
   of the block. block input.  The list of Q/R pairs data items is in turn followed
   by a list of per-client counts of particular IP events that occurred
   during collection of the block data.

   The exact nature of the DNS data will affect what block size is the
   best fit, however sample data for a root server indicated that block
   sizes in the low 1000's up to 10,000 Q/R data items give good results.  See
   Appendix D.2 C.6 for more details.

   If no field type is specified specified, then the field is unsigned.

   In the following

   o  For all quantities that contain bit flags, bit 0 indicates the least
   significant bit.

   o  Items  An item described as indexes are an index is the index of the
   Q/R data item in the referenced table.  Indexes are 1-based.  An
   index value of 0 is reserved to mean not present. "not present".

7.3.  File header contents

   The file header contains the following:

   +-------------+---------------+-------------------------------------+

   +---------------+---------------+-----------------------------------+
   | Field         | Type          | Description                       |
   +-------------+---------------+-------------------------------------+
   +---------------+---------------+-----------------------------------+
   | File type file-type-id  | Text string   | String "C-DNS" identifying the file type    |
   | ID               |               | file type.                        |
   |               |               |                                   |
   | File file-preamble | Map of items  | Collection information for the    |
   | preamble               |               | whole file.                       |
   |               |               |                                   |
   | File Blocks file-blocks   | Array of      | The data blocks blocks.                  |
   |               | Blocks        |                                   |
   +-------------+---------------+-------------------------------------+
   +---------------+---------------+-----------------------------------+

7.4.  File preamble contents

   The file preamble contains the following:

   +---------------+----------+----------------------------------------+

   +----------------------+----------+---------------------------------+
   | Field                | Type     | Description                     |
   +---------------+----------+----------------------------------------+
   +----------------------+----------+---------------------------------+
   | Format major-format-version | Unsigned | Indicates version of format used in Unsigned integer '1'. The major |
   |                      |          | version of format used in file. |
   |                      |          |                                 |
   | minor-format-version | Unsigned | Unsigned integer '0'. The minor |
   |                      |          | version of format used in file. |
   |                      |          |                                 |
   | Configuration private-version      | Unsigned | Version indicator available for |
   |                      |          | private use by applications.    |
   |                      |          | Optional.                       |
   |                      |          |                                 |
   | configuration        | Map of   | The collection configuration.   |
   |                      | items    | Optional.                       |
   |                      |          |                                 |
   | Generator ID generator-id         | Text     | String identifying the collection          |
   |                      | string   | collection program. Optional.   |
   |                      |          |                                 |
   | Host ID host-id              | Text     | String identifying the collecting          |
   |                      | string   | collecting host. Blank Empty if       |
   |                      |          | converting an existing packet   |
   |                      |          | PCAP capture file. Optional.         |
   +---------------+----------+----------------------------------------+
   +----------------------+----------+---------------------------------+

7.5.  Configuration contents

   The collection configuration contains the following items.  All are
   optional.

   +-------------+----------+------------------------------------------+

   +--------------------+----------+-----------------------------------+
   | Field              | Type     | Description                       |
   +-------------+----------+------------------------------------------+
   +--------------------+----------+-----------------------------------+
   | Query query-timeout      | Unsigned | To be matched with a query, a response     |
   | timeout                    |          | response must arrive within this number of  |
   |                    |          | number of seconds.                |
   |                    |          |                                   |
   | Skew skew-timeout       | Unsigned | The network stack may report a response    |
   | timeout                    |          | response before the corresponding query. A |
   |                    |          | query. A response is not          |
   |                    |          | considered to be missing a query  |
   |                    |          | a query until after this many micro-      |
   |                    |          | seconds.                          |
   |                    |          |                                   |
   | Snap length snaplen            | Unsigned | Collect up to this many bytes per |
   |                    |          | packet.                           |
   |                    |          |                                   |
   | Promiscuous promisc            | Unsigned | 1 if promiscuous mode was enabled on the |
   | mode                    |          | on the interface, 0 otherwise.    |
   |                    |          |                                   |
   | Interfaces interfaces         | Array of | Identifiers of the interfaces used for     |
   |                    | text     | used for collection.              |
   |                    | strings  |                                   |
   |                    |          |                                   |
   | VLAN IDs server-addresses   | Array of | Server collection IP addresses.   |
   |                    | byte     | Hint for downstream analysers;    |
   |                    | strings  | does not affect collection.       |
   |                    |          |                                   |
   | vlan-ids           | Array of | Identifiers of VLANs selected for |
   |                    | unsigned | collection.                       |
   |                    |          |                                   |
   | Filter filter             | Text     | "tcpdump" 'tcpdump' [pcap] style filter for input. |
   |                    | string   | input.                            |
   |                    |          |                                   |
   | Query query-options      | Unsigned | Bit flags indicating sections in Query  |
   | collection                    |          | Query packets to be collected.    |
   | options     |          |                                          |
   |                    |          | Bit 0. Collect second and subsequent         |
   |                    |          | subsequent question sections.     |
   |                    |          | Bit 1. Collect Answer sections.   |
   |                    |          | Bit 2. Collect Authority          |
   |                    |          | sections.                         |
   |                    |          | Bit 3. Collection Additional      |
   |                    |          | sections.                         |
   |                    |          |                                   |
   | Response response-options   | Unsigned | Bit flags indicating sections in  |
   | collection                    |          | Response packets to be collected. |
   | options     |          |                                          |
   |                    |          | Bit 0. Collect second and subsequent         |
   |                    |          | subsequent question sections.     |
   |                    |          | Bit 1. Collect Answer sections.   |
   |                    |          | Bit 2. Collect Authority          |
   |                    |          | sections.                         |
   |                    |          | Bit 3. Collection Additional      |
   |                    |          | sections.                         |
   |                    |          |                                   |
   | Accept RR accept-rr-types    | Array of | A set of RR type names [rrtypes]. If not |
   | types                    | text     | If not empty, only the nominated RR types are  |
   |                    | strings  | RR types are collected.           |
   |                    |          |                                   |
   | Ignore RR ignore-rr-types    | Array of | A set of RR type names [rrtypes]. If not |
   | types                    | text     | If not empty, all RR types are collected except    |
   |                    | strings  | collected except those listed. If |
   |                    |          | present, this item must be empty  |
   |                    |          | be empty if a non-empty list of Accept RR  |
   |                    |          | RR types is present.                 |
   +-------------+----------+------------------------------------------+

7.6.  Block contents

   Each block contains the following:

   +-------------+------------------+----------------------------------+
   | Field                    | Type          | Description                                   |
   +-------------+------------------+----------------------------------+
   | Block max-block-qr-items | Map Unsigned | Maximum number of Q/R data items  | Overall information for the
   |                    | preamble    |          | in a block.                       |
   |                    |          |                                   |
   | Block       | Map of collect-malformed  | Statistics about the block. Unsigned | 1 if malformed packet contents    | statistics
   | statistics                    |          | are collected, 0 otherwise.       |
   +--------------------+----------+-----------------------------------+

7.6.  Block contents

   Each block contains the following:

   +-----------------------+--------------+----------------------------+
   | Field                 | Type         | Description                |
   +-----------------------+--------------+----------------------------+
   | preamble              | Map of items | Overall information for    |
   |                       |              | the block.                 | Block
   |                       |              |                            |
   | statistics            | Map of       | Statistics about the       |
   |                       | statistics   | block. Optional.           |
   |                       |              |                            |
   | tables                | Map of       | The tables containing data |
   | tables                       | tables       | referenced by individual Q/R   |
   |                       |              | entries. Q/R data items.            |
   |                       |              |                            |
   | Q/Rs queries               | Array of Q/Rs Q/R | Details of individual Q/R pairs.  |
   |                       | data items   | data items.                |
   |                       | Address              |                            |
   | address-event-counts  | Array of Address     | Per client counts of ICMP  |
   | Event                       | Event counts Address      | messages and TCP resets.   |
   | Counts                       | Event counts | Optional.                  |
   |                       |              |                            |
   | malformed-packet-data | Array of     | Wire contents of malformed |
   |                       | malformed    | packets. Optional.         |
   |                       | packets      |                            |
   +-------------+------------------+----------------------------------+
   +-----------------------+--------------+----------------------------+

7.7.  Block preamble map

   The block preamble map contains overall information for the block.

   +-----------+----------+--------------------------------------------+

   +---------------+----------+----------------------------------------+
   | Field         | Type     | Description                            |
   +-----------+----------+--------------------------------------------+
   +---------------+----------+----------------------------------------+
   | Timestamp earliest-time | Array of | A timestamp for the earliest record in the |
   |               | unsigned | the block. The timestamp is specified as a  |
   |               |          | as a CBOR array with two or three      |
   |               |          | elements. The first two elements as in Posix are   |
   |               |          | as in Posix struct timeval. The first element is an  |
   |               |          | element is an unsigned integer time_t  |
   |               |          | and the second is an unsigned integer  |
   |               |          | number of microseconds. The third, if  |
   |               |          | present, is an unsigned integer number of |
   |               |          | microseconds. of picoseconds. The latter is microsecond and    |
   |               |          | picosecond items always have a value   |
   |               |          | between 0 and 999,999.                 |
   +-----------+----------+--------------------------------------------+

   [TODO: Extend to support pico/nano.  Also do this for Time offset and
   Response delay]
   +---------------+----------+----------------------------------------+

7.8.  Block statistics

   [TODO: Add block statistics]

7.9.  Block table map

   The block table map statistics section contains some basic statistical
   information about the block tables.  Each element, or
   table, is an array.  The following tables detail block.  All are optional.

   +---------------------+----------+----------------------------------+
   | Field               | Type     | Description                      |
   +---------------------+----------+----------------------------------+
   | total-packets       | Unsigned | Total number of packets          |
   |                     |          | processed from the contents input traffic |
   |                     |          | stream during collection of each
   block table.

   The Present column in the following tables indicates  |
   |                     |          | block data.                      |
   | total-pairs         | Unsigned | Total number of Q/R data items   |
   |                     |          | in the block.                    |
   | unmatched-queries   | Unsigned | Number of unmatched queries in   |
   |                     |          | the block.                       |
   | unmatched-responses | Unsigned | Number of unmatched responses in |
   |                     |          | the block.                       |
   | malformed-packets   | Unsigned | Number of malformed packets      |
   |                     |          | found in input for the block.    |
   +---------------------+----------+----------------------------------+

   Implementations may choose to add additional implementation-specific
   fields to the statistics.

7.9.  Block table map

   The block table map contains the block tables.  Each element, or
   table, is an array.  The following tables detail the contents of each
   block table.

   The Present column in the following tables indicates the
   circumstances when an optional field will be present.  A Q/R pair data
   item may be:

   o  A Query plus a Response.

   o  A Query without a Response.

   o  A Response without a Query.

   Also:

   o  A Query and/or a Response may contain an OPT section.

   o  A Question may or may not be present.  If the Query is available,
      the Question section of the Query is used.  If no Query is
      available, the Question section of the Response is used.  Unless
      otherwise noted, a Question refers to the first Question in the
      Question section.

   So, for example, a field listed with a Present value of QUERY is
   present whenever the Q/R pair data item contains a Query.  If the pair
   contains a Response only, the field will not be present.

7.10.  IP address table

   This

   The table "ip-address" holds all client and server IP addresses in
   the block.  Each item in the table is a single IP address.

   +---------+--------+------------------------------------------------+

   +------------+--------+---------------------------------------------+
   | Field      | Type   | Description                                 |
   +---------+--------+------------------------------------------------+
   +------------+--------+---------------------------------------------+
   | Address ip-address | Byte   | The IP address, in network byte order. The  |
   |            | string | string is 4 bytes long for an IPv4 address, 16 |
   |            |        | 16 bytes long for an IPv6 address.          |
   +---------+--------+------------------------------------------------+
   +------------+--------+---------------------------------------------+

7.11.  Class/Type table

   This

   The table "classtype" holds pairs of RR CLASS and TYPE values.  Each
   item in the table is a CBOR map.

                         +-------+--------------+
                         | Field | Description  |
                         +-------+--------------+
                         | Class type  | CLASS TYPE value.  |
                         |       |              |
                         | Type class | TYPE CLASS value. |
                         +-------+--------------+

   [TODO: Can this be optimized?  Should a class of IN be inferred if
   not present?]

7.12.  Name/RDATA table

   This

   The table "name-rdata" holds the contents of all NAME or RDATA items
   in the block.  Each item in the table is the content of a single NAME
   or RDATA.

   +-------+--------+--------------------------------------------------+

   +------------+--------+---------------------------------------------+
   | Field      | Type   | Description                                 |
   +-------+--------+--------------------------------------------------+
   +------------+--------+---------------------------------------------+
   | Data name-rdata | Byte   | The NAME or RDATA contents. NAMEs, and labels      |
   |            | string | labels within RDATA contents, are in uncompressed label        |
   |            |        | uncompressed label format.                  |
   +-------+--------+--------------------------------------------------+
   +------------+--------+---------------------------------------------+

7.13.  Query Signature table

   This

   The table "query-sig" holds elements of the Q/R data item that are
   often common to between different multiple individual Q/R records. data items.  Each item
   in the table is a CBOR map.  Each item in the map has an unsigned
   value and an unsigned key.

   The following abbreviations are used in the Present (P) column

   o  Q = QUERY

   o  A = Always

   o  QT = QUESTION

   o  QO = QUERY, OPT

   o  QR = QUERY & RESPONSE

   o  R = RESPONSE

   +------------+----+-------------------------------------------------+

   +-----------------------+----+--------------------------------------+
   | Field                 | P  | Description                          |
   +------------+----+-------------------------------------------------+
   +-----------------------+----+--------------------------------------+
   | Server server-address-index  | A  | The index in the IP address table of the server |
   | address                       |    | the server IP address.               |
   |                       |    |                                      |
   | Server server-port           | A  | The server port.                     |
   | port       |    |                                                 |                       |    |                                      |
   |
   | Transport transport-flags       | A  | Bit flags describing the protocol used to transport   |
   | flags                       |    | used to service the query. Bit 0 is the least  |
   |                       |    | the least significant bit.           |
   |                       |    | Bit 0. Transport type. 0 = UDP, 1 =  |
   |                       |    | TCP.                                 |
   |                       |    | Bit 1. IP type. 0 = IPv4, 1 = IPv6.  |
   |                       |    | Bit 2. Trailing bytes in query       |
   | Q/R                       | A    | Bit flags indicating information present payload. The DNS query message in    |
   | signature                       |    | this Q/R pair. Bit 0 is the least significant UDP payload was followed by some |
   | flags                       |    | bit. additional bytes, which were         |
   |                       |    | Bit 0. 1 if a Query is present. discarded.                           |
   |                       |    | Bit 1. 1 if a Response is present.                                      |
   | qr-sig-flags          | A  | Bit 2. 1 if flags indicating information     |
   |                       |    | present in this Q/R data item. Bit 0 |
   |                       |    | is the least significant bit.        |
   |                       |    | Bit 0. 1 if a Query is present.      |
   |                       |    | Bit 1. 1 if a Response is present.   |
   |                       |    | Bit 2. 1 if one or more Question is  |
   |                       |    | present.                             |
   |                       |    | Bit 3. 1 if a Query is present and it has an   |
   |                       |    | it has an OPT Resource Record.       |
   |                       |    | Bit 4. 1 if a Response is present and it has an    |
   |                       |    | and it has an OPT Resource Record.   |
   |                       |    | Bit 5. 1 if a Response is present but has no    |
   |                       |    | but has no Question.                 |
   |                       |    |                                      |
   | Query query-opcode          | Q  | Query OPCODE.                                   |
   | OPCODE     |    | Optional.              |
   |                       |    |                                      |
   | Q/R DNS qr-dns-flags          | A  | Bit flags with values from the Query and |
   | flags                       |    | and Response DNS flags. Bit 0 is the least |
   |                       |    | least significant bit. Flag values   |
   |                       |    | are 0 if the Query or Response is    |
   |                       |    | or Response is not present.                         |
   |                       |    | Bit 0. Query Checking Disabled (CD) flag. (CD). |
   |                       |    | Bit 1. Query Authenticated Data (AD) flag.      |
   |                       |    | (AD).                                |
   |                       |    | Bit 2. Query reserved (Z) flag. (Z).           |
   |                       |    | Bit 3. Query Recursion Available (RA) flag.     |
   |                       |    | (RA).                                |
   |                       |    | Bit 4. Query Recursion Desired (RD) flag. (RD). |
   |                       |    | Bit 5. Query TrunCation (TC) flag. (TC).        |
   |                       |    | Bit 6. Query Authoritative Answer (AA) flag.    |
   |                       |    | (AA).                                |
   |                       |    | Bit 7. Query DNSSEC answer OK (D0) flag. (DO).  |
   |                       |    | Bit 8. Response Checking Disabled (CD) flag.    |
   |                       |    | (CD).                                |
   |                       |    | Bit 9. Response Authenticated Data (AD) flag.   |
   |                       |    | (AD).                                |
   |                       |    | Bit 10. Response reserved (Z) flag. (Z).       |
   |                       |    | Bit 11. Response Recursion Available (RA) flag. |
   |                       |    | (RA).                                |
   |                       |    | Bit 12. Response Recursion Desired (RD) flag.   |
   |                       |    | (RD).                                |
   |                       |    | Bit 13. Response TrunCation (TC) flag. (TC).    |
   |                       |    | Bit 14. Response Authoritative Answer (AA)       |
   |                       |    | flag. Answer (AA).                         |
   |                       |    |                                      |
   | Query query-rcode           | Q  | Query RCODE. If the Query contains OPT, this   |
   | RCODE                       |    | OPT, this value incorporates any     |
   |                       |    | EXTENDED_RCODE_VALUE. Optional.      |
   |                       |    |                                      |
   | Question query-classtype-index | QT | The index in the Class/Type table of the CLASS |
   | Class/Type                       |    | the CLASS and TYPE of the first      |
   |                       |    | Question. Optional.                  |
   |                       |    |                                      |
   | Question query-qd-count        | QT | The QDCOUNT in the Query, or Response if no         |
   | QDCOUNT                       |    | Response if no Query present.        |
   |                       |    | Optional.                            |
   | Query                       | Q    | Query ANCOUNT.                                      |
   | ANCOUNT query-an-count        | Q  | Query ANCOUNT. Optional.             |
   |                       |    |                                      |
   | Query query-ar-count        | Q  | Query ARCOUNT.                                  |
   | ARCOUNT    |    | Optional.             |
   |                       |    |                                      |
   | Query query-ns-count        | Q  | Query NSCOUNT.                                  |
   | NSCOUNT    |    | Optional.             |
   |                       |    |                                      |
   | Query EDNS edns-version          | QO | The Query EDNS version.                         |
   | version    |    | Optional.    |
   |                       |    |                                      |
   | EDNS UDP udp-buf-size          | QO | The Query EDNS sender's UDO UDP payload size  |
   | size                       |    | size. Optional.                      |
   |                       |    |                                      |
   | Query OPT opt-rdata-index       | QO | The index in the NAME/RDATA table of the OPT |
   | RDATA                       |    | the OPT RDATA. Optional.             |
   |                       |    |                                      |
   | Response response-rcode        | R  | Response RCODE. If the Response contains OPT,      |
   | RCODE                       |    | contains OPT, this value             |
   |                       |    | incorporates any                     |
   |                       |    | EXTENDED_RCODE_VALUE. Optional.      |
   +------------+----+-------------------------------------------------+
   +-----------------------+----+--------------------------------------+

7.14.  Question table

   This

   The table "qrr" holds details on individual Questions in a Question
   section.  Each item in the table is a CBOR map containing a single
   Question.  Each item in the map has an unsigned value and an unsigned
   key.  This data is optionally collected.

   +------------+------------------------------------------------------+

   +-----------------+-------------------------------------------------+
   | Field           | Description                                     |
   +------------+------------------------------------------------------+
   +-----------------+-------------------------------------------------+
   | QNAME name-index      | The index in the NAME/RDATA table of the QNAME. |
   |                 |                                                 |
   | Class/Type classtype-index | The index in the Class/Type table of the CLASS and  |
   |                 | and TYPE of the Question.                       |
   +------------+------------------------------------------------------+
   +-----------------+-------------------------------------------------+

7.15.  Resource Record (RR) table

   This

   The table "rr" holds details on individual Resource Records in RR
   sections.  Each item in the table is a CBOR map containing a single
   Resource Record.  This data is optionally collected.

   +------------+------------------------------------------------------+

   +-----------------+-------------------------------------------------+
   | Field           | Description                                     |
   +------------+------------------------------------------------------+
   +-----------------+-------------------------------------------------+
   | NAME name-index      | The index in the NAME/RDATA table of the NAME.  |
   |                 |                                                 |
   | Class/Type classtype-index | The index in the Class/Type table of the CLASS and  |
   |                 | and TYPE of the RR.                             |
   |                 |                                                 |
   | TTL ttl             | The RR Time to Live.                            |
   |                 |                                                 |
   | RDATA rdata-index     | The index in the NAME/RDATA table of the RR     |
   |                 | RDATA.                                          |
   +------------+------------------------------------------------------+
   +-----------------+-------------------------------------------------+

7.16.  Question list table

   This

   The table "qlist" holds a list of second and subsequent individual
   Questions in a Question section.  Each item in the table is a CBOR unsigned.
   unsigned integer.  This data is optionally collected.

   +----------+--------------------------------------------------------+
   | Field    | Description                                            |
   +----------+--------------------------------------------------------+
   | Question question | The index in the Question table of the individual      |
   |          | Question.                                              |
   +----------+--------------------------------------------------------+

7.17.  Resource Record list table

   This

   The table "rrlist" holds a list of individual Resource Records in a
   Answer, Authority or Additional section.  Each item in the table is a
   CBOR
   unsigned. unsigned integer.  This data is optionally collected.

   +-------+-----------------------------------------------------------+
   | Field | Description                                               |
   +-------+-----------------------------------------------------------+
   | RR rr    | The index in the Resource Record table of the individual  |
   |       | Resource Record.                                          |
   +-------+-----------------------------------------------------------+

7.18.  Query/Response data

   The block Q/R data is a CBOR array of individual Q/R data items.
   Each item in the array is a CBOR map containing details on the
   individual Q/R pair. data item.

   Note that there is no requirement that the elements of the Q/R array
   are presented in strict chronological order.

   The following abbreviations are used in the Present (P) column

   o  Q = QUERY

   o  A = Always

   o  QT = QUESTION

   o  QO = QUERY, OPT

   o  QR = QUERY & RESPONSE

   o  R = RESPONSE

   Each item in the map has an unsigned value (with the exception of
   those listed below) and an unsigned key.

   o  Query extended information  query-extended and Response extended information response-extended which are of Type type Extended
      Information.

   o  Response delay  delay-useconds and delay-pseconds which is an integer (This are integers (The delay
      can be negative if the network stack/capture library returns them
      out of order.)

   +-------------+----+------------------------------------------------+

   +-----------------------+----+--------------------------------------+
   | Field                 | P  | Description                          |
   +-------------+----+------------------------------------------------+
   +-----------------------+----+--------------------------------------+
   | Time offset time-useconds         | A  | Q/R timestamp as an offset in microseconds        |
   |                       |    | microseconds from the Block pre-amble Timestamp. The preamble |
   |                       |    | Timestamp. The timestamp is the      |
   |                       |    | timestamp of the Query, or the       |
   |                       |    | the Response if there is no Query.       |
   |                       |    |                                      |
   | Client time-pseconds         | A  | Picosecond component of the          |
   |                       |    | timestamp. Optional.                 |
   |                       |    |                                      |
   | client-address-index  | A  | The index in the IP address table of the |
   | address                       |    | the client IP address.               |
   |                       |    |                                      |
   | Client port client-port           | A  | The client port.                     |
   |                       |    |                                      |
   | Transaction transaction-id        | A  | DNS transaction identifier.          |
   | ID          |    |                                                |                       |    |                                      |
   |
   | Query query-signature-index | A  | The index of the more information on the Q/R Query Signature     |
   | signature                       |    | in the Query Signature table. table record for this data item.     |
   |                       |    |                                      |
   | Client client-hoplimit       | Q  | The IPv4 TTL or IPv6 Hoplimit from the Query   |
   | hoplimit                       |    | the Query packet. Optional.          |
   |                       |    |                                      |
   | Response delay-useconds        | QR | The time different difference between Query and Response,    |
   | delay                       |    | and Response, in microseconds. Only  |
   |                       |    | present if there is a query and a    |
   |                       |    | response.                            |
   |                       |    |                                      |
   | delay-pseconds        | QR | Picosecond component of the time     |
   |                       |    | different between Query and          |
   |                       |    | Response. If delay-useconds is non-  |
   |                       |    | zero then delay-pseconds (if         |
   |                       |    | present) MUST be of the same sign as |
   |                       |    | delay-useconds, or be 0. Optional.   |
   |                       |    |                                      | Question
   | query-name-index      | QT | The index in the NAME/RDATA table of the QNAME |
   | NAME                       |    | the QNAME for the first Question.    |
   |                       |    | Optional.                            |
   |                       |    |                                      | Response
   | query-size            | R  | The size of the DNS query message (not the packet size (see below).  |
   | size                       |    | containing the message, just the Optional.                            |
   |                       |    |                                      |
   | response-size         | R  | DNS message) query message size (see below).  |
   |                       |    | that forms the Response. Optional.                            |
   |                       |    |                                      |
   | Query query-extended        | Q  | Extended Query information. This item is only     |
   | extended                       |    | item is only present if collection   |
   |                       |    | of extra Query information is        |
   | information                       |    | information is configured. Optional.                |
   |                       |    |                                      |
   | Response response-extended     | R  | Extended Response information. This item is  |
   | extended                       |    | item is only present if collection of extra Query   |
   | information                       |    | of extra Response information is configured.     |
   +-------------+----+------------------------------------------------+

   The collector
   |                       |    | configured. Optional.                |
   +-----------------------+----+--------------------------------------+

   An implementation must always collects collect basic Q/R information.  It may
   be configured to collect details on Question, Answer, Authority and
   Additional sections of the Query, the Response or both.  Note that
   only the second and subsequent Questions of any Question section are
   collected (the details of the first are in the basic information),
   and that OPT Records are not collected in the Additional section.

   The query-size and response-size fields hold the DNS message size.
   For UDP this is the size of the UDP payload that contained the DNS
   message and will therefore include any trailing bytes if present.
   Trailing bytes with queries are routinely observed in traffic to
   authoritative servers and this value allows a calculation of how many
   trailing bytes were present.  For TCP it is the size of the DNS
   message as specified in the two-byte message length header.

   The Extended information is a CBOR map as follows.  Each item in the
   map is present only of if collection of the relevant details is
   configured.  Each item in the map has an unsigned value and an
   unsigned key.

   +------------+------------------------------------------------------+

   +------------------+------------------------------------------------+
   | Field            | Description                                    |
   +------------+------------------------------------------------------+
   +------------------+------------------------------------------------+
   | Question question-index   | The index in the Questions list table of the entry   |
   |                  | entry listing the second and subsequent Question sections        |
   |                  | Question sections for the Query or Response.   |
   |                  |                                                |
   | Answer answer-index     | The index in the RR list table of the entry listing    |
   |                  | listing the Answer Resource Record sections    |
   |                  | for the Query or |
   |            | Response.                     |
   |                  |                                                |
   | Authority authority-index  | The index in the RR list table of the entry listing    |
   |                  | listing the Authority Resource Record sections for the Query |
   |                  | for the Query or Response.                     |
   |                  |                                                |
   | Additional additional-index | The index in the RR list table of the entry listing    |
   |                  | listing the Additional Resource Record sections for the         |
   |                  | sections for the Query or Response.            |
   +------------+------------------------------------------------------+
   +------------------+------------------------------------------------+

7.19.  Address Event counts

   This table holds counts of various IP related events relating to
   traffic with individual client addresses.

   +----------+----------+---------------------------------------------+

   +------------------+----------+-------------------------------------+
   | Field            | Type     | Description                         |
   +----------+----------+---------------------------------------------+
   +------------------+----------+-------------------------------------+
   | Event ae-type          | Unsigned | The type of event. The following events    |
   | type                  |          | events types are currently defined: |
   |                  |          | 0. TCP reset.                       |
   |                  |          | 1. ICMP time exceeded.              |
   |                  |          | 2. ICMP destination unreachable.    |
   |                  |          | 3. ICMPv6 time exceeded.            |
   |                  |          | 4. ICMPv6 destination unreachable.  |
   |                  |          | 5. ICMPv6 packet too big.           |
   |                  |          |                                     |
   | Event ae-code          | Unsigned | A code relating to the event. Optional.       |
   | code                  |          | Optional.                           |
   |                  |          |                                     |
   | Address ae-address-index | Unsigned | The index in the IP address table of the   |
   | index                  |          | of the client address.              |
   |                  |          |                                     |
   | Count ae-count         | Unsigned | The number of occurrences of this event   |
   |                  |          | event during the block collection   |
   |                  |          | period.                             |
   +----------+----------+---------------------------------------------+

8.  C-DNS to PCAP

   It is possible to re-construct PCAP files from the C-DNS format.
   However this is a lossy process and some of the issues with
   reconstructing both the DNS payload and the full
   +------------------+----------+-------------------------------------+

7.20.  Malformed packet stream are
   outlined here.

   Firstly the reconstruction depends on whether or not all the records

   This optional
   sections of both the query and response were captured in the C-DNS
   file.  Clearly if they were not all captured table records the reconstruction is
   imperfect.

   Secondly, even if all sections original wire format content of the response were captured name
   compression presents a challenge in reconstructing the DNS response
   payload byte for byte.
   malformed packets (see Section 8.1 discusses this is more detail.

   Thirdly, not all transport information is captured 8).

   +----------------+--------+-----------------------------------------+
   | Field          | Type   | Description                             |
   +----------------+--------+-----------------------------------------+
   | time-useconds  | A      | Packet timestamp as an offset in        |
   |                |        | microseconds from the C-DNS
   format.  For example, the following aspects Block preamble    |
   |                |        | Timestamp.                              |
   |                |        |                                         |
   | time-pseconds  | A      | Picosecond component of the original timestamp.  |
   |                |        | Optional.                               |
   |                |        |                                         |
   | packet-content | Byte   | The packet
   stream cannot be re-constructed from the C-DNS format:

   o  IP Fragmentation

   o  TCP stream information:

      *  Multiple DNS messages may have been sent content in a single TCP
         segment

      *  A DNS payload may have be split across multiple TCP segments

      *  Multiple DNS messages may have be sent on a single TCP session

   o wire format.      |
   |                | string |                                         |
   +----------------+--------+-----------------------------------------+

8.  Malformed DNS messages and non-DNS packets

   Simple assumptions can be made on Packets

   In the reconstruction - fragmented and
   DNS-over-TCP messages can be reconstructed into 'single' packets and context of generating a single TCP session can be constructed for each TCP packet.

   Additionally if the malformed and non-DNS C-DNS file it is assumed that only
   those packets are captured
   separately into PCAPs they which can be merged with PCAPs reconstructed
   from C-DNS parsed to produce a more complete packet stream.

8.1.  Name Compression

   All the names well-formed DNS
   message are stored in the C-DNS format are full domain names; no
   DNS style name compression is used on the individual names within the format.  Therefore when reconstructing  This means as a minimum:

   o  The packet name compression must
   be used in order to re-produce the on has a well-formed 12 bytes DNS Header

   o  The section counts are consistent with the wire representation section contents

   o  All of the
   packet.

   [RFC1035] name compression works by substituting trailing sections of resource records can be parsed

   In principle, packets that do not meet these criteria could be
   classified into two categories:

   o  Partially malformed: those packets which can be decoded
      sufficiently to extract

      *  a name with DNS header (and therefore a reference back to DNS transaction ID)

      *  a QDCOUNT

      *  the occurrence of those sections
   earlier first question in the packet.  Not all name server software uses the same
   algorithm when compressing domain names within the responses.  Some
   attempt maximum recompression at the expense of runtime resources,
   others use heuristics to balance compression and speed and others use
   different rules for what QUESTION section if QDCOUNT is a valid compression target.
         greater than 0

      but suffer other issues while parsing.  This means is the minimum
      information required to attempt packet matching as described in
      Section 10.1

   o  Completely malformed: those packets that responses cannot be decoded to the same this
      extent.

   An open question from different name
   server software which match is whether there is value in attempting to process
   partially malformed packets in an analogous manner to well formed
   packets in terms of DNS payload content (header,
   counts, RRs with name compression removed) do not necessarily attempting to match
   byte for byte on them with the wire.

   From corresponding
   query or response.  This could be done by creating 'placeholder'
   records during packet matching with just the information extracted as
   above.  If the packet were then matched the resulting C-DNS format it is not possible Q/R data
   item would include a flag to ensure that the DNS
   response payload is reconstructed byte for byte.  However it can at
   least, in principle, be reconstructed indicate a malformed record (in addition
   to have the correct payload
   length (since capturing the original response length is captured) if there is
   enough knowledge wire format of the commonly implemented name compression
   algorithms.  For example, a simplistic approach packet).

   An advantage of this would be to try each
   algorithm that it would result in turn more meaningful
   statistics about matched packets because, for example, some partially
   malformed queries could be matched to see if responses.  However it reproduces the original length,
   stopping at the first match.  This would not guarantee the correct
   algorithm has been used as it is possible to match the length whilst
   still not matching the on the wire bytes but without further
   information added
   only apply to those queries where the C-BOR this first QUESTION is the best that can be achieved.

   Appendix B presents an example of two differing compression
   algorithms used by well known name server software.

9.  Data Collection

   This section describes a non-normative proposed algorithm for formed.
   It could also simplify the
   processing of a captured stream downstream analysis of DNS queries and responses C-DNS files and
   matching queries/responses where possible.

   For the purposes
   reconstruction of this discussion, it packet streams from C-DNS.

   A disadvantage is assumed that this adds complexity to the input has
   been pre-processed such that:

   1.  All IP fragmentation reassembly, TCP stream reassembly etc. has
       already been performed

   2.  Each message is associated with transport meta-data packet matching
   and data representation, could potentially lead to false matches and
   some additional statistics would be required (e.g. counts for
   matched-partially-malformed, unmatched-partially-malformed,
   completely-malformed).

9.  C-DNS to
       generate PCAP

   It is possible to re-construct PCAP files from the Primary ID (see below)

   3.  Each message has C-DNS format in a well-formed DNS header
   lossy fashion.  Some of 12 bytes the issues with reconstructing both the DNS
   payload and (if
       present) the first RR in full packet stream are outlined here.

   The reconstruction depends on whether or not all the optional
   sections of both the query section can be parsed to
       generate the Secondary ID (see below).

       *  As noted earlier, this requirement can result in a malformed
          query being removed and response were captured in the pre-processing stage, but C-DNS
   file.  Clearly, if they were not all captured, the
          correctly formed response with RCODE reconstruction
   will be imperfect.

   Even if all sections of FORMERR being present

   DNS messages are processed in the order they are delivered response were captured, one cannot
   reconstruct the DNS response payload exactly due to the
   application.

   o  It should be noted fact that packet capture libraries do not necessary
      provide packets
   some DNS names in strict chronological order.

   [TODO: Discuss the corner cases resulting from message on the wire may have been compressed.
   Section 9.1 discusses this in is more detail.]

9.1.  Matching algorithm

   A schematic representation of the algorithm for matching Q/R pairs detail.

   Some transport information is
   shown not captured in the following diagram:

   Figure showing the packet matching algorithm format (PNG) [5]

   Figure showing C-DNS format.  For
   example, the packet matching algorithm format (SVG) [6]

   and further details following aspects of the algorithm are given in the following
   sections.

9.2.  Message identifiers

9.2.1.  Primary ID (required)

   A Primary ID can original packet stream cannot
   be constructed for each message which is composed of re-constructed from the following data:

   1.  Source IP Address

   2.  Destination C-DNS format:

   o  IP Address

   3.  Source Port

   4.  Destination Port

   5.  Transport

   6. fragmentation

   o  TCP stream information:

      *  Multiple DNS Message ID

9.2.2.  Secondary ID (optional)

   If present, the first question messages may have been sent in the Question section is used as a
   secondary ID for each message.  Note that there single TCP
         segment

      *  A DNS payload may have be well formed
   DNS queries that have a QDCOUNT of 0, and some responses split across multiple TCP segments

      *  Multiple DNS messages may have be sent on a
   QDCOUNT of 0 (for example, RCODE=FORMERR or NOTIMP)

9.3.  Algorithm Parameters

   1.  Configurable timeout

9.4.  Algorithm Requirements

   The algorithm is designed to handle single TCP session

   o  Malformed DNS messages if the following input data:

   1.  Multiple queries with wire format is not recorded

   o  Any Non-DNS messages that were in the same Primary ID (but different
       Secondary ID) arriving before any responses for these queries are
       seen.

   2.  Multiple queries with original packet stream e.g.
      ICMP

   Simple assumptions can be made on the same Primary reconstruction: fragmented and Secondary ID arriving
       before any responses for these queries are seen.

   3.  Queries for which no later response
   DNS-over-TCP messages can be found within the
       specified timeout.

   4.  Responses for which no previous query reconstructed into single packets and a
   single TCP session can be found within the
       specified timeout.

9.5.  Algorithm Limitations

   For cases 1 constructed for each TCP packet.

   Additionally, if malformed packets and 2 listed in the above requirements, it is not
   possible to unambiguously match queries Non-DNS packets are captured
   separately, they can be merged with responses.  The solution packet captures reconstructed
   from C-DNS to this employed produce a more complete packet stream.

9.1.  Name Compression

   All the names stored in this algorithm the C-DNS format are full domain names; no
   DNS style name compression is to match to used on the earliest query
   with individual names within the correct Primary and Secondary ID.

9.6.  Workspace

   A FIFO structure is
   format.  Therefore when reconstructing a packet, name compression
   must be used in order to hold reproduce the Q/R items during processing.

9.7.  Output

   The output is a list of Q/R data items.  Both on the Query and Response
   elements are optional in these items, therefore Q/R items have one of
   three types wire representation of content:

   1.  Paired Q/R messages

   2.  A query message (no response)

   3.  A response message (no query)

   The timestamp
   the packet.

   [RFC1035] name compression works by substituting trailing sections of
   a list item is that of name with a reference back to the query for cases 1 and 2
   and that occurrence of those sections
   earlier in the response for case 3.

9.8.  Post Processing

   When ending capture, packet.  Not all remaining entries in name server software uses the Q/R FIFO should be
   treated as timed out queries.

10.  IANA Considerations

   None

11.  Security Considerations

   Any control interface MUST perform authentication and encryption.

   Any data upload MUST be authenticated and encrypted.

12.  Acknowledgements

   The authors wish same
   algorithm when compressing domain names within the responses.  Some
   attempt maximum recompression at the expense of runtime resources,
   others use heuristics to thank CZ.NIC, in particular Tomas Gavenciak, for
   many useful discussions on binary formats, balance compression and packet
   matching.  Also Jan Vcelak and Wouter Wijngaards for discussions on
   name compression.

   Thanks to Robert Edmonds, Paul Hoffman speed and Jerry Lundstroem for
   review.

   Also, Miek Gieben others use
   different rules for mmark [7]

13.  Changelog

   draft-ietf-dnsop-dns-capture-format-00

   o  Changed dnstap.io to dnstap.info

   o  qr_data_format.png was cut off at the bottom

   o  Update authors address

   o  Improve wording in Abstract

   o  Changed DNS-STAT what is a valid compression target.

   This means that responses to C-DNS in CDDL

   o  Set the format version same question from different name
   server software which match in the CDDL

   o  Added a TODO: Add block statistics

   o  Added a TODO: Add extend to support pico/nano.  Also terms of DNS payload content (header,
   counts, RRs with name compression removed) do this for
      Time offset and Response delay

   o  Added a TODO: Need not necessarily match
   byte-for-byte on the wire.

   Therefore, it is not possible to develop optional representation of malformed
      packets within CBOR and what this means for packet matching.  This
      may influence which fields are optional ensure that the DNS response payload
   is reconstructed byte-for-byte from C-DNS data.  However, it can at
   least, in principle, be reconstructed to have the rest correct payload
   length (since the original response length is captured) if there is
   enough knowledge of the
      representation.

   o  Added section commonly implemented name compression
   algorithms.  For example, a simplistic approach would be to try each
   algorithm in turn to see if it reproduces the original length,
   stopping at the first match.  This would not guarantee the correct
   algorithm has been used as it is possible to match the length whilst
   still not matching the on design goals the wire bytes but, without further
   information added to Introduction

   o  Added a TODO: Can Class the C-DNS data, this is the best that can be optimised?  Should
   achieved.

   Appendix B presents an example of two different compression
   algorithms used by well-known name server software.

10.  Data Collection

   This section describes a class non-normative proposed algorithm for the
   processing of IN be
      inferred if not present?

   draft-dickinson-dnsop-dns-capture-format-00

   o  Initial commit

14.  References

14.1.  Normative References

   [RFC1035]  Mockapetris, P., "Domain names - implementation a captured stream of DNS queries and
              specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
              November 1987, <http://www.rfc-editor.org/info/rfc1035>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

   [RFC7049]  Bormann, C. and P. Hoffman, "Concise Binary Object
              Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
              October 2013, <http://www.rfc-editor.org/info/rfc7049>.

14.2.  Informative References

   [ditl]     DNS-OARC, "DITL", 2016, <https://www.dns-
              oarc.net/oarc/data/ditl>.

   [dnscap]   DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
              dnscap>.

   [dnstap]   dnstap.info, "dnstap", 2016, <http://dnstap.info/>.

   [dsc]      Wessels, D. and J. Lundstrom, "DSC", 2016,
              <https://www.dns-oarc.net/tools/dsc>.

   [I-D.daley-dnsxml]
              Daley, J., Morris, S., responses and J. Dickinson, "dnsxml - A
              standard XML representation
   matching queries/responses where possible.

   For the purposes of DNS data", draft-daley-
              dnsxml-00 (work in progress), July 2013.

   [I-D.greevenbosch-appsawg-cbor-cddl]
              Vigano, C. this discussion, it is assumed that the input has
   been pre-processed such that:

   1.  All IP fragmentation reassembly, TCP stream reassembly, and H. Birkholz, "CBOR data definition language
              (CDDL): so
       on, has already been performed

   2.  Each message is associated with transport metadata required to
       generate the Primary ID (see Section 10.2.1)

   3.  Each message has a notational convention well-formed DNS header of 12 bytes and (if
       present) the first RR in the Question section can be parsed to express CBOR data
              structures", draft-greevenbosch-appsawg-cbor-cddl-09 (work
       generate the Secondary ID (see below).  As noted earlier, this
       requirement can result in progress), September 2016.

   [I-D.hoffman-dns-in-json]
              Hoffman, P., "Representing a malformed query being removed in the
       pre-processing stage, but the correctly formed response with
       RCODE of FORMERR being present.

   DNS Messages messages are processed in JSON", draft-
              hoffman-dns-in-json-09 (work the order they are delivered to the
   application.  It should be noted that packet capture libraries do not
   necessary provide packets in progress), October 2016.

   [packetq]  .SE - The Internet Infrastructure Foundation, "PacketQ",
              2014, <https://github.com/dotse/PacketQ>.

   [pcap]     tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.

   [pcapng]   Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
              Harris, "pcap-ng", 2016, <https://github.com/pcapng/
              pcapng>.

   [rrtypes]  IANA, "RR types", 2016, <http://www.iana.org/assignments/
              dns-parameters/dns-parameters.xhtml#dns-parameters-4>.

14.3.  URIs

   [1] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/cdns_format.png

   [2] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/cdns_format.svg

   [3] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/qr_data_format.png

   [4] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/qr_data_format.svg

   [5] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/packet_matching.png

   [6] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/packet_matching.svg

   [7] https://github.com/miekg/mmark

   [8] https://www.nlnetlabs.nl/projects/nsd/

   [9] https://www.knot-dns.cz/

Appendix A.  CDDL

; CDDL specification strict chronological order.

   TODO: Discuss the corner cases resulting from this in more detail.

10.1.  Matching algorithm

   A schematic representation of the file format algorithm for C-DNS,
; which describes a collection matching Q/R data
   items is shown in the following diagram:

   Figure showing the packet matching algorithm format (PNG) [5]

   Figure showing the packet matching algorithm format (SVG) [6]

   Further details of DNS Query/Response pairs.

File = [
    file-type-id  : tstr,          ; "C-DNS"
    file-preamble : FilePreamble,
    file-blocks   : [* Block],
]

FilePreamble = {
    format-version  => uint, ; 1
    ? configuration => Configuration,
    ? generator-id  => tstr,
    ? host-id       => tstr,
}

format-version = 0
configuration  = 1
generator-id   = 2
host-id        = 3
Configuration = {
    ? query-timeout    => uint,
    ? skew-timeout     => uint,
    ? snaplen          => uint,
    ? promisc          => uint,
    ? interfaces       => [* tstr],
    ? vlan-ids         => [* uint],
    ? filter           => tstr,
    ? query-options    => uint,    ; See below
    ? response-options => uint,
    ? accept-rr-types  => [* tstr],
    ? ignore-rr-types  => [* tstr],
}

; query-options and response-options the algorithm are bitmasks. A bit set adds given in the
; specified following sections.
;
; second & subsequent question sections = 1
; answer sections = 2
; authority sections = 4
; additional sections = 8

query-timeout    = 0
skew-timeout     = 1
snaplen          = 2
promisc          = 3
interfaces       = 4
vlan-ids         = 5
filter           = 6
query-options    = 7
response-options = 8
accept-rr-types  = 9;
ignore-rr-types  = 10;

Block = {
    preamble             => BlockPreamble,
    ? statistics         => BlockStatistics, ; Much

10.2.  Message identifiers

10.2.1.  Primary ID (required)

   A Primary ID is constructed for each message.  It is composed of this could the
   following data:

   1.  Source IP Address

   2.  Destination IP Address

   3.  Source Port

   4.  Destination Port

   5.  Transport

   6.  DNS Message ID

10.2.2.  Secondary ID (optional)

   If present, the first question in the Question section is used as a
   secondary ID for each message.  Note that there may be derived
    tables               => BlockTables, well formed
   DNS queries              => [* QueryResponse],
    address-event-counts => [* AddressEventCount],
}

preamble             = that have a QDCOUNT of 0, and some responses may have a
   QDCOUNT of 0
statistics           = 1
tables               = 2 (for example, responses with RCODE=FORMERR or NOTIMP).
   In this case the secondary ID is not used in matching.

10.3.  Algorithm Parameters

   1.  Query timeout

   2.  Skew timeout

10.4.  Algorithm Requirements

   The algorithm is designed to handle the following input data:

   1.  Multiple queries              = 3
address-event-counts = 4
BlockPreamble = {
    start-time => Timeval
}

start-time = 1

Timeval = [
    seconds      : uint,
    microseconds : uint,
]

BlockStatistics = {
    ? total-packets        => uint,
    ? total-pairs          => uint,
    ? unmatched_queries    => uint,
    ? unmatched_responses  => uint,
    ? malformed-packets    => uint,
    ? non-dns-packets      => uint,
    ? out-of-order-packets => uint,
    ? missing-pairs        => uint,
    ? missing-packets      => uint,
    ? missing-non-dns      => uint,
}

total-packets        = 0
total-pairs          = 1
unmatched_queries    = 2
unmatched_responses  = 3
malformed-packets    = 4
non-dns-packets      = 5
out-of-order-packets = 6
missing-pairs        = 7
missing-packets      = 8
missing-non-dns      = 9

BlockTables = {
    ip-address => [* bstr],
    classtype  => [* ClassType],
    name-rdata => [* bstr],            ; Holds both Name RDATA with the same Primary ID (but different
       Secondary ID) arriving before any responses for these queries are
       seen.

   2.  Multiple queries with the same Primary and RDATA
    query_sig  => [* QuerySignature]
    ? qlist    => [* QuestionList],
    ? qrr      => [* Question],
    ? rrlist   => [* RRList],
    ? rr       => [* RR],
}

ip-address = 0
classtype  = Secondary ID arriving
       before any responses for these queries are seen.

   3.  Queries for which no later response can be found within the
       specified timeout.

   4.  Responses for which no previous query can be found within the
       specified timeout.

10.5.  Algorithm Limitations

   For cases 1
name-rdata = and 2
query_sig  = 3
qlist      = 4
qrr        = 5
rrlist     = 6
rr         = 7

QueryResponse = {
    time-useconds         => uint,        ; Time offset from listed in the above requirements, it is not
   possible to unambiguously match queries with responses.  This
   algorithm chooses to match to the earliest record
    client-address-index  => uint,
    client-port           => uint,
    transaction-id        => uint,
    query-signature-index => uint,
    ? client-hoplimit     => uint,
    ? delay-useconds      => int,        ; Times may be -ve at capture
    ? query-name-index    => uint,
    ? response-size       => uint,       ; DNS size query with the correct
   Primary and Secondary ID.

10.6.  Workspace

   A FIFO structure is used to hold the Q/R data items during
   processing.

10.7.  Output

   The output is a list of response
    ? query-extended      => QueryResponseExtended,
    ? response-extended   => QueryResponseExtended,
}

time-useconds         = 0
client-address-index  = 1
client-port           = 2
transaction-id        = 3
query-signature-index = 4
client-hoplimit       = 5
delay-useconds        = 6
query-name-index      = 7
response-size         = 8
query-extended        = 9
response-extended     = 10

ClassType = {
    type  => uint,
    class => uint,
}

type  = 0
class = 1

QuerySignature = {
    server-address-index    => uint,
    server-port             => uint,
    transport-flags         => uint,
    qr-sig-flags            => uint,
    ? query-opcode          => uint,
    qr-dns-flags            => uint,
    ? query-rcode           => uint,
    ? query-classtype-index => uint,
    ? query-qd-count        => uint,
    ? query-an-count        => uint,
    ? query-ar-count        => uint,
    ? query-ns-count        => uint,
    ? edns-version          => uint,
    ? udp-buf-size          => uint,
    ? opt-rdata-index       => uint,
    ? response-rcode        => uint,
}

server-address-index  = 0
server-port           = 1
transport-flags       = 2
qr-sig-flags          = 3
query-opcode          = 4
qr-dns-flags          = 5
query-rcode           = 6
query-classtype-index = 7
query-qd-count        = 8
query-an-count        = 9
query-ar-count        = 10
query-ns-count        = 11
edns-version          = 12
udp-buf-size          = 13
opt-rdata-index       = 14
response-rcode        = 15

QuestionList = [
    * uint,                           ; Index Q/R data items.  Both the Query and Response
   elements are optional in these items, therefore Q/R data items have
   one of Question
]

Question = {                          ; Second three types of content:

   1.  A matched pair of query and subsequent questions
    name-index      => uint,          ; Index to response messages

   2.  A query message with no response

   3.  A response message with no query

   The timestamp of a name in list item is that of the name-rdata table
    classtype-index => uint,
}

name-index      = 0
classtype-index = query for cases 1

RRList = [
    * uint,                           ; Index and 2
   and that of RR
]

RR = {
    name-index      => uint,          ; Index to a name in the name-rdata table
    classtype-index => uint,
    ttl             => uint,
    rdata-index     => uint,          ; Index to RDATA response for case 3.

10.8.  Post Processing

   When ending capture, all remaining entries in the name-rdata table
}

ttl         = 2
rdata-index = 3

QueryResponseExtended = {
    ? question-index   => uint,       ; Index of QuestionList
    ? answer-index     => uint,       ; Index of RRList
    ? authority-index  => uint,
    ? additional-index => uint,
}

question-index   = 0
answer-index     = 1
authority-index  = 2
additional-index = 3

AddressEventCount = {
    ae-type          => &AddressEventType,
    ? ae-code        => uint,
    ae-address-index => uint,
    ae-count         => uint,
}

ae-type          = 0
ae-code          = 1
ae-address-index = 2
ae-count         = 3

AddressEventType = (
    tcp-reset: 0,
    icmp-time-exceeded     : 1,
    icmp-dest-unreachable  : 2,
    icmpv6-time-exceeded   : 3,
    icmpv6-dest-unreachable: 4,
    icmpv6-packet-too-big  : 5,
)

Appendix B.  DNS Name compression example Q/R data item FIFO
   should be treated as timed out queries.

11.  IANA Considerations

   None

12.  Security Considerations

   Any control interface MUST perform authentication and encryption.

   Any data upload MUST be authenticated and encrypted.

13.  Acknowledgements

   The basic algorithm which follows the guidance in [RFC1035] is simply authors wish to collect each name, and the offset thank CZ.NIC, in the packet at which it
   starts, during particular Tomas Gavenciak, for
   many useful discussions on binary formats, compression and packet construction.  As each
   matching.  Also Jan Vcelak and Wouter Wijngaards for discussions on
   name is added, it is
   offered to each of the collected names in order compression and Paul Hoffman for a detailed review of collection,
   starting from the first name.  If labels at the end of
   document and the name can
   be replaced with a reference back C-DNS CDDL.

   Thanks also to part (or all) of the earlier
   name, Robert Edmonds and if Jerry Lundstroem for review.

   Also, Miek Gieben for mmark [7]

14.  Changelog

   draft-ietf-dnsop-dns-capture-format-01

   o  Many editorial improvements by Paul Hoffman

   o  Included discussion of malformed packet handling

   o  Improved Appendix C on Comparison of Binary Formats

   o  Now using C-DNS field names in the uncompressed part tables in section 8
   o  A handful of new fields included (CDDL updated)

   o  Timestamps now include optional picoseconds

   o  Added details of block statistics

   draft-ietf-dnsop-dns-capture-format-00

   o  Changed dnstap.io to dnstap.info

   o  qr_data_format.png was cut off at the name is shorter than any
   compression already found, bottom

   o  Update authors address

   o  Improve wording in Abstract

   o  Changed DNS-STAT to C-DNS in CDDL

   o  Set the earlier name is noted as format version in the
   compression target CDDL

   o  Added a TODO: Add block statistics

   o  Added a TODO: Add extend to support pico/nano.  Also do this for
      Time offset and Response delay

   o  Added a TODO: Need to develop optional representation of malformed
      packets within C-DNS and what this means for packet matching.
      This may influence which fields are optional in the name.

   The following tables illustrate the process.  In an example packet, rest of the first name is example.com.

          +---+-------------+--------------+--------------------+
      representation.

   o  Added section on design goals to Introduction

   o  Added a TODO: Can Class be optimised?  Should a class of IN be
      inferred if not present?

   draft-dickinson-dnsop-dns-capture-format-00

   o  Initial commit

15.  References

15.1.  Normative References

   [RFC1035]  Mockapetris, P., "Domain names - implementation and
              specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
              November 1987, <http://www.rfc-editor.org/info/rfc1035>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

   [RFC7049]  Bormann, C. and P. Hoffman, "Concise Binary Object
              Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
              October 2013, <http://www.rfc-editor.org/info/rfc7049>.

15.2.  Informative References

   [ditl]     DNS-OARC, "DITL", 2016, <https://www.dns-
              oarc.net/oarc/data/ditl>.

   [dnscap]   DNS-OARC, "DNSCAP", 2016, <https://www.dns-oarc.net/tools/
              dnscap>.

   [dnstap]   dnstap.info, "dnstap", 2016, <http://dnstap.info/>.

   [dsc]      Wessels, D. and J. Lundstrom, "DSC", 2016,
              <https://www.dns-oarc.net/tools/dsc>.

   [I-D.daley-dnsxml]
              Daley, J., Morris, S., and J. Dickinson, "dnsxml - A
              standard XML representation of DNS data", draft-daley-
              dnsxml-00 (work in progress), July 2013.

   [I-D.greevenbosch-appsawg-cbor-cddl]
              Vigano, C. and H. Birkholz, "CBOR data definition language
              (CDDL): a notational convention to express CBOR data
              structures", draft-greevenbosch-appsawg-cbor-cddl-09 (work
              in progress), September 2016.

   [I-D.hoffman-dns-in-json]
              Hoffman, P., "Representing DNS Messages in JSON", draft-
              hoffman-dns-in-json-10 (work in progress), October 2016.

   [packetq]  .SE - The Internet Infrastructure Foundation, "PacketQ",
              2014, <https://github.com/dotse/PacketQ>.

   [pcap]     tcpdump.org, "PCAP", 2016, <http://www.tcpdump.org/>.

   [pcapng]   Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G.
              Harris, "pcap-ng", 2016, <https://github.com/pcapng/
              pcapng>.

   [RFC7159]  Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
              Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March
              2014, <http://www.rfc-editor.org/info/rfc7159>.

   [rrtypes]  IANA, "RR types", 2016, <http://www.iana.org/assignments/
              dns-parameters/dns-parameters.xhtml#dns-parameters-4>.

15.3.  URIs

   [1] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/cdns_format.png

   [2] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/cdns_format.svg

   [3] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/qr_data_format.png

   [4] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/qr_data_format.svg

   [5] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/packet_matching.png

   [6] https://github.com/dns-stats/draft-dns-capture-
       format/blob/master/packet_matching.svg

   [7] https://github.com/miekg/mmark

   [8] https://www.nlnetlabs.nl/projects/nsd/

   [9] https://www.knot-dns.cz/

   [10] https://avro.apache.org/

   [11] https://developers.google.com/protocol-buffers/

   [12] http://cbor.io

   [13] https://github.com/kubo/snzip

   [14] http://google.github.io/snappy/

   [15] http://lz4.github.io/lz4/

   [16] http://www.gzip.org/

   [17] http://facebook.github.io/zstd/

   [18] http://tukaani.org/xz/

   [19] https://github.com/dns-stats/draft-dns-capture-
        format/blob/master/file-size-versus-block-size.png

   [20] https://github.com/dns-stats/draft-dns-capture-
        format/blob/master/file-size-versus-block-size.svg

Appendix A.  CDDL

  ; CDDL specification of the file format for C-DNS,
  ; which describes a collection of DNS messages and
  ; traffic meta-data.

  File = [
      file-type-id  : tstr, ; = "C-DNS"
      file-preamble : FilePreamble,
      file-blocks   : [* Block],
  ]

  FilePreamble = {
      major-format-version => uint,  ; = 1
      minor-format-version => uint,  ; = 0
      ? private-version    => uint,
      ? configuration      => Configuration,
      ? generator-id       => tstr,
      ? host-id            => tstr,
  }

  major-format-version = 0
  minor-format-version = 1
  private-version      = 2
  configuration        = 3
  generator-id         = 4
  host-id              = 5

  Configuration = {
      ? query-timeout      => uint,
      ? skew-timeout       => uint,
      ? snaplen            => uint,
      ? promisc            => uint,
      ? interfaces         => [* tstr],
      ? server-addresses   => [* IPAddress], ; Hint for later analysis
      ? vlan-ids           => [* uint],
      ? filter             => tstr,
      ? query-options      => QRCollectionSections,
      ? response-options   => QRCollectionSections,
      ? accept-rr-types    => [* uint],
      ? ignore-rr-types    => [* uint],
      ? max-block-qr-items => uint,
      ? collect-malformed  => uint,
  }

  QRCollectionSectionValues = &(
      question  : 0, ; Second & subsequent question sections
      answer    : 1,
      authority : 2,
      additional: 3,
  )
  QRCollectionSections = uint .bits QRCollectionSectionValues

  query-timeout      = 0
  skew-timeout       = 1
  snaplen            = 2
  promisc            = 3
  interfaces         = 4
  vlan-ids           = 5
  filter             = 6
  query-options      = 7
  response-options   = 8
  accept-rr-types    = 9
  ignore-rr-types    = 10
  server-addresses   = 11
  max-block-qr-items = 12
  collect-malformed  = 13

  Block = {
      preamble                => BlockPreamble,
      ? statistics            => BlockStatistics,
      tables                  => BlockTables,
      queries                 => [* QueryResponse],
      ? address-event-counts  => [* AddressEventCount],
      ? malformed-packet-data => [* MalformedPacket],
  }

  preamble              = 0
  statistics            = 1
  tables                = 2
  queries               = 3
  address-event-counts  = 4
  malformed-packet-data = 5

  BlockPreamble = {
      earliest-time => Timeval
  }
  earliest-time = 1

  Timeval = [
      seconds       : uint,
      microseconds  : uint,
      ? picoseconds : uint,
  ]

  BlockStatistics = {
      ? total-packets                => uint,
      ? total-pairs                  => uint,
      ? unmatched-queries            => uint,
      ? unmatched-responses          => uint,
      ? malformed-packets            => uint,
  }

  total-packets                = 0
  total-pairs                  = 1
  unmatched-queries            = 2
  unmatched-responses          = 3
  malformed-packets            = 4

  BlockTables = {
      ip-address => [* IPAddress],
      classtype  => [* ClassType],
      name-rdata => [* bstr], ; Holds both Name RDATA and RDATA
      query-sig  => [* QuerySignature]
      ? qlist    => [* QuestionList],
      ? qrr      => [* Question],
      ? rrlist   => [* RRList],
      ? rr       => [* RR],
  }

  ip-address = 0
  classtype  = 1
  name-rdata = 2
  query-sig  = 3
  qlist      = 4
  qrr        = 5
  rrlist     = 6
  rr         = 7

  QueryResponse = {
      time-useconds         => uint, ; Time offset from start of block
      ? time-pseconds       => uint, ; in microseconds and picoseconds
      client-address-index  => uint,
      client-port           => uint,
      transaction-id        => uint,
      query-signature-index => uint,
      ? client-hoplimit     => uint,
      ? delay-useconds      => int,
      ? delay-pseconds      => int, ; Has same sign as delay-useconds
      ? query-name-index    => uint,
      ? query-size          => uint, ; DNS size of query
      ? response-size       => uint, ; DNS size of response
      ? query-extended      => QueryResponseExtended,
      ? response-extended   => QueryResponseExtended,
  }

  time-useconds         = 0
  time-pseconds         = 1
  client-address-index  = 2
  client-port           = 3
  transaction-id        = 4
  query-signature-index = 5
  client-hoplimit       = 6
  delay-useconds        = 7
  delay-pseconds        = 8
  query-name-index      = 9
  query-size            = 10
  response-size         = 11
  query-extended        = 12
  response-extended     = 13

  ClassType = {
      type  => uint,
      class => uint,
  }

  type  = 0
  class = 1

  DNSFlagValues = &(
      query-cd   : 0,
      query-ad   : 1,
      query-z    : 2,
      query-ra   : 3,
      query-rd   : 4,
      query-tc   : 5,
      query-aa   : 6,
      query-d0   : 7,
      response-cd: 8,
      response-ad: 9,
      response-z : 10,
      response-ra: 11,
      response-rd: 12,
      response-tc: 13,
      response-aa: 14,
  )
  DNSFlags = uint .bits DNSFlagValues

  QueryResponseFlagValues = &(
      has-query               : 0,
      has-reponse             : 1,
      query-has-question      : 2,
      query-has-opt           : 3,
      response-has-opt        : 4,
      response-has-no-question: 5,
  )
  QueryResponseFlags = uint .bits QueryResponseFlagValues

  TransportFlagValues = &(
      tcp               : 0,
      ipv6              : 1,
      query-trailingdata: 2,
  )
  TransportFlags = uint .bits TransportFlagValues

  QuerySignature = {
      server-address-index    => uint,
      server-port             => uint,
      transport-flags         => TransportFlags,
      qr-sig-flags            => QueryResponseFlags,
      ? query-opcode          => uint,
      qr-dns-flags            => DNSFlags,
      ? query-rcode           => uint,
      ? query-classtype-index => uint,
      ? query-qd-count        => uint,
      ? query-an-count        => uint,
      ? query-ar-count        => uint,
      ? query-ns-count        => uint,
      ? edns-version          => uint,
      ? udp-buf-size          => uint,
      ? opt-rdata-index       => uint,
      ? response-rcode        => uint,
  }

  server-address-index  = 0
  server-port           = 1
  transport-flags       = 2
  qr-sig-flags          = 3
  query-opcode          = 4
  qr-dns-flags          = 5
  query-rcode           = 6
  query-classtype-index = 7
  query-qd-count        = 8
  query-an-count        = 9
  query-ar-count        = 10
  query-ns-count        = 11
  edns-version          = 12
  udp-buf-size          = 13
  opt-rdata-index       = 14
  response-rcode        = 15

  QuestionList = [
      * uint, ; Index of Question
  ]

  Question = {                 ; Second and subsequent questions
      name-index      => uint, ; Index to a name in the name-rdata table
      classtype-index => uint,
  }

  name-index      = 0
  classtype-index = 1

  RRList = [
      * uint, ; Index of RR
  ]

  RR = {
      name-index      => uint, ; Index to a name in the name-rdata table
      classtype-index => uint,
      ttl             => uint,
      rdata-index     => uint, ; Index to RDATA in the name-rdata table
  }

  ttl         = 2
  rdata-index = 3

  QueryResponseExtended = {
      ? question-index   => uint, ; Index of QuestionList
      ? answer-index     => uint, ; Index of RRList
      ? authority-index  => uint,
      ? additional-index => uint,
  }

  question-index   = 0
  answer-index     = 1
  authority-index  = 2
  additional-index = 3
  AddressEventCount = {
      ae-type          => &AddressEventType,
      ? ae-code        => uint,
      ae-address-index => uint,
      ae-count         => uint,
  }

  ae-type          = 0
  ae-code          = 1
  ae-address-index = 2
  ae-count         = 3

  AddressEventType = (
      tcp-reset              : 0,
      icmp-time-exceeded     : 1,
      icmp-dest-unreachable  : 2,
      icmpv6-time-exceeded   : 3,
      icmpv6-dest-unreachable: 4,
      icmpv6-packet-too-big  : 5,
  )

  MalformedPacket = {
      time-useconds   => uint, ; Time offset from start of block
      ? time-pseconds => uint, ; in microseconds and picoseconds
      packet-content  => bstr, ; Raw packet contents
  }

  time-useconds    = 0
  time-pseconds    = 1
  packet-content   = 2

  IPv4Address = bstr .size 4
  IPv6Address = bstr .size 16
  IPAddress = IPv4Address / IPv6Address

Appendix B.  DNS Name compression example

   The basic algorithm, which follows the guidance in [RFC1035], is
   simply to collect each name, and the offset in the packet at which it
   starts, during packet construction.  As each name is added, it is
   offered to each of the collected names in order of collection,
   starting from the first name.  If labels at the end of the name can
   be replaced with a reference back to part (or all) of the earlier
   name, and if the uncompressed part of the name is shorter than any
   compression already found, the earlier name is noted as the
   compression target for the name.

   The following tables illustrate the process.  In an example packet,
   the first name is example.com.

          +---+-------------+--------------+--------------------+
          | N | Name        | Uncompressed | Compression Target |
          +---+-------------+--------------+--------------------+
          | 1 | example.com |              |                    |
          +---+-------------+--------------+--------------------+

   The next name added is bar.com.  This is matched against example.com.
   The com part of this can be used as a compression target, with the
   remaining uncompressed part of the name being bar.

          +---+-------------+--------------+--------------------+
          | N | Name        | Uncompressed | Compression Target |
          +---+-------------+--------------+--------------------+
          | 1 | example.com |              |                    |
          | 2 | bar.com     | bar          | 1 + offset to com  |
          +---+-------------+--------------+--------------------+

   The third name added is www.bar.com.  This is first matched against
   example.com, and as before this is recorded as a compression target,
   with the remaining uncompressed part of the name being www.bar.  It
   is then matched against the second name, which again can be a
   compression target.  Because the remaining uncompressed part of the
   name is www, this is an improved compression, and so it is adopted.

          +---+-------------+--------------+--------------------+
          | N | Name        | Uncompressed | Compression Target |
          +---+-------------+--------------+--------------------+
          | 1 | example.com |              |                    |
          | 2 | bar.com     | bar          | 1 + offset to com  |
          | 3 | www.bar.com | www          | 2                  |
          +---+-------------+--------------+--------------------+

   As an optimization, if a name is already perfectly compressed (in
   other words, the uncompressed part of the name is empty), then no
   further names will be considered for compression.

B.1.  NSD compression algorithm

   Using the above basic algorithm the packet lengths of responses
   generated by NSD [8] can be matched almost exactly.  At the time of
   writing, a tiny number (<.01%) of the reconstructed packets had
   incorrect lengths.

B.2.  Knot Authoritative compression algorithm

   The Knot Authoritative [9] name server uses different compression
   behavior, which is the result of internal optimization designed to
   balance runtime speed with compression size gains.  In brief, and
   omitting complications, Knot Authoritative will only consider the
   QNAME and names in the immediately preceding RR section in an RRSET
   as compression targets.

   A set of smart heuristics as described below can be implemented to
   mimic this and while not perfect it produces output nearly, but not
   quite, as good a match as with NSD.  The heuristics are:

   1.  A match is only perfect if the name is completely compressed AND
       the TYPE of the section in which the name occurs matches the TYPE
       of the name used as the compression target.

   2.  If the name occurs in RDATA:

       *  If the compression target name is in a query, then only the
          first RR in an RRSET can use that name as a compression
          target.

       *  The compression target name MUST be in RDATA.

       *  The name section TYPE must match the compression target name
          section TYPE.

       *  The compression target name MUST be in the immediately
          preceding RR in the RRSET.

   Using this algorithm less than 0.1% of the reconstructed packets had
   incorrect lengths.

B.3.  Observed differences

   In sample traffic collected on a root name server around 2-4% of
   responses generated by Knot had different packet lengths to those
   produced by NSD.

Appendix C.  Comparison of Binary Formats

   Several binary serialisation formats were considered, and for
   completeness were also compared to JSON.

   o  Apache Avro [10].  Data is stored according to a pre-defined
      schema.  The schema itself is always included in the data file.

      Data can therefore be stored untagged, for a smaller serialisation
      size, and be written and read by an Avro library.

      *  At the time of writing, Avro libraries are available for C,
         C++, C#, Java, Python, Ruby and PHP.  Optionally tools are
         available for C++, Java and C# to generate code for encoding
         and decoding.

   o  Google Protocol Buffers [11].  Data is stored according to a pre-
      defined schema.  The schema is used by a generator to generate
      code for encoding and decoding the data.  Data can therefore be
      stored untagged, for a smaller serialisation size.  The schema is
      not stored with the data, so unlike Avro cannot be read with a
      generic library.

      *  Code must be generated for a particular data schema to to read
         and write data using that schema.  At the time of writing, the
         Google code generator can currently generate code for encoding
         and decoding a schema for C++, Go, Java, Python, Ruby, C#,
         Objective-C, Javascript and PHP.

   o  CBOR [12].  Defined in [RFC7049], this serialisation format is
      comparable to JSON but with a binary representation.  It does not
      use a pre-defined schema, so data is always stored tagged.
      However, CBOR data schemas can be described using CDDL
      [I-D.greevenbosch-appsawg-cbor-cddl] and tools exist to verify
      data files conform to the schema.

      *  CBOR is a simple format, and simple to implement.  At the time
         of writing, the CBOR website lists implementations for 16
         languages.

   Avro and Protocol Buffers both allow storage of untagged data, but
   because they rely on the data schema for this, their implementation
   is considerably more complex than CBOR.  Using Avro or Protocol
   Buffers in an unsupported environment would require notably greater
   development effort compared to CBOR.

   A test program was written which reads input from a PCAP file and
   writes output using one of two basic structures; either a simple
   structure, where each query/response pair is represented in a single
   record entry, or the C-DNS block structure.

   The resulting output files were then compressed using a variety of
   common general-purpose lossless compression tools to explore the
   compressibility of the formats.  The compression tools employed were:

   o  snzip [13].  A command line compression tool based on the Google
      Snappy [14] library.

   o  lz4 [15].  The command line compression tool from the reference C
      LZ4 implementation.

   o  gzip [16].  The ubiquitous GNU zip tool.

   o  zstd [17].  Compression using the Zstandard algorithm.

   o  xz [18].  A popular compression tool noted for high compression.

   In all cases the compression tools were run using their default
   settings.

   Note that this draft does not mandate the use of compression, nor any
   particular compression scheme, but it anticipates that in practice
   output data will be subject to general-purpose compression, and so
   this should be taken into consideration.

   "test.pcap", a 662Mb capture of sample data from a root instance was
   used for the comparison.  The following table shows the formatted
   size and size after compression (abbreviated to Comp. in the table
   headers), together with the task resident set size (RSS) and the user
   time taken by the compression.  File sizes are in Mb, RSS in kb and
   user time in seconds.

   +-------------+-----------+-------+------------+-------+-----------+
   | Format      | File size | Comp. | Comp. size |   RSS | User time |
   +-------------+-----------+-------+------------+-------+-----------+
   | PCAP        |    661.87 | snzip |     212.48 |  2696 |      1.26 |
   |             |           | lz4   |     181.58 |  6336 |      1.35 |
   |             |           | gzip  |     153.46 |  1428 |     18.20 |
   |             |           | zstd  |      87.07 |  3544 |      4.27 |
   |             |           | xz    |      49.09 | 97416 |    160.79 |
   |             |           |       |            |       |           |
   | JSON simple |   4113.92 | snzip |     603.78 |  2656 |      5.72 |
   |             |           | lz4   |     386.42 |  5636 |      5.25 |
   |             |           | gzip  |     271.11 |  1492 |     73.00 |
   |             |           | zstd  |     133.43 |  3284 |      8.68 |
   |             |           | xz    |      51.98 | 97412 |    600.74 |
   |             |           |       |            |       |           |
   | Avro simple |    640.45 | snzip |     148.98 |  2656 |      0.90 |
   |             |           | lz4   |     111.92 |  5828 |      0.99 |
   |             |           | gzip  |     103.07 |  1540 |     11.52 |
   |             |           | zstd  |      49.08 |  3524 |      2.50 |
   |             |           | xz    |      22.87 | 97308 |     90.34 |
   |             |           |       |            |       |           |
   | CBOR simple |    764.82 | snzip |     164.57 |  2664 |      1.11 |
   |             |           | lz4   |     120.98 |  5892 |      1.13 |
   |             |           | gzip  |     110.61 |  1428 |     12.88 |
   |             |           | zstd  |      54.14 |  3224 |      2.77 |
   |             |           | xz    |      23.43 | 97276 |    111.48 |
   |             |           |       |            |       |           |
   | PBuf simple |    749.51 | snzip |     167.16 |  2660 |      1.08 |
   |             |           | lz4   |     123.09 |  5824 |      1.14 |
   |             |           | gzip  |     112.05 |  1424 |     12.75 |
   |             |           | zstd  |      53.39 |  3388 |      2.76 |
   |             |           | xz    |      23.99 | 97348 |    106.47 |
   |             |           |       |            |       |           |
   | JSON block  |    519.77 | snzip |     106.12 |  2812 |      0.93 |
   |             |           | lz4   |     104.34 |  6080 |      0.97 |
   |             |           | gzip  |      57.97 |  1604 |     12.70 |
   |             |           | zstd  |      61.51 |  3396 |      3.45 |
   |             |           | xz    |      27.67 | 97524 |    169.10 |
   |             |           |       |            |       |           |
   | Avro block  |     60.45 | snzip |      48.38 |  2688 |      0.20 |
   |             |           | lz4   |      48.78 |  8540 |      0.22 |
   |             |           | gzip  |      39.62 |  1576 |      2.92 |
   |             |           | zstd  |      29.63 |  3612 |      1.25 |
   |             |           | xz    |      18.28 | 97564 |     25.81 |
   |             |           |       |            |       |           |
   | CBOR block  |     75.25 | snzip |      53.27 |  2684 |      0.24 |
   |             |           | lz4   |      51.88 |  8008 |      0.28 |
   |             |           | gzip  |      41.17 |  1548 |      4.36 |
   |             |           | zstd  |      30.61 |  3476 |      1.48 |
   |             | N           | Name xz    | Uncompressed      18.15 | Compression Target 97556 |
          +---+-------------+--------------+--------------------+     38.78 | 1
   | example.com             |           |       |
          +---+-------------+--------------+--------------------+

   The next name added is bar.com.  This is matched against example.com.
   The com part of this can be used as a compression target, with the
   remaining uncompressed part of the name being bar.

          +---+-------------+--------------+--------------------+            | N       | Name           | Uncompressed
   | Compression Target PBuf block  |
          +---+-------------+--------------+--------------------+     67.98 | 1 snzip | example.com      51.10 |  2636 |      0.24 |
   | 2             | bar.com           | bar lz4   | 1 + offset to com      52.39 |
          +---+-------------+--------------+--------------------+

   The third name added is www.bar.com.  This is first matched against
   example.com, and as before this is recorded as a compression target,
   with the remaining uncompressed part of the name being www.bar.  It
   is then matched against the second name, which again can be a
   compression target.  Because the remaining uncompressed part of the
   name is www, this is an improved compression, and so it is adopted.

          +---+-------------+--------------+--------------------+  8304 | N      0.24 | Name
   | Uncompressed             | Compression Target           |
          +---+-------------+--------------+--------------------+ gzip  | 1      40.19 | example.com  1520 |      3.63 |
   |             | 2           | bar.com zstd  | bar      31.61 | 1 + offset to com  3576 |      1.40 | 3
   | www.bar.com             | www           | 2 xz    |
          +---+-------------+--------------+--------------------+

   As an optimization, if a name is already perfectly compressed - in
   other words, the uncompressed part of the name is empty - no further
   names will be considered for compression.

B.1.  NSD compression algorithm

   Using the      17.94 | 97440 |     33.99 |
   +-------------+-----------+-------+------------+-------+-----------+

   The above basic algorithm the packet lengths of responses
   generated by NSD [8] can be matched almost exactly.  At the time of
   writing, a tiny number (<.01%) of results are discussed in the reconstructed packets had
   incorrect lengths.

B.2.  Knot Authoritative compression algorithm

   The Knot Authoritative [9] name server uses different compression
   behavior, which following sections.

C.1.  Comparison with full PCAP files

   An important first consideration is whether moving away from PCAP
   offers significant benefits.

   The simple binary formats are typically larger than PCAP, even though
   they omit some information such as Ethernet MAC addresses.  But not
   only do they require less CPU to compress than PCAP, the result resulting
   compressed files are smaller than compressed PCAP.

C.2.  Simple versus block coding

   The intention of internal optimization designed the block coding is to
   balance runtime speed with compression size gains.  In brief, and
   omitting complications, Knot Authoritative will only consider perform data de-duplication
   on query/response records within the
   QNAME block.  The simple and names block
   formats above store exactly the same information for each query/
   response record.  This information is parsed from the DNS traffic in
   the immediately preceding RR section input PCAP file, and in all cases each field has an RRSET
   as compression targets.

   A set of smart heuristics as described below can be implemented to
   mimic this identifier
   and while not perfect it produces output nearly, but not
   quite, as good a match as with NSD.  The heuristics are:

   1.  A match is only perfect if the name field data is completely compressed AND typed.

   The data de-duplication on the TYPE block formats show an order of the section
   magnitude reduction in which the name occurs matches the TYPE size of the name used as format file size against the compression target.

   2.  If
   simple formats.  As would be expected, the name occurs in RDATA: compression tools are able
   to find and exploit a  If lot of this duplication, but as the compression target name de-
   duplication process uses knowledge of DNS traffic, it is in able to
   retain a query, then only the
          first RR in an RRSET can use that name size advantage.  This advantage reduces as a compression
          target.

       b  The compression target name MUST be in RDATA.

       c  The name section TYPE must match the compression target name
          section TYPE.

       d  The stronger
   compression target name MUST is applied, as again would be in expected, but even with the immediately
          preceding RR in
   strongest compression applied the RRSET.

   Using this algorithm less than 0.1% block formatted data remains around
   75% of the reconstructed packets had
   incorrect lengths.

B.3.  Observed differences

   In sample traffic collected on a root name server around 2-4% size of
   responses generated by Knot had different packet lengths to those
   produced by NSD.

Appendix C.  Comparison the simple format and its compression requires
   roughly a third of the CPU time.

C.3.  Binary Formats

   Several versus text formats

   Text data formats offer many advantages over binary representations were considered formats,
   particularly in particular CBOR,
   Apache Avro and Protocol Buffers.

   Protocol Buffers and Avro both require a data schema, and validate the areas of ad-hoc data being stored against that schema.

   [TODO: Finish pros inspection and cons extraction.
   It was therefore felt worthwhile to carry out a direct comparison,
   implementing JSON versions of CBOR vs Avro vs Protocol buffers -
   tools, schema, adoption, etc.]

   The difference in file sizes were mostly minimal See Appendix D.3.

Appendix D.  Sample data the simple and block formats.

   Concentrating on JSON block format, the C-DNS format

   This section presents some example figures for the output size of
   capture files when using different block sizes, data representations
   and produced are a
   significant fraction of an order of magnitude larger than binary
   formats.  The data impact on file size after compression is sample data for a root instance.

   [TODO: This section needs more work..]

D.1.  Comparison to full PCAPS

   As can as might be seen in more detail below for this sample data
   expected from that starting point; the
   compressed C-DNS stronger compression produces
   files that are around 30% 150% of the size of the full similarly compressed
   PCAPs.  It should also be noted that experiments showed that
   compression of binary
   format, and require over 4x more CPU to compress.

C.4.  Performance

   Concentrating again on the C-DNS block formats, all three produce format required very roughly
   files that are close to an order of magnitude less CPU resources smaller that the
   original "test.pcap" file.  CBOR produces the largest files and Avro
   the smallest, 20% smaller than CBOR.

   However, once compression of full PCAPSs when
   using one core from a 3.5GHz i7 processor.

D.2.  Block size choices

   [TODO: Discuss trade-off of file block is taken into account, the size vs memory consumption.]

   [TODO: Add graph that demonstrates block difference
   narrows.  At medium compression (with gzip), the size of 5000 difference is optimal for
   4%.  Using strong compression (with xz) the sample data used.]

D.3.  Blocking vs more simple output

   Some experiments were conducted producing output in a very simple
   format involving a single record per Q/R data item (akin to a .csv
   representation).  The aim here was difference reduces to examine whether the blocking
   mechanism (using a block size of 5000) was worth 2%,
   with Avro the complexity,
   particularly after compression of largest and Protocol Buffers the output file using several
   general purpose smallest, although
   CBOR and Protocol Buffers require slightly more compression tools. CPU.

   The original PCAP file was
   325.79M and compressed using xz measurements presented above do not include data on the CPU
   required to 24.3Mb.

          +-------------+-------------+--------+--------+-------+
          | Format      | Output size |    lz4 |   gzip |    xz |
          +-------------+-------------+--------+--------+-------+
          | cbor-simple |      44.23M | 16.06M | 11.50M | 7.51M |
          | cbor-block  |      22.44M | 15.14M | 10.70M | 7.23M |
          +-------------+-------------+--------+--------+-------+ generate the format files.  Measurements indicate that
   writing Avro requires 10% more CPU than CBOR or Protocol Buffers.  It might be expected
   appears, therefore, that blocking Avro's advantage in compression CPU usage is exploiting commonality that
   probably offset by a
   general purpose compression engine could also exploit, and the
   figures do indeed bear this out. larger CPU requirement in writing Avro.

C.5.  Conclusions

   The more powerful (and resource-
   consuming) above assessments lead us to the compression, choice of a binary format file
   using blocking.

   As noted previously, this draft anticipates that output data will be
   subject to compression.  There is no compelling case for one
   particular binary serialisation format in terms of either final file
   size or machine resources consumed, so the closer choice must be largely
   based on other factors.  CBOR was therefore chosen as the compressed simple file binary
   serialisation format for the reasons listed in Section 6.

C.6.  Block size gets to choice

   Given the compressed chunk file size.  With no compression, choice of a CBOR format using blocking, the blocked output size is typically half that question arises
   of what an appropriate default value for the simple output,
   but as greater degrees maximum number of compression are applied query/
   response pairs in a block should be.  This has two components; what
   is the gap shrinks.
   However, even with impact on performance of using different block sizes in the stronger compressor,
   format file, and what is the chunked output
   remains roughly 5-10% smaller than impact on the simple output.  This, size of the format file
   before and after compression.

   The following table addresses the
   higher gains at lower compression, might be significant, depending on performance question, showing the target environment.

   [TODO: Add data
   impact on reduction in CPU overhead the performance of compressing blocked
   output vs simple output.]

   This was repeated using some other binary representations:

        +-----------------+-------------+--------+--------+-------+ a C++ program converting "test.pcap" to
   C-DNS.  File size is in Mb, resident set size (RSS) in kb.

              +------------+-----------+--------+-----------+
              | Format Block size | Output File size |    lz4    RSS |   gzip User time |    xz
              +------------+-----------+--------+-----------+
              |
        +-----------------+-------------+--------+--------+-------+       1000 | json-simple    133.46 |     189.85M 612.27 | 25.59M     15.25 | 16.03M
              | 9.74M       5000 |     89.85 | avro-simple 676.82 |      43.31M     14.99 | 16.07M
              | 11.92M      10000 | 7.99M     76.87 | 752.40 | avro-block     14.53 |      17.44M
              | 12.94M      20000 | 10.08M     67.86 | 7.18M 750.75 |     14.49 | protobuf-simple
              |      46.02M      40000 | 15.79M     61.88 | 11.59M 736.30 | 7.94M     14.29 |
              | protobuf-block      80000 |      22.08M     58.08 | 15.43M 694.16 | 10.91M     14.28 | 7.40M
              |
        +-----------------+-------------+--------+--------+-------+

   There's not a lot     160000 |     55.94 | 733.84 |     14.44 |
              |     320000 |     54.41 | 799.20 |     13.97 |
              +------------+-----------+--------+-----------+

   Increasing block size, therefore, tends to choose between the three contenders increase maximum RSS a
   little, with simple
   output.  Avro produces the smaller output, CBOR the next and Protocol
   Buffers the largest, but no significant effect (if anything a small reduction) on
   CPU consumption.

   The following figure plots the effect of increasing block size on
   output file size for different is under 10%. However, with
   blocking, while CBOR and Protocol Buffers are again within a few
   percentage points compressions.

   Figure showing effect of each other (though Protocol Buffers now has a
   slight advantage), Avro produces files in block size on file size (PNG) [19]

   Figure showing effect of block size on file size (SVG) [20]

   From the above, there is obviously scope for tuning the default block
   size to the region compression being employed, traffic characteristics,
   frequency of 20% smaller,
   and holds output file rollover etc.  Using a diminishing advantage through increased compression. strong compression,
   block sizes over 10,000 query/response pairs would seem to offer
   limited improvements.

Authors' Addresses

   John Dickinson
   Sinodun IT
   Magdalen Centre
   Oxford Science Park
   Oxford  OX4 4GA

   Email: jad@sinodun.com

   Jim Hague
   Sinodun IT
   Magdalen Centre
   Oxford Science Park
   Oxford  OX4 4GA

   Email: jim@sinodun.com

   Sara Dickinson
   Sinodun IT
   Magdalen Centre
   Oxford Science Park
   Oxford  OX4 4GA

   Email: sara@sinodun.com
   Terry Manderson
   ICANN
   12025 Waterfront Drive
   Suite 300
   Los Angeles  CA 90094-2536

   Email: terry.manderson@icann.org

   John Bond
   ICANN
   12025 Waterfront Drive
   Suite 300
   Los Angeles  CA 90094-2536

   Email: john.bond@icann.org