<pre>Network Working Group                                        B. Trammell
Request for Comments: 5655                                     E. Boschi
Category: Standards Track                                 Hitachi Europe
                                                                 L. Mark
                                                         Fraunhofer IFAM
                                                                T. Zseby
                                                        Fraunhofer FOKUS
                                                               A. Wagner
                                                              ETH Zurich
                                                            October 2009


  <span class="h1">Specification of the IP Flow Information Export (IPFIX) File Format</span>

Abstract

   This document describes a file format for the storage of flow data
   based upon the IP Flow Information Export (IPFIX) protocol.  It
   proposes a set of requirements for flat-file, binary flow data file
   formats, then specifies the IPFIX File format to meet these
   requirements based upon IPFIX Messages.  This IPFIX File format is
   designed to facilitate interoperability and reusability among a wide
   variety of flow storage, processing, and analysis tools.

Status of This Memo

   This document specifies an Internet standards track protocol for the
   Internet community, and requests discussion and suggestions for
   improvements.  Please refer to the current edition of the "Internet
   Official Protocol Standards" (STD 1) for the standardization state
   and status of this protocol.  Distribution of this memo is unlimited.

Copyright Notice

   Copyright (c) 2009 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to <a href="https://www.rfc-editor.org/bcp/bcp78">BCP 78</a> and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (<a href="http://trustee.ietf.org/license-info">http://trustee.ietf.org/license-info</a>) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the BSD License.





<span class="grey">Trammell, et al.            Standards Track                     [Page 1]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


Table of Contents

   <a href="#section-1">1</a>. Introduction ....................................................<a href="#page-4">4</a>
      <a href="#section-1.1">1.1</a>. IPFIX Documents Overview ...................................<a href="#page-4">4</a>
   <a href="#section-2">2</a>. Terminology .....................................................<a href="#page-5">5</a>
   <a href="#section-3">3</a>. Design Overview .................................................<a href="#page-6">6</a>
   <a href="#section-4">4</a>. Motivation ......................................................<a href="#page-7">7</a>
   <a href="#section-5">5</a>. Requirements ...................................................<a href="#page-10">10</a>
      <a href="#section-5.1">5.1</a>. Record Format Flexibility .................................<a href="#page-10">10</a>
      <a href="#section-5.2">5.2</a>. Self-Description ..........................................<a href="#page-10">10</a>
      <a href="#section-5.3">5.3</a>. Data Compression ..........................................<a href="#page-11">11</a>
      <a href="#section-5.4">5.4</a>. Indexing and Searching ....................................<a href="#page-11">11</a>
      <a href="#section-5.5">5.5</a>. Error Recovery ............................................<a href="#page-12">12</a>
      <a href="#section-5.6">5.6</a>. Authentication, Confidentiality, and Integrity ............<a href="#page-12">12</a>
      <a href="#section-5.7">5.7</a>. Anonymization and Obfuscation .............................<a href="#page-13">13</a>
      <a href="#section-5.8">5.8</a>. Session Auditability and Replayability ....................<a href="#page-13">13</a>
      <a href="#section-5.9">5.9</a>. Performance Characteristics ...............................<a href="#page-14">14</a>
   <a href="#section-6">6</a>. Applicability ..................................................<a href="#page-14">14</a>
      <a href="#section-6.1">6.1</a>. Storage of IPFIX-Collected Flow Data ......................<a href="#page-14">14</a>
      <a href="#section-6.2">6.2</a>. Storage of NetFlow-V9-Collected Flow Data .................<a href="#page-15">15</a>
      <a href="#section-6.3">6.3</a>. Testing IPFIX Collecting Processes ........................<a href="#page-15">15</a>
      <a href="#section-6.4">6.4</a>. IPFIX Device Diagnostics ..................................<a href="#page-16">16</a>
   <a href="#section-7">7</a>. Detailed File Format Specification .............................<a href="#page-16">16</a>
      <a href="#section-7.1">7.1</a>. File Reader Specification .................................<a href="#page-16">16</a>
      <a href="#section-7.2">7.2</a>. File Writer Specification .................................<a href="#page-17">17</a>
      <a href="#section-7.3">7.3</a>. Specific File Writer Use Cases ............................<a href="#page-18">18</a>
           7.3.1. Collocating a File Writer with a Collecting
                  Process ............................................<a href="#page-18">18</a>
           <a href="#section-7.3.2">7.3.2</a>. Collocating a File Writer with a Metering Process ..19
           <a href="#section-7.3.3">7.3.3</a>. Using IPFIX Files for Archival Storage .............<a href="#page-20">20</a>
           <a href="#section-7.3.4">7.3.4</a>. Using IPFIX Files as Documents .....................<a href="#page-20">20</a>
           <a href="#section-7.3.5">7.3.5</a>. Using IPFIX Files for Testing ......................<a href="#page-21">21</a>
           <a href="#section-7.3.6">7.3.6</a>. Writing IPFIX Files for Device Diagnostics .........<a href="#page-22">22</a>
           <a href="#section-7.3.7">7.3.7</a>. IPFIX File Manipulation ............................<a href="#page-22">22</a>
      <a href="#section-7.4">7.4</a>. Media Type of IPFIX Files .................................<a href="#page-22">22</a>
   <a href="#section-8">8</a>. File Format Metadata Specification .............................<a href="#page-22">22</a>
      <a href="#section-8.1">8.1</a>. Recommended Options Templates for IPFIX Files .............<a href="#page-22">22</a>
           <a href="#section-8.1.1">8.1.1</a>. Message Checksum Options Template ..................<a href="#page-23">23</a>
           <a href="#section-8.1.2">8.1.2</a>. File Time Window Options Template ..................<a href="#page-23">23</a>
           <a href="#section-8.1.3">8.1.3</a>. Export Session Details Options Template ............<a href="#page-24">24</a>
           <a href="#section-8.1.4">8.1.4</a>. Message Details Options Template ...................<a href="#page-26">26</a>
      <a href="#section-8.2">8.2</a>. Recommended Information Elements for IPFIX Files ..........<a href="#page-29">29</a>
           <a href="#section-8.2.1">8.2.1</a>. collectionTimeMilliseconds .........................<a href="#page-29">29</a>
           <a href="#section-8.2.2">8.2.2</a>. collectorCertificate ...............................<a href="#page-29">29</a>
           <a href="#section-8.2.3">8.2.3</a>. exporterCertificate ................................<a href="#page-29">29</a>
           <a href="#section-8.2.4">8.2.4</a>. exportSctpStreamId .................................<a href="#page-30">30</a>
           <a href="#section-8.2.5">8.2.5</a>. maxExportSeconds ...................................<a href="#page-30">30</a>
           <a href="#section-8.2.6">8.2.6</a>. maxFlowEndMicroseconds .............................<a href="#page-30">30</a>



<span class="grey">Trammell, et al.            Standards Track                     [Page 2]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


           <a href="#section-8.2.7">8.2.7</a>. maxFlowEndMilliseconds .............................<a href="#page-31">31</a>
           <a href="#section-8.2.8">8.2.8</a>. maxFlowEndNanoseconds ..............................<a href="#page-31">31</a>
           <a href="#section-8.2.9">8.2.9</a>. maxFlowEndSeconds ..................................<a href="#page-32">32</a>
           <a href="#section-8.2.10">8.2.10</a>. messageMD5Checksum ................................<a href="#page-32">32</a>
           <a href="#section-8.2.11">8.2.11</a>. messageScope ......................................<a href="#page-32">32</a>
           <a href="#section-8.2.12">8.2.12</a>. minExportSeconds ..................................<a href="#page-33">33</a>
           <a href="#section-8.2.13">8.2.13</a>. minFlowStartMicroseconds ..........................<a href="#page-33">33</a>
           <a href="#section-8.2.14">8.2.14</a>. minFlowStartMilliseconds ..........................<a href="#page-34">34</a>
           <a href="#section-8.2.15">8.2.15</a>. minFlowStartNanoseconds ...........................<a href="#page-34">34</a>
           <a href="#section-8.2.16">8.2.16</a>. minFlowStartSeconds ...............................<a href="#page-34">34</a>
           <a href="#section-8.2.17">8.2.17</a>. opaqueOctets ......................................<a href="#page-35">35</a>
           <a href="#section-8.2.18">8.2.18</a>. sessionScope ......................................<a href="#page-35">35</a>
   <a href="#section-9">9</a>. Signing and Encryption of IPFIX Files ..........................<a href="#page-36">36</a>
      <a href="#section-9.1">9.1</a>. CMS Detached Signatures ...................................<a href="#page-36">36</a>
           <a href="#section-9.1.1">9.1.1</a>. ContentInfo ........................................<a href="#page-37">37</a>
           <a href="#section-9.1.2">9.1.2</a>. SignedData .........................................<a href="#page-38">38</a>
           <a href="#section-9.1.3">9.1.3</a>. SignerInfo .........................................<a href="#page-38">38</a>
           <a href="#section-9.1.4">9.1.4</a>. EncapsulatedContentInfo ............................<a href="#page-39">39</a>
      <a href="#section-9.2">9.2</a>. Encryption Error Resilience ...............................<a href="#page-39">39</a>
   <a href="#section-10">10</a>. Compression of IPFIX Files ....................................<a href="#page-39">39</a>
      <a href="#section-10.1">10.1</a>. Supported Compression Formats ............................<a href="#page-40">40</a>
      <a href="#section-10.2">10.2</a>. Compression Recognition at the File Reader ...............<a href="#page-40">40</a>
      <a href="#section-10.3">10.3</a>. Compression Error Resilience .............................<a href="#page-40">40</a>
   <a href="#section-11">11</a>. Recommended File Integration Strategies .......................<a href="#page-41">41</a>
      <a href="#section-11.1">11.1</a>. Encapsulation of Non-IPFIX Data in IPFIX Files ...........<a href="#page-41">41</a>
      <a href="#section-11.2">11.2</a>. Encapsulation of IPFIX Files within Other File Formats ...<a href="#page-42">42</a>
   <a href="#section-12">12</a>. Security Considerations .......................................<a href="#page-42">42</a>
      12.1. Relationship between IPFIX File and Transport
            Encryption ...............................................<a href="#page-43">43</a>
      <a href="#section-12.2">12.2</a>. End-to-End Assertions for IPFIX Files ....................<a href="#page-43">43</a>
      12.3. Recommendations for Strength of Cryptography for
            IPFIX Files ..............................................<a href="#page-44">44</a>
   <a href="#section-13">13</a>. IANA Considerations ...........................................<a href="#page-44">44</a>
   <a href="#section-14">14</a>. Acknowledgements ..............................................<a href="#page-46">46</a>
   <a href="#section-15">15</a>. References ....................................................<a href="#page-47">47</a>
      <a href="#section-15.1">15.1</a>. Normative References .....................................<a href="#page-47">47</a>
      <a href="#section-15.2">15.2</a>. Informative References ...................................<a href="#page-48">48</a>
   <a href="#appendix-A">Appendix A</a>.  Example IPFIX File ...................................<a href="#page-49">49</a>
     <a href="#appendix-A.1">A.1</a>.  Example Options Templates .................................<a href="#page-50">50</a>
     <a href="#appendix-A.2">A.2</a>.  Example Supplemental Options Data .........................<a href="#page-52">52</a>
     <a href="#appendix-A.3">A.3</a>.  Example Message Checksum ..................................<a href="#page-54">54</a>
     <a href="#appendix-A.4">A.4</a>.  File Example Data Set .....................................<a href="#page-55">55</a>
     <a href="#appendix-A.5">A.5</a>.  Complete File Example .....................................<a href="#page-55">55</a>
   <a href="#appendix-B">Appendix B</a>.  Applicability of IPFIX Files to NetFlow V9 Flow
                Storage ..............................................<a href="#page-57">57</a>
     <a href="#appendix-B.1">B.1</a>.  Comparing NetFlow V9 to IPFIX .............................<a href="#page-57">57</a>
       <a href="#appendix-B.1.1">B.1.1</a>.  Message Header Format .................................<a href="#page-57">57</a>
       <a href="#appendix-B.1.2">B.1.2</a>.  Set Header Format .....................................<a href="#page-58">58</a>



<span class="grey">Trammell, et al.            Standards Track                     [Page 3]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


       <a href="#appendix-B.1.3">B.1.3</a>.  Template Format .......................................<a href="#page-59">59</a>
       <a href="#appendix-B.1.4">B.1.4</a>.  Information Model .....................................<a href="#page-59">59</a>
       <a href="#appendix-B.1.5">B.1.5</a>.  Template Management ...................................<a href="#page-59">59</a>
       <a href="#appendix-B.1.6">B.1.6</a>.  Transport .............................................<a href="#page-59">59</a>
     <a href="#appendix-B.2">B.2</a>.  A Method for Transforming NetFlow V9 Messages to IPFIX ....<a href="#page-60">60</a>
     <a href="#appendix-B.3">B.3</a>.  NetFlow V9 Transformation Example .........................<a href="#page-61">61</a>

<span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>.  Introduction</span>

   This document specifies a file format based upon IPFIX, designed to
   facilitate interoperability and reusability among a wide variety of
   flow storage, processing, and analysis tools.  It begins with an
   overview of the IPFIX File format, and a quick summary of how IPFIX
   Files work in <a href="#section-3">Section 3</a>.  The detailed specification of the IPFIX
   File format appears in <a href="#section-7">Section 7</a>; this section includes general
   specifications for IPFIX File Readers and IPFIX File Writers and
   specific recommendations for common situations in which they are
   used.  The format makes use of the IPFIX Options mechanism for
   additional file metadata, in order to avoid requiring any protocol
   extensions, and to minimize the effort required to adapt IPFIX
   implementations to use the file format; a detailed definition of the
   Options Templates used for storage metadata appears in <a href="#section-8">Section 8</a>.
   <a href="#appendix-A">Appendix A</a> contains a detailed example IPFIX File.

   An advantage of file-based storage is that files can be readily
   encapsulated within each other and other data storage and
   transmission formats.  The IPFIX File format leverages this to
   provide encryption, described in <a href="#section-9">Section 9</a> and compression, described
   in <a href="#section-10">Section 10</a>.  <a href="#section-11">Section 11</a> provides specific recommendations for
   integration of IPFIX File data with other formats.

   The IPFIX File format was designed to be applicable to a wide variety
   of flow storage situations; the motivation behind its creation is
   described in <a href="#section-4">Section 4</a>.  The document outlines of the set of
   requirements the format is designed to meet in <a href="#section-5">Section 5</a>, and
   explores the applicability of such a format to various specific
   application areas in <a href="#section-6">Section 6</a>.  These sections are intended to give
   background on the development of IPFIX Files.

<span class="h3"><a class="selflink" id="section-1.1" href="#section-1.1">1.1</a>.  IPFIX Documents Overview</span>

   "Specification of the IP Flow Information Export (IPFIX) Protocol for
   the Exchange of IP Traffic Flow Information" [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] and its
   associated documents define the IPFIX protocol, which provides
   network engineers and administrators with access to IP traffic flow
   information.





<span class="grey">Trammell, et al.            Standards Track                     [Page 4]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   "Architecture for IP Flow Information Export" [<a href="./rfc5470" title="&quot;Architecture for IP Flow Information Export&quot;">RFC5470</a>] defines the
   architecture for the export of measured IP flow information out of an
   IPFIX Exporting Process to an IPFIX Collecting Process, and the basic
   terminology used to describe the elements of this architecture, per
   the requirements defined in "Requirements for IP Flow Information
   Export" [<a href="./rfc3917" title="&quot;Requirements for IP Flow Information Export (IPFIX)&quot;">RFC3917</a>].  [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] then covers the details of the method
   for transporting IPFIX Data Records and Templates via a congestion-
   aware transport protocol from an IPFIX Exporting Process to an IPFIX

   Collecting Process.

   "Information Model for IP Flow Information Export" [<a href="./rfc5102" title="&quot;Information Model for IP Flow Information Export&quot;">RFC5102</a>]
   describes the Information Elements used by IPFIX, including details
   on Information Element naming, numbering, and data type encoding.

   "IP Flow Information Export (IPFIX) Applicability" [<a href="./rfc5472" title="&quot;IP Flow Information Export (IPFIX) Applicability&quot;">RFC5472</a>]
   describes the various applications of the IPFIX protocol and their
   use of information exported via IPFIX, and it relates the IPFIX
   architecture to other measurement architectures and frameworks.

   In addition, "Exporting Type Information for IP Flow Information
   Export (IPFIX) Information Elements" [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>] specifies a method for
   encoding Information Model properties within an IPFIX Message stream.

   This document references [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] and [<a href="./rfc5470" title="&quot;Architecture for IP Flow Information Export&quot;">RFC5470</a>] for terminology,
   defines IPFIX File Writer and IPFIX File Reader in terms of the IPFIX
   Exporting Process and IPFIX Collecting Process definitions from
   [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], and extends the IPFIX Information Model defined in
   [<a href="./rfc5102" title="&quot;Information Model for IP Flow Information Export&quot;">RFC5102</a>] to provide new Information Elements for IPFIX File
   metadata.  It uses the method described in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>] to support the
   self-description of IPFIX Files containing enterprise-specific
   Information Elements.

<span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>.  Terminology</span>

   This section defines terminology related to the IPFIX File format.
   In addition, terms used in this document that are defined in the
   "Terminology" section of [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] are to be interpreted as defined
   there.

   IPFIX File:   An IPFIX File is a serialized stream of IPFIX Messages;
      this stream may be stored on a filesystem or transported using any
      technique customarily used for files.  Any IPFIX Message stream
      that would be considered valid when transported over one or more
      of the specified IPFIX transports (Stream Control Transmission
      Protocol (SCTP), TCP, or UDP) as defined in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] is





<span class="grey">Trammell, et al.            Standards Track                     [Page 5]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


      considered an IPFIX File.  However, this document extends that
      definition with recommendations on the construction of IPFIX Files
      that meet the requirements identified in <a href="#section-5">Section 5</a>.

   IPFIX File Reader:   An IPFIX File Reader is a process that reads
      IPFIX Files from a filesystem.  An IPFIX File Reader operates as
      an IPFIX Collecting Process as specified in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], except as
      modified by this document.

   IPFIX File Writer:   An IPFIX File Writer is a process that writes
      IPFIX Files to a filesystem.  An IPFIX File Writer operates as an
      IPFIX Exporting Process as specified in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], except as
      modified by this document.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [<a href="./rfc2119" title="&quot;Key words for use in RFCs to Indicate Requirement Levels&quot;">RFC2119</a>].

<span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>.  Design Overview</span>

   An IPFIX File is simply a data stream containing one or more IPFIX
   Messages serialized to some filesystem.  Though any set of valid
   IPFIX Messages can be serialized into an IPFIX File, the
   specification includes guidelines designed to ease storage and
   retrieval of flow data using the IPFIX File format.

   IPFIX Files contain only IPFIX Messages; any file metadata such as
   checksums or export session details are stored using Options within
   the IPFIX Message.  This design is completely compatible with the
   IPFIX protocol on the wire.  A schematic of a typical IPFIX File is
   shown below:




















<span class="grey">Trammell, et al.            Standards Track                     [Page 6]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


             +=======================================+
             | IPFIX File                            |
             | +===================================+ |
             | | IPFIX Message                     | |
             | | +-------------------------------+ | |
             | | | IPFIX Message Header          | | |
             | | +-------------------------------+ | |
             | | +-------------------------------+ | |
             | | | Options Template Set          | | |
             | | |   Options Template Record     | | |
             | | |           . . .               | | |
             | | +-------------------------------+ | |
             | | +-------------------------------+ | |
             | | | Template Set                  | | |
             | | |   Template Record             | | |
             | | |            . . .              | | |
             | | +-------------------------------+ | |
             | +===================================+ |
             | | IPFIX Message                     | |
             | | +-------------------------------+ | |
             | | | IPFIX Message Header          | | |
             | | +-------------------------------+ | |
             | | +-------------------------------+ | |
             | | | Data Set                      | | |
             | | |   Data Record                 | | |
             | | |            . . .              | | |
             | | +-------------------------------+ | |
             | | +-------------------------------+ | |
             | | | Data Set                      | | |
             | | |   Data Record                 | | |
             | | |            . . .              | | |
             | | +-------------------------------+ | |
             | |              . . .                | |
             | +===================================+ |
             |                . . .                  |
             +=======================================+

                     Figure 1: Typical File Structure

<span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>.  Motivation</span>

   There is a wide variety of applications for the file-based storage of
   IP flow data, across a continuum of time scales.  Tools used in the
   analysis of flow data and creation of analysis products often use
   files as a convenient unit of work, with an ephemeral lifetime.  A
   set of flows relevant to a security investigation may be stored in a
   file for the duration of that investigation, and further exchanged
   among incident handlers via email or within an external incident



<span class="grey">Trammell, et al.            Standards Track                     [Page 7]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   handling workflow application.  Sets of flow data relevant to
   Internet measurement research may be published as files, much as
   libpcap [<a href="#ref-pcap" title="&quot;libpcap (http://www.tcpdump.org/)&quot;">pcap</a>] packet trace files are, to provide common datasets for
   the repeatability of research efforts; these files would have
   lifetimes measured in months or years.  Operational flow measurement
   systems also have a need for long-term, archival storage of flow
   data, either as a primary flow data repository, or as a backing tier
   for online storage in a relational database management system
   (RDBMS).

   The variety of applications of flow data, and the variety of
   presently deployed storage approaches, indicates the need for a
   standard approach to flow storage with applicability across the
   continuum of time scales over which flow data is stored.  A storage
   format based around flat files would best address the variety of
   storage requirements.  While much work has been done on structured
   storage via RDBMS, relational database systems are not a good basis
   for format standardization owing to the fact that their internal data
   structures are generally private to a single implementation and
   subject to change for internal reasons.  Also, there are a wide
   variety of operations available on flat files, and external tools and
   standards can be leveraged to meet file-based flow storage
   requirements.  Further, flow data is often not very semantically
   complicated, and is managed in very high volume; therefore, an RDBMS-
   based flow storage system would not benefit much from the advantages
   of relational database technology.

   The simplest way to create a new file format is simply to serialize
   some internal data model to disk, with either textual or binary
   representation of data elements, and some framing strategy for
   delimiting fields and records.  "Ad hoc" file formats such as this
   have several important disadvantages.  They impose the semantics of
   the data model from which they are derived on the file format, and as
   such, they are difficult to extend, describe, and standardize.

   Indeed, one de facto standard for the storage of flow data is one of
   these ad hoc formats.  A common method of storing data collected via
   Cisco NetFlow is to serialize a stream of raw NetFlow datagrams into
   files.  These NetFlow PDU files consist of a collection of header-
   prefixed blocks (corresponding to the datagrams as received on the
   wire) containing fixed-length binary flow records.  NetFlow V5, V7,
   and V8 data may be mixed within a given file, as the header on each
   datagram defines the NetFlow version of the records following.  While
   this NetFlow PDU file format has all the disadvantages of an ad hoc
   format, and is not extensible to data models other than that defined
   by Cisco NetFlow, it is at least reasonably well understood due to
   its ubiquity.




<span class="grey">Trammell, et al.            Standards Track                     [Page 8]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Over the past decade, XML has emerged as a new "universal"
   representation format for structured data.  It is intended to be
   human readable; indeed, that is one reason for its rapid adoption.
   However, XML has limited usefulness for representing network flow
   data.  Network flow data has a simple, repetitive, non-hierarchical
   structure that does not benefit much from XML.  An XML representation
   of flow data would be an essentially flat list of the attributes and
   their values for each flow record.

   The XML approach to data encoding is very heavyweight when compared
   to binary flow encoding.  XML's use of start- and end-tags, and
   plaintext encoding of the actual values, leads to significant
   inefficiency in encoding size.  Typical network traffic datasets can
   contain millions or billions of flows per hour of traffic
   represented.  Any increase in storage size per record can have
   dramatic impact on flow data storage and transfer sizes.  While data
   compression algorithms can partially remove the redundancy introduced
   by XML encoding, they introduce additional overhead of their own.

   A further problem is that XML processing tools require a full XML
   parser.  XML parsers are fully general and therefore complex,
   resource-intensive, and relatively slow, introducing significant
   processing time overhead for large network-flow datasets.  In
   contrast, parsers for typical binary flow data encodings are simply
   structured, since they only need to parse a very small header and
   then have complete knowledge of all following fields for the
   particular flow.  These can then be read in a very efficient linear
   fashion.

   This leads us to propose the IPFIX Message format as the basis for a
   new flow data file format.  The IPFIX Working Group, in defining the
   IPFIX protocol, has already defined an information model and data
   formatting rules for representation of flow data.  Especially at
   shorter time scales, when a file is a unit of data interchange, the
   filesystem may be viewed as simply another IPFIX Message transport
   between processes.  This format is especially well suited to
   representing flow data, as it was designed specifically for flow data
   export; it is easily extensible, unlike ad hoc serialization, and
   compact, unlike XML.  In addition, IPFIX is an IETF Standards-Track
   protocol for the export and collection of flow data; using a common
   format for storage and analysis at the collection side allows
   implementors to use substantially the same information model and data
   formatting implementation for transport as well as storage.








<span class="grey">Trammell, et al.            Standards Track                     [Page 9]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>.  Requirements</span>

   In this section, we outline a proposed set of requirements
   [<a href="#ref-SAINT2007" title="&quot;Requirements for a standardized flow storage solution&quot;">SAINT2007</a>] for any persistent storage format for flow data.  First
   and foremost, a flow data file format should support storage across
   the continuum of time scales important to flow storage applications.
   Each of the requirements enumerated in the sections below is broadly
   applicable to flow storage applications, though each may be more
   important at certain time scales.  For each, we first identify the
   requirement, then explain how the IPFIX Message format addresses it,
   or briefly outline the changes that must be made in order for an
   IPFIX-based file format to meet the requirement.

<span class="h3"><a class="selflink" id="section-5.1" href="#section-5.1">5.1</a>.  Record Format Flexibility</span>

   Due to the wide variety of flow attributes collected by different
   network flow attribute measurement systems, the ideal flow storage
   format will not impose a single data model or a specific record type
   on the flows it stores.  The file format must be flexible and
   extensible; that is, it must support the definition of multiple
   record types within the file itself, and must be able to support new
   field types for data within the records in a graceful way.

   IPFIX provides record format flexibility through the use of Templates
   to describe each Data Record, through the use of an IANA Registry to
   define its Information Elements, and through the use of enterprise-
   specific Information Elements.

<span class="h3"><a class="selflink" id="section-5.2" href="#section-5.2">5.2</a>.  Self-Description</span>

   Archived data may be read at a time in the future when any external
   reference to the meaning of the data may be lost.  The ideal flow
   storage format should be self-describing; that is, a process reading
   flow data from storage should be able to properly interpret the
   stored flows without reference to anything other than standard
   sources (e.g., the standards document describing the file format) and
   the stored flow data itself.

   The IPFIX Message format is partially self-describing; that is, IPFIX
   Templates containing only IANA-assigned Information Elements can be
   completely interpreted according to the IPFIX Information Model
   without additional external data.

   However, Templates containing private information elements lack
   detailed type and semantic information; a Collecting Process
   receiving Data Records described by a Template containing enterprise-
   specific Information Elements it does not understand can only treat
   the data contained within those Information Elements as octet arrays.



<span class="grey">Trammell, et al.            Standards Track                    [Page 10]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   To be fully self-describing, enterprise-specific Information Elements
   must be additionally described via IPFIX Options according to the
   Information Element Type Options Template defined in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>].

<span class="h3"><a class="selflink" id="section-5.3" href="#section-5.3">5.3</a>.  Data Compression</span>

   Regardless of the representation format, flow data describing traffic
   on real networks tends to be highly compressible.  Compression tends
   to improve the scalability of flow collection systems, by reducing
   the disk storage and I/O bandwidth requirement for a given workload.
   The ideal flow storage format should support applications that wish
   to leverage this fact by supporting compression of stored data.

   The IPFIX Message format has no support for data compression, as the
   IPFIX protocol was designed for speed and simplicity of export.  Of
   course, any flat file is readily compressible using a wide variety of
   external data compression tools, formats, and algorithms; therefore,
   this requirement can be met via encapsulation in one of these
   formats.  <a href="#section-10">Section 10</a> specifies an encapsulation based on bzip2 or
   gzip, to maximize interoperability.

   A few simple optimizations can be made by File Writers to increase
   the integrity and usability of compressed IPFIX data; these are
   outlined in <a href="#section-10.3">Section 10.3</a>.

<span class="h3"><a class="selflink" id="section-5.4" href="#section-5.4">5.4</a>.  Indexing and Searching</span>

   Binary, record-stream-oriented file formats natively support only one
   form of searching: sequential scan in file order.  By choosing the
   order of records in a file carefully (e.g., by flow end time), a file
   can be indexed by a single key.

   Beyond this, properly addressing indexing is an application-specific
   problem, as it inherently involves trade-offs between storage
   complexity and retrieval speed, and requirements vary widely based on
   time scales and the types of queries used from site to site.
   However, a generic standard flow storage format may provide limited
   direct support for indexing and searching.

   The ideal flow storage format will support a limited table of
   contents facility noting that the records in a file contain data
   relating only to certain keys or values of keys, in order to keep
   multi-file search implementations from having to scan a file for data
   it does not contain.

   The IPFIX Message format has no direct support for indexing.
   However, the technique described in "Reducing Redundancy in IP Flow
   Information Export (IPFIX) and Packet Sampling (PSAMP) Reports"



<span class="grey">Trammell, et al.            Standards Track                    [Page 11]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   [<a id="ref-RFC5473">RFC5473</a>] can be used to describe the contents of a file in a limited
   way.  Additionally, as flow data is often sorted and divided by time,
   the start and end time of the flows in a file may be declared using
   the File Time Window Options Template defined in <a href="#section-8.1.2">Section 8.1.2</a>.

<span class="h3"><a class="selflink" id="section-5.5" href="#section-5.5">5.5</a>.  Error Recovery</span>

   When storing flow data for archival purposes, it is important to
   ensure that hardware or software faults do not introduce errors into
   the data over time.  The ideal flow storage format will support the
   detection and correction of encoding-level errors in the data.

   Note that more advanced error correction is best handled at a layer
   below that addressed by this document.  Error correction is a topic
   well addressed by the storage industry in general (e.g., by Redundant
   Array of Independent Disks (RAID) and other technologies).  By
   specifying a flow storage format based upon files, we can leverage
   these features to meet this requirement.

   However, the ideal flow storage format will be resilient against
   errors, providing an internal facility for the detection of errors
   and the ability to isolate errors to as few data records as possible.

   Note that this requirement interacts with the choice of data
   compression or encryption algorithm.  For example, the use of block
   compression algorithms can serve to isolate errors to a single
   compression block, unlike stream compressors, which may fail to
   resynchronize after a single bit error, invalidating the entire
   message stream.

   The IPFIX Message format does not support data integrity assurance.
   It is assumed that advanced error correction will be provided
   externally.  Compression and encryption, if used, provide some
   allowance for detection, if not correction, of errors.  For simple
   error detection support in the absence of compression or encryption,
   checksums may be attached to messages via IPFIX Options according to
   the Message Checksum Options Template defined in <a href="#section-8.1.1">Section 8.1.1</a>.

<span class="h3"><a class="selflink" id="section-5.6" href="#section-5.6">5.6</a>.  Authentication, Confidentiality, and Integrity</span>

   Archival storage of flow data may also require assurance that no
   unauthorized entity can read or modify the stored data.  Cryptography
   can be applied to this problem to ensure integrity and
   confidentiality by signing and encryption.







<span class="grey">Trammell, et al.            Standards Track                    [Page 12]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   As with error correction, this problem has been addressed well at a
   layer below that addressed by this document.  We can leverage the
   fact that existing cryptographic technologies work quite well on data
   stored in files to meet this requirement.

   Beyond support for the use of Transport Layer Security (TLS) for
   transport over TCP or Datagram Transport Layer Security (DTLS) for
   transport over SCTP or UDP, both of which provide transient
   authentication and confidentiality, the IPFIX protocol does not
   support this requirement directly.  The IETF has specified the
   Cryptographic Message Syntax (CMS) [<a href="./rfc3852" title="&quot;Cryptographic Message Syntax (CMS)&quot;">RFC3852</a>] for creating detached
   signatures for integrity and authentication; <a href="#section-9">Section 9</a> specifies a
   CMS-based method for signing IPFIX Files.  Confidentiality protection
   is assumed to be met by methods external to this specification,
   leveraging one of the many such technologies for encrypting files to
   meet specific application and process requirements; however, notes on
   improving archival integrity of encrypted IPFIX Files are given in
   <a href="#section-9.2">Section 9.2</a>.

<span class="h3"><a class="selflink" id="section-5.7" href="#section-5.7">5.7</a>.  Anonymization and Obfuscation</span>

   To ensure the privacy of individuals and organizations at the
   endpoints of communications represented by flow records, it is often
   necessary to obfuscate or anonymize stored and exported flow data.
   The ideal flow storage format will provide for a notation that a
   given information element on a given record type represents
   anonymized, rather than real, data.

   The IPFIX protocol presently has no support for anonymization
   notation.  It should be noted that anonymization is one of the
   requirements given for IPFIX in [<a href="./rfc3917" title="&quot;Requirements for IP Flow Information Export (IPFIX)&quot;">RFC3917</a>].  The decision to qualify
   this requirement with 'MAY' and not 'MUST' in the requirements
   document, and its subsequent lack of specification in the current
   version of the IPFIX protocol, is due to the fact that anonymization
   algorithms are still an open area of research, and that there
   currently exist no standardized methods for anonymization.

   No support is presently defined in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] or this IPFIX-based File
   format for anonymization, as anonymization notation is an area of
   open work for the IPFIX Working Group.

<span class="h3"><a class="selflink" id="section-5.8" href="#section-5.8">5.8</a>.  Session Auditability and Replayability</span>

   Certain use cases for archival flow storage require the storage of
   collection infrastructure details alongside the data itself.  These
   details include information about how and when data was received, and
   where it was received from.  They are useful for auditing as well as
   for the replaying received data for testing purposes.



<span class="grey">Trammell, et al.            Standards Track                    [Page 13]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   The IPFIX protocol contains no direct support for auditability and
   replayability, though the IPFIX Information Model does define various
   Information Elements required to represent collection infrastructure
   details.  These details may be stored in IPFIX Files using the Export
   Session Details Options Template defined in <a href="#section-8.1.3">Section 8.1.3</a>, and the
   Message Details Options Template defined in <a href="#section-8.1.4">Section 8.1.4</a>.

<span class="h3"><a class="selflink" id="section-5.9" href="#section-5.9">5.9</a>.  Performance Characteristics</span>

   The ideal standard flow storage format will not have a significant
   negative impact on the performance of the application generating or
   processing flow data stored in the format.  This is a non-functional
   requirement, but it is important to note that a standard that implies
   a significant performance penalty is unlikely to be widely
   implemented and adopted.

   An examination of the IPFIX protocol would seem to suggest that
   implementations of it are not particularly prone to slowness; indeed,
   a template-based data representation is more easily subject to
   optimization for common cases than representations that embed
   structural information directly in the data stream (e.g., XML).
   However, a full analysis of the impact of using IPFIX Messages as a
   basis for flow data storage on read/write performance will require
   more implementation experience and performance measurement.

<span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>.  Applicability</span>

   This section describes the specific applicability of IPFIX Files to
   various use cases.  IPFIX Files are particularly useful in a flow
   collection and processing infrastructure using IPFIX for flow export.
   We explore the applicability and provide guidelines for using IPFIX
   Files for the storage of flow data collected by IPFIX Collecting
   Processes and NetFlow V9 collectors, the testing of IPFIX Collecting
   Processes, and diagnostics of IPFIX Devices.

<span class="h3"><a class="selflink" id="section-6.1" href="#section-6.1">6.1</a>.  Storage of IPFIX-Collected Flow Data</span>

   IPFIX Files can naturally be used to store flow data collected by an
   IPFIX Collecting Process; indeed, this was one of the primary initial
   motivations behind the file format described within this document.
   Using IPFIX Files as such provides a single, standard, well-
   understood encoding to be used for flow data on disk and on the wire,
   and allows IPFIX implementations to leverage substantially the same
   code for flow export and flow storage.  In addition, the storage of
   single Transport Sessions in IPFIX Files is particularly important
   for network measurement research, allowing repeatability of





<span class="grey">Trammell, et al.            Standards Track                    [Page 14]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   experiments by providing a format for the storage and exchange of
   IPFIX flow trace data much as the libpcap [<a href="#ref-pcap" title="&quot;libpcap (http://www.tcpdump.org/)&quot;">pcap</a>] format is used for
   experiments on packet trace data.

<span class="h3"><a class="selflink" id="section-6.2" href="#section-6.2">6.2</a>.  Storage of NetFlow-V9-Collected Flow Data</span>

   Although the IPFIX protocol is based on the Cisco NetFlow Services,
   Version 9 (NetFlow V9) protocol [<a href="./rfc3954" title="&quot;Cisco Systems NetFlow Services Export Version 9&quot;">RFC3954</a>], the two have diverged
   since work began on IPFIX.  However, since the NetFlow V9 information
   model is a compatible subset of the IPFIX Information Model, it is
   possible to use IPFIX Files to store collected NetFlow V9 flow data.
   This approach may be particularly useful in multi-vendor, multi-
   protocol collection infrastructures using both NetFlow V9 and IPFIX
   to export flow data.

   The applicability of IPFIX Files to this use case is outlined in
   <a href="#appendix-B">Appendix B</a>.

<span class="h3"><a class="selflink" id="section-6.3" href="#section-6.3">6.3</a>.  Testing IPFIX Collecting Processes</span>

   IPFIX Files can be used to store IPFIX Messages for the testing of
   IPFIX Collecting Processes.  A variety of test cases may be stored in
   IPFIX Files.  First, IPFIX data collected in real network
   environments and stored in an IPFIX File can be used as input to
   check the behavior of new or extended implementations of IPFIX
   Collectors.  Furthermore, IPFIX Files can be used to validate the
   operation of a given IPFIX Collecting Process in a new environment,
   i.e., to test with recorded IPFIX data from the target network before
   installing the Collecting Process in the network.

   The IPFIX File format can also be used to store artificial, non-
   compliant reference messages for specific Collecting Process test
   cases.  Examples for such test cases are sets of IPFIX records with
   undefined Information Elements, Data Records described by missing
   Templates, or incorrectly framed Messages or Data Sets.
   Representative error handling test cases are defined in [<a href="./rfc5471" title="&quot;Guidelines for IP Flow Information Export (IPFIX) Testing&quot;">RFC5471</a>].

   Furthermore, fast replay of IPFIX Messages stored in a file can be
   used for stress/load tests (e.g., high rate of incoming Data Records,
   large Templates with high Information Element counts), as described
   in [<a href="./rfc5471" title="&quot;Guidelines for IP Flow Information Export (IPFIX) Testing&quot;">RFC5471</a>].  The provisioning and use of a set of reference files
   for testing simplifies the performance of tests and increases the
   comparability of test results.








<span class="grey">Trammell, et al.            Standards Track                    [Page 15]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="section-6.4" href="#section-6.4">6.4</a>.  IPFIX Device Diagnostics</span>

   As an IPFIX File can be used to store any collection of flows, the
   format may also be used for dumping and storing various types of flow
   data for IPFIX Device diagnostics (e.g., the open flow cache of a
   Metering Process or the flow backlog of an Exporting or Collecting
   Process at the time of a process reset or crash).  File-based storage
   is preferable to remote transmission in such error-recovery
   situations.

<span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>.  Detailed File Format Specification</span>

   Any valid serialized IPFIX Message stream MUST be accepted by a File
   Reader as a valid IPFIX File.  In this way, the filesystem is simply
   treated as another IPFIX transport alongside SCTP, TCP, and UDP,
   albeit a potentially high-latency transport, as the File Reader and
   File Writer do not necessarily run at the same time.

   This section specifies the detailed actions of File Readers and File
   Writers in handling IPFIX Files, and further specifies actions of
   File Writers in specific use cases.  Unless otherwise specified
   herein, IPFIX File Writers MUST behave as IPFIX Exporting Processes,
   and IPFIX File Readers MUST behave as IPFIX Collecting Processes,
   where appropriate.

<span class="h3"><a class="selflink" id="section-7.1" href="#section-7.1">7.1</a>.  File Reader Specification</span>

   An IPFIX File Reader MUST act as an IPFIX Collecting Process as
   specified in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], except as modified by this document.

   An IPFIX File Reader MUST accept as valid any serialized IPFIX
   Message stream that would be considered valid by one or more of the
   other defined IPFIX transport layers.  Practically, this means that
   the union of IPFIX Template management features supported by SCTP,
   TCP, and UDP MUST be supported in IPFIX Files.  File Readers MUST:

   o  accept IPFIX Messages containing Template Sets, Options Template
      Sets, and Data Sets within the same message, as with IPFIX over
      TCP or UDP;

   o  accept Template Sets that define Templates already defined within
      the File, as may occur with retransmission of Templates when using
      IPFIX over UDP as described in <a href="./rfc5101#section-10.3.6">Section&nbsp;10.3.6 of [RFC5101]</a>;

   o  resolve any conflict between a resent definition and a previous
      definition by assuming that the new Template replaces the old, as
      consistent with Template expiration and ID reuse when using UDP at
      the IPFIX transport protocol; and



<span class="grey">Trammell, et al.            Standards Track                    [Page 16]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   o  accept Template Withdrawals as described in <a href="./rfc5101#section-8">Section&nbsp;8 of
      [RFC5101]</a>, provided that the Template to be withdrawn is defined,
      as is the case with IPFIX over TCP and SCTP.

   Considering the filesystem-as-transport view, in the general case, an
   IPFIX File SHOULD be treated as containing a single Transport Session
   as defined by [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>].  However, some applications may benefit from
   the ability to treat a collection of IPFIX Files as a single
   Transport Session; see especially <a href="#section-7.3.3">Section 7.3.3</a> below.  A File Reader
   MAY be configurable to treat a collection of Files as a single
   Transport Session.  However, a File Reader MUST NOT treat a single
   IPFIX File as containing multiple Transport Sessions.

   If an IPFIX File uses the technique described in [<a href="./rfc5473" title="&quot;Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports&quot;">RFC5473</a>] AND all of
   the non-Options Templates in the File contain the commonPropertiesId
   Information Element, a File Reader MAY assume the set of
   commonPropertiesId definitions provides a complete table of contents
   for the File for searching purposes.

<span class="h3"><a class="selflink" id="section-7.2" href="#section-7.2">7.2</a>.  File Writer Specification</span>

   An IPFIX File Writer MUST act as an IPFIX Exporting Process as
   specified in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], except as modified by this document.  This
   section contains specifications for IPFIX File Writers in all
   situations; specifications and recommendations for specific File
   Writer use cases are found in <a href="#section-7.3">Section 7.3</a> below.

   File Writers SHOULD store the Templates and Options required to
   decode the data within the File itself, unless modified by the
   requirements of a specific use case in a subsection of <a href="#section-7.3">Section 7.3</a>.
   In this way, a single IPFIX File generally contains a single notional
   Transport Session as defined by [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>].

   File Writers SHOULD emit each Template Set or Options Template Set to
   appear in the File before any Data Set described by the Templates
   within that Set, to ensure the File Reader can decode every Data Set
   without waiting to process subsequent Templates or Options Templates.

   File Writers SHOULD emit Data Records described by Options Templates
   to appear in the File before any Data Records that depend on the
   scopes defined by those options.

   File Writers SHOULD use Template Withdrawals to withdraw Templates if
   Template IDs need to be reused.  Template Withdrawals SHOULD NOT be
   used unless it is necessary to reuse Template IDs.

   File Writers SHOULD write IPFIX Messages within an IPFIX File in
   ascending Export Time order.



<span class="grey">Trammell, et al.            Standards Track                    [Page 17]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-18" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   File Writers MAY write Data Records to an IPFIX File in any order.
   However, File Writers that write flow records to an IPFIX File in
   flowStartTime or flowEndTime order SHOULD be consistent in this
   ordering within each File.

<span class="h3"><a class="selflink" id="section-7.3" href="#section-7.3">7.3</a>.  Specific File Writer Use Cases</span>

   The specifications in this section apply to specific situations.
   Each section below extends or modifies the base File Writer
   specification in <a href="#section-7.2">Section 7.2</a>.  Considerations for collocation of a
   File Writer with IPFIX Collecting Processes and Metering Processes
   are given, as are specific guidelines for using IPFIX Files for
   archival storage, or as documents.  Also covered are the use of IPFIX
   Files in the testing and diagnostics of IPFIX Devices.

<span class="h4"><a class="selflink" id="section-7.3.1" href="#section-7.3.1">7.3.1</a>.  Collocating a File Writer with a Collecting Process</span>

   When collocating a File Writer with an IPFIX Collecting Process for
   archival storage of collected data in IPFIX Files as described in
   <a href="#section-6.1">Section 6.1</a>, the following recommendations may improve the usefulness
   of the stored data.

   The simplest way for a File Writer to store the data collected in a
   single Transport Session is to simply write the incoming IPFIX
   Messages to an IPFIX File as they are collected.  This approach has
   several drawbacks.  First, if the original Exporting Process did not
   conform to the recommendations in <a href="#section-7.2">Section 7.2</a> with respect to
   Template and Data Record ordering, the written file can be difficult
   to use later; in this case, File Writers MAY reorder records as
   received in order to ensure that Templates appear before the Data
   Records they describe.

   A File Writer collocated with a Collecting Process that starts
   writing data from a running Transport Session SHOULD write all the
   Templates currently active within that Transport Session before
   writing any Data Records described by them.

   Also, the resulting IPFIX Files will lack information about the IPFIX
   Transport Session used to export them, such as the network addresses
   of the Exporting and Collecting Processes and the protocols used to
   transport them.  In this case, if information about the Transport
   Session is required, the File Writer SHOULD store a single IPFIX
   Transport Session in an IPFIX File and SHOULD record information
   about the Transport Session using the Export Session Details Options
   Template described in <a href="#section-8.1.3">Section 8.1.3</a>.






<span class="grey">Trammell, et al.            Standards Track                    [Page 18]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-19" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Additional per-Message information MAY be recorded by the File Writer
   using the Message Details Options Template described in
   <a href="#section-8.1.4">Section 8.1.4</a>.  Per-Message information includes the time at which
   each IPFIX Message was received at the Collecting Process, and can be
   used to resend IPFIX Messages while keeping the original measurement
   plane traffic profile.

   When collocating a File Writer with a Collecting Process, the Export
   Time of each Message SHOULD be the Export Time of the Message
   received by the Collecting Process containing the first Data Record
   in the Message.  Note that File Writers storing IPFIX data collected
   from an IPFIX Collecting Process using SCTP as the transport protocol
   SHOULD interleave messages from multiple streams in order to preserve
   Export Time order, and SHOULD reorder the written messages as
   necessary to ensure that each Template Set or Options Template Set
   appears in the File before any Data Set described by the Templates
   within that Set.  Template reordering MUST preserve the sequence of
   Template Sets with Template Withdrawals in order to ensure
   consistency of Templates.

   Note that when adding additional information to IPFIX Messages
   received from Collecting Processes (e.g., Message Checksum Options,
   Message Detail Options), the File Writer SHOULD extend the length of
   the Message for the additional data if possible; otherwise, the
   Message SHOULD be split into two approximately equal-size Messages
   aligned on a Data Set or Template Set boundary from the original
   Message if possible; otherwise, the Message SHOULD be split into two
   approximately equal-size Messages aligned on a Data Record boundary.
   Note that, since the Maximum Segment Size (MSS) or MTU of most
   network links (1500-9000 for common Ethernets) is smaller than the
   maximum IPFIX Message size (65536) within an IPFIX File, it is
   expected that message length extension will suffice in most
   circumstances.

   A File Writer collocated with a Collecting Process SHOULD NOT sign a
   File as specified in <a href="#section-9.1">Section 9.1</a> unless the Transport Session over
   which the data was exported was protected via TLS or DTLS, and the
   Collecting Process positively identified the Exporting Process by its
   certificate.  See <a href="#section-12.2">Section 12.2</a> for more information on this issue.

<span class="h4"><a class="selflink" id="section-7.3.2" href="#section-7.3.2">7.3.2</a>.  Collocating a File Writer with a Metering Process</span>

   Note that File Writers may also be collocated directly with IPFIX
   Metering Processes, for writing measured information directly to disk
   without intermediate IPFIX Exporting or Collecting Processes.  This
   arrangement may be particularly useful when providing data to an





<span class="grey">Trammell, et al.            Standards Track                    [Page 19]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-20" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   analysis environment with an IPFIX-File-based workflow, when testing
   Metering Processes during development, or when the authentication of
   a Metering Process is important.

   When collocating a File Writer with a Metering Process, note that
   Information Elements associated with Exporting or Collecting
   Processes are meaningless, and SHOULD NOT appear in the Export
   Session Details Options Template described in <a href="#section-8.1.3">Section 8.1.3</a> or the
   Message Details Options Template described in <a href="#section-8.1.4">Section 8.1.4</a>.

   When collocating a File Writer with a Metering Process, the Export
   Time of each Message SHOULD be the time at which the first Data
   Record in the Message was received from the Metering Process.

   Note that collocating a File Writer with a Metering Process is the
   only way to provide positive authentication of a Metering Process
   through signatures as in <a href="#section-9.1">Section 9.1</a>.  See <a href="#section-12.2">Section 12.2</a> for more
   information on this issue.

<span class="h4"><a class="selflink" id="section-7.3.3" href="#section-7.3.3">7.3.3</a>.  Using IPFIX Files for Archival Storage</span>

   While in the general case File Writers should store one Transport
   Session per IPFIX File, some applications storing large collections
   of data over long periods of time may benefit from the ability to
   treat a collection of IPFIX Files as a single Transport Session.  A
   File Writer MAY be configurable to write data from a single Transport
   Session into multiple IPFIX Files; however, File Writers supporting
   such a configuration option MUST provide a configuration option to
   support one-file-per-session behavior for interoperability purposes.

   File Writers using IPFIX Files for archival storage SHOULD support
   compression as in <a href="#section-10">Section 10</a>.

<span class="h4"><a class="selflink" id="section-7.3.4" href="#section-7.3.4">7.3.4</a>.  Using IPFIX Files as Documents</span>

   When IPFIX Files are used as documents, to store a set of flows
   relevant to query, investigation, or other common context, or for the
   publication of traffic datasets relevant to network research, each
   File MUST be readable as a single Transport Session, self-contained
   aside from any detached signature as in <a href="#section-9.1">Section 9.1</a>, and making no
   reference to metadata stored in separate Files, in order to ensure
   interoperability.

   When writing Files to be used as documents, File Writers MAY emit the
   special Data Records described by Options Templates before any other
   Data Records in the File in the following order to ease the
   inspection and use of documents by File Readers:




<span class="grey">Trammell, et al.            Standards Track                    [Page 20]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-21" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   o  Time Window records described by the File Time Window Options
      Template as defined in <a href="#section-8.1.2">Section 8.1.2</a> below; followed by:

   o  Information Element Type Records as described in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>];
      followed by

   o  commonPropertiesId definitions as described in [<a href="./rfc5473" title="&quot;Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports&quot;">RFC5473</a>]; followed
      by

   o  Export Session details records described by the Export Session
      Details Options Template as defined in <a href="#section-8.1.3">Section 8.1.3</a> below.

   The Export Time of each Message within a File used as a document
   SHOULD be the time at which the Message was written by the File
   Writer.

   If an IPFIX File used as a document uses the technique described in
   [<a href="./rfc5473" title="&quot;Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports&quot;">RFC5473</a>] AND all of the non-Options Templates in the File contain
   the commonPropertiesId Information Element, a File Reader MAY assume
   the set of commonPropertiesId definitions provides a complete table
   of contents for the File for searching purposes.

<span class="h4"><a class="selflink" id="section-7.3.5" href="#section-7.3.5">7.3.5</a>.  Using IPFIX Files for Testing</span>

   IPFIX Files can be used for testing IPFIX Collecting Processes in two
   ways.  First, IPFIX Files can be used to store specific flow data for
   regression and stress testing of Collectors; there are no special
   considerations for IPFIX Files used in this way.

   Second, IPFIX Files are useful for storing reference messages that do
   not comply to the IPFIX protocol in order to test the error handling
   and recovery behavior of Collectors.  Of course, IPFIX Files intended
   to be used in this application necessarily MAY violate any of the
   specifications in this document or in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], and such Files MUST
   NOT be transmitted to Collecting Processes or given as input to File
   Readers not under test.

   Note that an extremely simple IPFIX Exporting Process may be crafted
   for testing purposes by simply reading an IPFIX File and transmitting
   it directly to a Collecting Process.  Similarly, an extremely simple
   Collecting Process may be crafted for testing purposes by simply
   accepting connections and/or IPFIX Messages from Exporting Processes
   and writing the session's message stream to an IPFIX File.








<span class="grey">Trammell, et al.            Standards Track                    [Page 21]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-22" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h4"><a class="selflink" id="section-7.3.6" href="#section-7.3.6">7.3.6</a>.  Writing IPFIX Files for Device Diagnostics</span>

   IPFIX Files can be used in the debugging of devices that use flow
   data as internal state, as a common format for the representation of
   flow tables.  In such situations, the opaqueOctets information
   element can be used to store additional non-IPFIX encoded, non-flow
   information (e.g., stack backtraces, process state, etc.) within the
   IPFIX File as in <a href="#section-11.1">Section 11.1</a>; the IPFIX flow table information could
   also be embedded in a larger proprietary diagnostic format using
   delimiters as in <a href="#section-11.2">Section 11.2</a>

<span class="h4"><a class="selflink" id="section-7.3.7" href="#section-7.3.7">7.3.7</a>.  IPFIX File Manipulation</span>

   For many applications, it may prove useful for implementations to
   provide functionality for the manipulation of IPFIX Files; for
   example, to select data from a File, to change the Templates used
   within a File, or to split or join data in Files.  Any such utility
   should take special care to ensure that its output remains a valid
   IPFIX File, specifically with respect to Templates and Options, which
   are scoped to Transport Sessions.

   Any operation that splits one File into multiple Files SHOULD write
   all necessary Templates and Options to each resulting File, and
   ensure that written Options are valid for each resulting File (e.g.,
   the Time Window Options Template in <a href="#section-8.1.2">Section 8.1.2</a>).  Any operation
   that joins multiple Files into a single File should do the same,
   additionally ensuring that Template IDs do not collide, through the
   use of different Observation Domain IDs or Template ID rewriting.
   Combining operations may also want to ensure any desired ordering of
   flow records is maintained.

<span class="h3"><a class="selflink" id="section-7.4" href="#section-7.4">7.4</a>.  Media Type of IPFIX Files</span>

   The media type for IPFIX Files is application/ipfix.  The
   registration information [<a href="./rfc4288" title="&quot;Media Type Specifications and Registration Procedures&quot;">RFC4288</a>] for this media type is given in
   the IANA Considerations section.

<span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>.  File Format Metadata Specification</span>

   This section defines the Options Templates used for IPFIX File
   metadata, and the Information Elements they require.

<span class="h3"><a class="selflink" id="section-8.1" href="#section-8.1">8.1</a>.  Recommended Options Templates for IPFIX Files</span>

   The following Options Templates allow IPFIX Message streams to meet
   the requirements outlined above without extension to the message
   format or protocol.  They are defined in terms of existing
   Information Elements defined in [<a href="./rfc5102" title="&quot;Information Model for IP Flow Information Export&quot;">RFC5102</a>], the Information Elements



<span class="grey">Trammell, et al.            Standards Track                    [Page 22]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-23" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   defined in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>], as well as Information Elements defined in
   <a href="#section-8.2">Section 8.2</a>.  IPFIX File Readers and Writers SHOULD support these
   Options Templates as defined below.

   In addition, IPFIX File Readers and Writers SHOULD support the
   Options Templates defined in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>] in order to support self-
   description of enterprise-specific Information Elements.

<span class="h4"><a class="selflink" id="section-8.1.1" href="#section-8.1.1">8.1.1</a>.  Message Checksum Options Template</span>

   The Message Checksum Options Template specifies the structure of a
   Data Record for attaching an MD5 message checksum to an IPFIX
   Message.  An MD5 message checksum as described MAY be used if data
   integrity is important to the application but file signing is not
   available or desired.  The described Data Record MUST appear only
   once per IPFIX Message, but MAY appear anywhere within the Message.

   This Options Template SHOULD contain the following Information
   Elements:

   +--------------------+----------------------------------------------+
   | IE                 | Description                                  |
   +--------------------+----------------------------------------------+
   | messageScope       | A marker denoting this Option applies to the |
   | [scope]            | whole IPFIX Message; content is ignored.     |
   |                    | This Information Element MUST be defined as  |
   |                    | a Scope Field.                               |
   | messageMD5Checksum | The MD5 checksum of the containing IPFIX     |
   |                    | Message.                                     |
   +--------------------+----------------------------------------------+

<span class="h4"><a class="selflink" id="section-8.1.2" href="#section-8.1.2">8.1.2</a>.  File Time Window Options Template</span>

   The File Time Window Options Template specifies the structure of a
   Data Record for attaching a time window to an IPFIX File; this Data
   Record is referred to as a time window record.  A time window record
   defines the earliest flow start time and the latest flow end time of
   the flow records within a File.  One and only one time window record
   MAY appear within an IPFIX File if the time window information is
   available; a File Writer MUST NOT write more than one time window
   record to an IPFIX File.  A File Writer that writes a time window
   record to a File MUST NOT write any Flow with a start time before the
   beginning of the window or an end time after the end of the window to
   that File.

   This Options Template SHOULD contain the following Information
   Elements:




<span class="grey">Trammell, et al.            Standards Track                    [Page 23]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-24" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   +---------------+---------------------------------------------------+
   | IE            | Description                                       |
   +---------------+---------------------------------------------------+
   | sessionScope  | A marker denoting this Option applies to the      |
   | [scope]       | whole IPFIX Transport Session (i.e., the IPFIX    |
   |               | File in the common case); content is ignored.     |
   |               | This Information Element MUST be defined as a     |
   |               | Scope Field.                                      |
   | minFlowStart* | Exactly one of minFlowStartSeconds,               |
   |               | minFlowStartMilliseconds,                         |
   |               | minFlowStartMicroseconds, or                      |
   |               | minFlowStartNanoseconds SHOULD match the          |
   |               | precision of the accompanying maxFlowEnd*         |
   |               | Information Element.  The start time of the       |
   |               | earliest flow in the Transport Session (i.e.,     |
   |               | File).                                            |
   | maxFlowEnd*   | Exactly one of maxFlowEndSeconds,                 |
   |               | maxFlowEndMilliseconds, maxFlowEndMicroseconds,   |
   |               | or maxFlowEndNanoseconds SHOULD match the         |
   |               | precision of the accompanying minFlowStart*       |
   |               | Information Element.  The end time of the latest  |
   |               | flow in the Transport Session (i.e., File).       |
   +---------------+---------------------------------------------------+

<span class="h4"><a class="selflink" id="section-8.1.3" href="#section-8.1.3">8.1.3</a>.  Export Session Details Options Template</span>

   The Export Session Details Options Template specifies the structure
   of a Data Record for recording the details of an IPFIX Transport
   Session in an IPFIX File.  It is intended for use in storing a single
   complete IPFIX Transport Session in a single IPFIX File.  The
   described Data Record SHOULD appear only once in a given IPFIX File.

   This Options Template SHOULD contain at least the following
   Information Elements, subject to applicability as noted on each
   Information Element:
















<span class="grey">Trammell, et al.            Standards Track                    [Page 24]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-25" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   +----------------------------+--------------------------------------+
   | IE                         | Description                          |
   +----------------------------+--------------------------------------+
   | sessionScope [scope]       | A marker denoting this Option        |
   |                            | applies to the whole IPFIX Transport |
   |                            | Session (i.e., the IPFIX File in the |
   |                            | common case); content is ignored.    |
   |                            | This Information Element MUST be     |
   |                            | defined as a Scope Field.            |
   | exporterIPv4Address        | IPv4 address of the IPFIX Exporting  |
   |                            | Process from which the Messages in   |
   |                            | this Transport Session were          |
   |                            | received.  Present only for          |
   |                            | Exporting Processes with an IPv4     |
   |                            | interface.  For multi-homed SCTP     |
   |                            | associations, this SHOULD be the     |
   |                            | primary path endpoint address of the |
   |                            | Exporting Process.                   |
   | exporterIPv6Address        | IPv6 address of the IPFIX Exporting  |
   |                            | Process from which the Messages in   |
   |                            | this Transport Session were          |
   |                            | received.  Present only for          |
   |                            | Exporting Processes with an IPv6     |
   |                            | interface.  For multi-homed SCTP     |
   |                            | associations, this SHOULD be the     |
   |                            | primary path endpoint address of the |
   |                            | Exporting Process.                   |
   | exporterTransportPort      | The source port from which the       |
   |                            | Messages in this Transport Session   |
   |                            | were received.                       |
   | exporterCertificate        | The certificate used by the IPFIX    |
   |                            | Exporting Process from which the     |
   |                            | Messages in this Transport Session   |
   |                            | were received.  Present only for     |
   |                            | Transport Sessions protected by TLS  |
   |                            | or DTLS.                             |
   | collectorIPv4Address       | IPv4 address of the IPFIX Collecting |
   |                            | Process that received the Messages   |
   |                            | in this Transport Session.  Present  |
   |                            | only for Collecting Processes with   |
   |                            | an IPv4 interface.  For multi-homed  |
   |                            | SCTP associations, this SHOULD be    |
   |                            | the primary path endpoint address of |
   |                            | the Collecting Process.              |
   | collectorIPv6Address       | IPv6 address of the IPFIX Collecting |
   |                            | Process that received the Messages   |
   |                            | in this Transport Session.  Present  |
   |                            | only for Collecting Processes with   |



<span class="grey">Trammell, et al.            Standards Track                    [Page 25]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-26" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   |                            | an IPv6 interface.  For multi-homed  |
   |                            | SCTP associations, this SHOULD be    |
   |                            | the primary path endpoint address of |
   |                            | the Collecting Process.              |
   | collectorTransportPort     | The destination port on which the    |
   |                            | Messages in this Transport Session   |
   |                            | were received.                       |
   | collectorTransportProtocol | The IP Protocol Identifier of the    |
   |                            | transport protocol used to transport |
   |                            | Messages within this Transport       |
   |                            | Session.                             |
   | collectorProtocolVersion   | The version of the export protocol   |
   |                            | used to transport Messages within    |
   |                            | this Transport Session.  Applicable  |
   |                            | only in mixed NetFlow V9-IPFIX       |
   |                            | collection environments when storing |
   |                            | NetFlow V9 data in IPFIX Messages,   |
   |                            | as in <a href="#appendix-B">Appendix B</a>.                    |
   | collectorCertificate       | The certificate used by the IPFIX    |
   |                            | Collecting Process that received the |
   |                            | Messages in this Transport Session.  |
   |                            | Present only for Transport Sessions  |
   |                            | protected by TLS or DTLS.            |
   | minExportSeconds           | The Export Time of the first Message |
   |                            | in the Transport Session.            |
   | maxExportSeconds           | The Export Time of the last Message  |
   |                            | in the Transport Session.            |
   +----------------------------+--------------------------------------+

<span class="h4"><a class="selflink" id="section-8.1.4" href="#section-8.1.4">8.1.4</a>.  Message Details Options Template</span>

   The Message Details Options Template specifies the structure of a
   Data Record for attaching additional export details to an IPFIX
   Message.  These details include the time at which a message was
   received and information about the export and collection
   infrastructure used to transport the Message.  This Options Template
   also allows the storage of the export session metadata provided the
   Export Session Details Options Template, for storing information from
   multiple Transport Sessions in the same IPFIX File.

   This Options Template SHOULD contain at least the following
   Information Elements, subject to applicability as noted for each
   Information Element.  Note that when used in conjunction with the
   Export Session Details Options Template, when storing a single
   complete IPFIX Transport Session in an IPFIX File, this Options
   Template SHOULD contain only the messageScope and





<span class="grey">Trammell, et al.            Standards Track                    [Page 26]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-27" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   collectionTimeMilliseconds Information Elements, and the
   exportSctpStreamId Information Element for Messages transported via
   SCTP.

   +----------------------------+--------------------------------------+
   | IE                         | Description                          |
   +----------------------------+--------------------------------------+
   | messageScope [scope]       | A marker denoting this Option        |
   |                            | applies to the whole IPFIX message;  |
   |                            | content is ignored.  This            |
   |                            | Information Element MUST be defined  |
   |                            | as a Scope Field.                    |
   | collectionTimeMilliseconds | The absolute time at which this      |
   |                            | Message was received by the IPFIX    |
   |                            | Collecting Process.                  |
   | exporterIPv4Address        | IPv4 address of the IPFIX Exporting  |
   |                            | Process from which this Message was  |
   |                            | received.  Present only for          |
   |                            | Exporting Processes with an IPv4     |
   |                            | interface, and if this information   |
   |                            | is not available via the Export      |
   |                            | Session Details Options Template.    |
   |                            | For multi-homed SCTP associations,   |
   |                            | this SHOULD be the primary path      |
   |                            | endpoint address of the Exporting    |
   |                            | Process.                             |
   | exporterIPv6Address        | IPv6 address of the IPFIX Exporting  |
   |                            | Process from which this Message was  |
   |                            | received.  Present only for          |
   |                            | Exporting Processes with an IPv6     |
   |                            | interface and if this information is |
   |                            | not available via the Export Session |
   |                            | Details Options Template.  For       |
   |                            | multi-homed SCTP associations, this  |
   |                            | SHOULD be the primary path endpoint  |
   |                            | address of the Exporting Process.    |
   | exporterTransportPort      | The source port from which this      |
   |                            | Message was received.  Present only  |
   |                            | if this information is not available |
   |                            | via the Export Session Details       |
   |                            | Options Template.                    |
   | exporterCertificate        | The certificate used by the IPFIX    |
   |                            | Exporting Process from which this    |
   |                            | Message was received.  Present only  |
   |                            | for Transport Sessions protected by  |
   |                            | TLS or DTLS.                         |
   | collectorIPv4Address       | IPv4 address of the IPFIX Collecting |
   |                            | Process that received this Message.  |



<span class="grey">Trammell, et al.            Standards Track                    [Page 27]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-28" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   |                            | Present only for Collecting          |
   |                            | Processes with an IPv4 interface,    |
   |                            | and if this information is not       |
   |                            | available via the Export Session     |
   |                            | Details Options Template.  For       |
   |                            | multi-homed SCTP associations, this  |
   |                            | SHOULD be the primary path endpoint  |
   |                            | address of the Collecting Process.   |
   | collectorIPv6Address       | IPv6 address of the IPFIX Collecting |
   |                            | Process that received this Message.  |
   |                            | Present only for Collecting          |
   |                            | Processes with an IPv6 interface,    |
   |                            | and if this information is not       |
   |                            | available via the Export Session     |
   |                            | Details Options Template.  For       |
   |                            | multi-homed SCTP associations, this  |
   |                            | SHOULD be the primary path endpoint  |
   |                            | address of the Collecting Process.   |
   | collectorTransportPort     | The destination port on which this   |
   |                            | Message was received.  Present only  |
   |                            | if this information is not available |
   |                            | via the Export Session Details       |
   |                            | Options Template.                    |
   | collectorTransportProtocol | The IP Protocol Identifier of the    |
   |                            | transport protocol used to transport |
   |                            | this Message.  Present only if this  |
   |                            | information is not available via the |
   |                            | Export Session Details Options       |
   |                            | Template.                            |
   | collectorProtocolVersion   | The version of the export protocol   |
   |                            | used to transport this Message.      |
   |                            | Present only if necessary and if     |
   |                            | this information is not available    |
   |                            | via the Export Session Details       |
   |                            | Options Template.                    |
   | collectorCertificate       | The certificate used by the IPFIX    |
   |                            | Collecting Process that received     |
   |                            | this Message.  Present only for      |
   |                            | Transport Sessions protected by TLS  |
   |                            | or DTLS.                             |
   | exportSctpStreamId         | The SCTP stream used to transport    |
   |                            | this Message.  Present only if the   |
   |                            | Message was transported via SCTP.    |
   +----------------------------+--------------------------------------+







<span class="grey">Trammell, et al.            Standards Track                    [Page 28]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-29" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="section-8.2" href="#section-8.2">8.2</a>.  Recommended Information Elements for IPFIX Files</span>

   The following Information Elements are used by the Options Templates
   in <a href="#section-8.1">Section 8.1</a> to allow IPFIX Message streams to meet the
   requirements outlined above without extension of the protocol.  IPFIX
   File Readers and Writers SHOULD support these Information Elements as
   defined below.

   In addition, IPFIX File Readers and Writers SHOULD support the
   Information Elements defined in [<a href="./rfc5610" title="&quot;Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements&quot;">RFC5610</a>] in order to support full
   self-description of Information Elements.

<span class="h4"><a class="selflink" id="section-8.2.1" href="#section-8.2.1">8.2.1</a>.  collectionTimeMilliseconds</span>

   Description:   The absolute timestamp at which the data within the
      scope containing this Information Element was received by a
      Collecting Process.  This Information Element SHOULD be bound to
      its containing IPFIX Message via IPFIX Options and the
      messageScope Information Element, as defined below.

   Abstract Data Type:   dateTimeMilliseconds

   ElementId:   258

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.2" href="#section-8.2.2">8.2.2</a>.  collectorCertificate</span>

   Description:   The full X.509 certificate, encoded in ASN.1 DER
      format, used by the Collector when IPFIX Messages were transmitted
      using TLS or DTLS.  This Information Element SHOULD be bound to
      its containing IPFIX Transport Session via an options record and
      the sessionScope Information Element, or to its containing IPFIX
      Message via an options record and the messageScope Information
      Element.

   Abstract Data Type:   octetArray

   ElementId:   274

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.3" href="#section-8.2.3">8.2.3</a>.  exporterCertificate</span>

   Description:   The full X.509 certificate, encoded in ASN.1 DER
      format, used by the Collector when IPFIX Messages were transmitted
      using TLS or DTLS.  This Information Element SHOULD be bound to
      its containing IPFIX Transport Session via an options record and



<span class="grey">Trammell, et al.            Standards Track                    [Page 29]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-30" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


      the sessionScope Information Element, or to its containing IPFIX
      Message via an options record and the messageScope Information
      Element.

   Abstract Data Type:   octetArray

   ElementId:   275

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.4" href="#section-8.2.4">8.2.4</a>.  exportSctpStreamId</span>

   Description:   The value of the SCTP Stream Identifier used by the
      Exporting Process for exporting IPFIX Message data.  This is
      carried in the Stream Identifier field of the header of the SCTP
      DATA chunk containing the IPFIX Message(s).

   Abstract Data Type:   unsigned16

   Data Type Semantics:   identifier

   ElementId:   259

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.5" href="#section-8.2.5">8.2.5</a>.  maxExportSeconds</span>

   Description:   The absolute Export Time of the latest IPFIX Message
      within the scope containing this Information Element.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.

   Abstract Data Type:   dateTimeSeconds

   ElementId:   260

   Status:   current

   Units:   seconds

<span class="h4"><a class="selflink" id="section-8.2.6" href="#section-8.2.6">8.2.6</a>.  maxFlowEndMicroseconds</span>

   Description:   The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope



<span class="grey">Trammell, et al.            Standards Track                    [Page 30]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-31" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

   Abstract Data Type:   dateTimeMicroseconds

   ElementId:   268

   Status:   current

   Units:   microseconds

<span class="h4"><a class="selflink" id="section-8.2.7" href="#section-8.2.7">8.2.7</a>.  maxFlowEndMilliseconds</span>

   Description:   The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the millisecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via IPFIX Options and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with millisecond-
      precision (or better) timestamp Information Elements.

   Abstract Data Type:   dateTimeMilliseconds

   ElementId:   269

   Status:   current

   Units:   milliseconds

<span class="h4"><a class="selflink" id="section-8.2.8" href="#section-8.2.8">8.2.8</a>.  maxFlowEndNanoseconds</span>

   Description:   The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element.  This Information Element SHOULD be bound to its
      containing IPFIX Transport Session via IPFIX Options and the
      sessionScope Information Element.  This Information Element SHOULD
      be used only in Transport Sessions containing Flow Records with
      nanosecond-precision timestamp Information Elements.

   Abstract Data Type:   dateTimeNanoseconds

   ElementId:   270

   Status:   current

   Units:   nanoseconds



<span class="grey">Trammell, et al.            Standards Track                    [Page 31]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-32" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h4"><a class="selflink" id="section-8.2.9" href="#section-8.2.9">8.2.9</a>.  maxFlowEndSeconds</span>

   Description:   The latest absolute timestamp of the last packet
      within any Flow within the scope containing this Information
      Element, rounded up to the second if necessary.  This Information
      Element SHOULD be bound to its containing IPFIX Transport Session
      via IPFIX Options and the sessionScope Information Element.

   Abstract Data Type:   dateTimeSeconds

   ElementId:   261

   Status:   current

   Units:   seconds

<span class="h4"><a class="selflink" id="section-8.2.10" href="#section-8.2.10">8.2.10</a>.  messageMD5Checksum</span>

   Description:   The MD5 checksum of the IPFIX Message containing this
      record.  This Information Element SHOULD be bound to its
      containing IPFIX Message via an options record and the
      messageScope Information Element, as defined below, and SHOULD
      appear only once in a given IPFIX Message.  To calculate the value
      of this Information Element, first buffer the containing IPFIX
      Message, setting the value of this Information Element to all
      zeroes.  Then calculate the MD5 checksum of the resulting buffer
      as defined in [<a href="./rfc1321" title="&quot;The MD5 Message-Digest Algorithm&quot;">RFC1321</a>], place the resulting value in this
      Information Element, and export the buffered message.  This
      Information Element is intended as a simple checksum only;
      therefore collision resistance and algorithm agility are not
      required, and MD5 is an appropriate message digest.

   Abstract Data Type:   octetArray (16 bytes)

   ElementId:   262

   Status:   current

   Reference:   <a href="./rfc1321">RFC 1321</a>, The MD5 Message-Digest Algorithm [<a href="./rfc1321" title="&quot;The MD5 Message-Digest Algorithm&quot;">RFC1321</a>]

<span class="h4"><a class="selflink" id="section-8.2.11" href="#section-8.2.11">8.2.11</a>.  messageScope</span>

   Description:   The presence of this Information Element as scope in
      an Options Template signifies that the options described by the
      Template apply to the IPFIX Message that contains them.  It is
      defined for general purpose message scoping of options, and
      proposed specifically to allow the attachment of checksum and
      collection information to a message via IPFIX Options.  The value



<span class="grey">Trammell, et al.            Standards Track                    [Page 32]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-33" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


      of this Information Element MUST be written as 0 by the File
      Writer or Exporting Process.  The value of this Information
      Element MUST be ignored by the File Reader or the Collecting
      Process.

   Abstract Data Type:   unsigned8

   ElementId:   263

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.12" href="#section-8.2.12">8.2.12</a>.  minExportSeconds</span>

   Description:   The absolute Export Time of the earliest IPFIX Message
      within the scope containing this Information Element.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via an options record and the sessionScope
      Information Element.

   Abstract Data Type:   dateTimeSeconds

   ElementId:   264

   Status:   current

   Units:   seconds

<span class="h4"><a class="selflink" id="section-8.2.13" href="#section-8.2.13">8.2.13</a>.  minFlowStartMicroseconds</span>

   Description:   The earliest absolute timestamp of the first packet
      within any Flow within the scope containing this Information
      Element, rounded down to the microsecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via an options record and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with microsecond-
      precision (or better) timestamp Information Elements.

   Abstract Data Type:   dateTimeMicroseconds

   ElementId:   271

   Status:   current

   Units:   microseconds






<span class="grey">Trammell, et al.            Standards Track                    [Page 33]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-34" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h4"><a class="selflink" id="section-8.2.14" href="#section-8.2.14">8.2.14</a>.  minFlowStartMilliseconds</span>

   Description:   The earliest absolute timestamp of the first packet
      within any Flow within the scope containing this Information
      Element, rounded down to the millisecond if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via an options record and the sessionScope
      Information Element.  This Information Element SHOULD be used only
      in Transport Sessions containing Flow Records with millisecond-
      precision (or better) timestamp Information Elements.

   Abstract Data Type:   dateTimeMilliseconds

   ElementId:   272

   Status:   current

   Units:   milliseconds

<span class="h4"><a class="selflink" id="section-8.2.15" href="#section-8.2.15">8.2.15</a>.  minFlowStartNanoseconds</span>

   Description:   The earliest absolute timestamp of the first packet
      within any Flow within the scope containing this Information
      Element.  This Information Element SHOULD be bound to its
      containing IPFIX Transport Session via an options record and the
      sessionScope Information Element.  This Information Element SHOULD
      be used only in Transport Sessions containing Flow Records with
      nanosecond-precision timestamp Information Elements.

   Abstract Data Type:   dateTimeNanoseconds

   ElementId:   273

   Status:   current

   Units:   nanoseconds

<span class="h4"><a class="selflink" id="section-8.2.16" href="#section-8.2.16">8.2.16</a>.  minFlowStartSeconds</span>

   Description:   The earliest absolute timestamp of the first packet
      within any Flow within the scope containing this Information
      Element, rounded down to the second if necessary.  This
      Information Element SHOULD be bound to its containing IPFIX
      Transport Session via an options record and the sessionScope
      Information Element.

   Abstract Data Type:   dateTimeSeconds




<span class="grey">Trammell, et al.            Standards Track                    [Page 34]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-35" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   ElementId:   265

   Status:   current

   Units:   seconds

<span class="h4"><a class="selflink" id="section-8.2.17" href="#section-8.2.17">8.2.17</a>.  opaqueOctets</span>

   Description:   This Information Element is used to encapsulate non-
      IPFIX data into an IPFIX Message stream, for the purpose of
      allowing a non-IPFIX data processor to store a data stream inline
      within an IPFIX File.  A Collecting Process or File Writer MUST
      NOT try to interpret this binary data.  This Information Element
      differs from paddingOctets as its contents are meaningful in some
      non-IPFIX context, while the contents of paddingOctets MUST be
      0x00 and are intended only for Information Element alignment.

   Abstract Data Type:   octetArray

   ElementId:   266

   Status:   current

<span class="h4"><a class="selflink" id="section-8.2.18" href="#section-8.2.18">8.2.18</a>.  sessionScope</span>

   Description:   The presence of this Information Element as scope in
      an Options Template signifies that the options described by the
      Template apply to the IPFIX Transport Session that contains them.
      Note that as all options are implicitly scoped to Transport
      Session and Observation Domain, this Information Element is
      equivalent to a "null" scope.  It is defined for general purpose
      session scoping of options, and proposed specifically to allow the
      attachment of time window and collection information to an IPFIX
      File via IPFIX Options.  The value of this Information Element
      MUST be written as 0 by the File Writer or Exporting Process.  The
      value of this Information Element MUST be ignored by the File
      Reader or the Collecting Process.

   Abstract Data Type:   unsigned8

   ElementId:   267

   Status:   current








<span class="grey">Trammell, et al.            Standards Track                    [Page 35]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-36" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>.  Signing and Encryption of IPFIX Files</span>

   In order to ensure the integrity of IPFIX Files and the identity of
   IPFIX File Writers, File Writers and File Readers SHOULD provide for
   an interoperable and easily implemented method for signing IPFIX
   Files, and verifying those signatures.  This section specifies method
   via CMS detached signatures.

   Note that while CMS specifies an encapsulation format that can be
   used for encryption as well as signing, no method is specified for
   encapsulation for confidentiality protection.  It is assumed that
   application-specific or process-specific requirements outweigh the
   need for interoperability for encrypted files.

<span class="h3"><a class="selflink" id="section-9.1" href="#section-9.1">9.1</a>.  CMS Detached Signatures</span>

   The Cryptographic Message Syntax (CMS) [<a href="./rfc3852" title="&quot;Cryptographic Message Syntax (CMS)&quot;">RFC3852</a>] defines an
   encapsulation syntax for data protection, used to digitally sign,
   authenticate, or encrypt arbitrary message content.  CMS can also be
   used to create detached signatures, in which the signature is stored
   in a separate file.  This arrangement maximizes interoperability, as
   File Readers that are not aware of CMS detached signatures and have
   no requirement for them can simply ignore them; the content of the
   IPFIX File itself is unchanged by the signature.

   The detached signature file for an IPFIX File SHOULD be stored,
   transported, or otherwise made available (e.g., by FTP or HTTP)
   alongside the signed IPFIX File, with the same filename as the IPFIX
   File, except that the file extension ".p7s" is added to the end,
   conforming to the naming convention in [<a href="./rfc3851" title="&quot;Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.1 Message Specification&quot;">RFC3851</a>].

   Within the detached signature, the CMS ContentInfo type MUST always
   be present, and it MUST encapsulate the CMS SignedData content type,
   which in turn MUST NOT encapsulate the signed IPFIX File content.
   The CMS detached signature is summarized as follows:
















<span class="grey">Trammell, et al.            Standards Track                    [Page 36]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-37" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   ContentInfo {
     contentType          id-signedData, -- (1.2.840.113549.1.7.2)
     content              SignedData
   }

   SignedData {
     version              CMSVersion, -- Always set to 3
     digestAlgorithms     DigestAlgorithmIdentifiers,
     encapContentInfo     EncapsulatedContentInfo,
     certificates         CertificateSet, -- File Writer certificate(s)
     crls                 CertificateRevocationLists, -- Optional
     signerInfos          SET OF SignerInfo -- Only one signer
   }

   SignerInfo {
     version              CMSVersion, -- Always set to 3
     sid                  SignerIdentifier,
     digestAlgorithm      DigestAlgorithmIdentifier,
     signedAttrs          SignedAttributes,
     signatureAlgorithm   SignatureAlgorithmIdentifier,
     signature            SignatureValue,
     unsignedAttrs        UnsignedAttributes
   }

   EncapsulatedContentInfo {
     eContentType         id-data, -- (1.2.840.113549.1.7.1)
     eContent             OCTET STRING  -- Always absent
   }

   The details of the contents of each CMS encapsulation are detailed in
   the subsections below.

<span class="h4"><a class="selflink" id="section-9.1.1" href="#section-9.1.1">9.1.1</a>.  ContentInfo</span>

   [<a id="ref-RFC3852">RFC3852</a>] requires the outer-most encapsulation to be ContentInfo;
   the fields of ContentInfo are as follows:

   contentType:  the type of the associated content.  For the detached
      signature file, the encapsulated type is always SignedData, so the
      id-signedData (1.2.840.113549.1.7.2) object identifier MUST be
      present in this field.

   content:  a SignedData content, detailed in <a href="#section-9.1.2">Section 9.1.2</a>.








<span class="grey">Trammell, et al.            Standards Track                    [Page 37]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-38" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h4"><a class="selflink" id="section-9.1.2" href="#section-9.1.2">9.1.2</a>.  SignedData</span>

   The SignedData content type contains the signature of the IPFIX File
   and information to aid in validation; the fields of SignedData are as
   follows:

   version:  MUST be 3.

   digestAlgorithms:  a collection of one-way hash function identifiers.
      It MUST contain the identifier used by the File Writer to generate
      the digital signature.

   encapContentInfo:  the signed content, including a content type
      identifier.  Since a detached signature is being created, it does
      not encapsulate the IPFIX File.  The EncapsulatedContentInfo is
      detailed in <a href="#section-9.1.4">Section 9.1.4</a>.

   certificates:  a collection of certificates.  It SHOULD include the
      X.509 certificate needed to validate the digital signature file.
      Certification Authority (CA) and File Writer certificates MUST
      conform to the certificate profile specified in [<a href="./rfc5280" title="&quot;Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile&quot;">RFC5280</a>].

   crls:  an optional collection of certificate revocation lists (CRLs).
      It SHOULD NOT contain any CRLs; any CRLs that are present MUST
      conform to the certificate profile specified in [<a href="./rfc5280" title="&quot;Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile&quot;">RFC5280</a>].

   signerInfos:  a collection of per-signer information; this identifies
      the File Writer.  More than one SignerInfo MAY appear to
      facilitate transitions between keys or algorithms.  The SignerInfo
      type is detailed in <a href="#section-9.1.3">Section 9.1.3</a>.

<span class="h4"><a class="selflink" id="section-9.1.3" href="#section-9.1.3">9.1.3</a>.  SignerInfo</span>

   The SignerInfo type identifies the File Writer; the fields of
   SignerInfo are as follows:

   version:  MUST be 3.

   sid:  identifies the File Writer's public key.  This identifier MUST
      match the value included in the subjectKeyIdentifier certificate
      extension on the File Writer's X.509 certificate.

   digestAlgorithm:  identifies the one-way hash function and associated
      parameters used to generate the signature.







<span class="grey">Trammell, et al.            Standards Track                    [Page 38]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-39" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   signedAttrs:  an optional set of attributes that are signed along
      with the content.

   digestAlgorithm:  identifies the digital signature algorithm and
      associated parameters used to generate the signature.

   signature:  the digital signature of the associated file.

   unsignedAttrs:  an optional set of attributes that are not signed.

<span class="h4"><a class="selflink" id="section-9.1.4" href="#section-9.1.4">9.1.4</a>.  EncapsulatedContentInfo</span>

   The EncapsulatedContentInfo structure contains a content type
   identifier.  Since a detached signature is being created, it does not
   encapsulate the IPFIX File.  The fields of EncapsulatedContentInfo
   are as follows:

   eContentType:  an object identifier that uniquely specifies the
      content type.  The content type associated with IPFIX File MUST be
      id-data (1.2.840.113549.1.7.1).

   eContent:  an optional field containing the signed content.  Since
      this is a detached signature, eContent MUST be absent.

<span class="h3"><a class="selflink" id="section-9.2" href="#section-9.2">9.2</a>.  Encryption Error Resilience</span>

   Note that single bit errors in the encrypted data stream can result
   in larger errors in the decrypted stream, depending on the encryption
   scheme used.

   In applications (e.g., archival storage) in which error resilience is
   very important, File Writers SHOULD use an encryption scheme that can
   resynchronize after bit errors.  A common example is a block cipher
   in CBC (Cipher Block Chaining) mode.  In this case, File Writers MAY
   also use the Message Checksum Options Template to attach a checksum
   to each IPFIX Message in the IPFIX File, in order to support the
   recognition of errors in the decrypted data.

<span class="h2"><a class="selflink" id="section-10" href="#section-10">10</a>.  Compression of IPFIX Files</span>

   Network traffic measurement data is generally highly compressible.
   IPFIX Templates tend to increase the information content per record
   by not requiring the export of irrelevant or non-present fields in
   records, and the technique described in [<a href="./rfc5473" title="&quot;Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports&quot;">RFC5473</a>] also reduces the
   export of redundant information.  However, even with these
   techniques, generalized compression can decrease storage requirements
   significantly; therefore, IPFIX File Writers and File Readers SHOULD
   support compression as described in this section.



<span class="grey">Trammell, et al.            Standards Track                    [Page 39]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-40" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="section-10.1" href="#section-10.1">10.1</a>.  Supported Compression Formats</span>

   IPFIX Files support two compression encapsulation formats: bzip2
   [<a href="#ref-bzip2" title="&quot;bzip2 (http://www.bzip.org/)&quot;">bzip2</a>] and gzip [<a href="./rfc1952" title="&quot;GZIP file format specification version 4.3&quot;">RFC1952</a>]. bzip2 provides better compression than
   gzip and, as a block compression algorithm, better error recovery
   characteristics, at the expense of slower compression. gzip is
   potentially a better choice when compression time is an issue.  These
   two algorithms and encapsulation formats were chosen for ubiquity and
   ease of implementation.

   IPFIX File Readers and Writers supporting compression MUST support
   bzip2, and SHOULD support gzip.

<span class="h3"><a class="selflink" id="section-10.2" href="#section-10.2">10.2</a>.  Compression Recognition at the File Reader</span>

   bzip2, gzip, and uncompressed IPFIX Files have distinct magic
   numbers.  IPFIX File Readers SHOULD use these magic numbers to
   determine what compression, if any, is in use for an IPFIX File, and
   invoke the proper decompression. bzip2 files are identified by the
   initial three-octet string 0x42, 0x5A, 0x68 ("BZh"). gzip files are
   identified by the initial two-octet string 0x1F, 0x8B.  IPFIX Files
   are identified by the initial two-octet string 0x00, 0x0A; these are
   the version bytes of the first IPFIX Message header in the File.

<span class="h3"><a class="selflink" id="section-10.3" href="#section-10.3">10.3</a>.  Compression Error Resilience</span>

   Compression at the file level, like encryption, is not particularly
   resilient to errors; in the worst case, a single bit error in a
   stream-compressed file could result in the loss of the entire file.

   Since block compression algorithms that support the identification
   and isolation of blocks containing errors limit the impact of errors
   on the recoverability of compressed data, the use of bzip2 in
   applications where error resilience is important is RECOMMENDED.

   Since the block boundary of a block-compressed IPFIX File may fall in
   the middle of an IPFIX Message, resynchronization of an IPFIX Message
   stream by a File Reader after a compression error requires some care.
   The beginning of an IPFIX Message may be identified by its header
   signature (the Version field of the Message Header, 0x00 0x0A,
   followed by a 16-bit Message Length), but simply searching for the
   first occurrence of the Version field is insufficient, since these
   two bytes may occur in valid IPFIX Template or Data Sets.

   Therefore, we specify the following algorithm for File Readers to
   resynchronize an IPFIX Message Stream after skipping a compressed
   block containing errors:




<span class="grey">Trammell, et al.            Standards Track                    [Page 40]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-41" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   1.  Search after the error for the first occurrence of the octet
       string 0x00, 0x0A (the IPFIX Message Header Version field).

   2.  Treat this field as the beginning of a candidate IPFIX Message.
       Read the two bytes following the Version field as a Message
       Length, and seek to that offset from the beginning of the
       candidate IPFIX Message.

   3.  If the first two octets after the candidate IPFIX Message are
       0x00, 0x0A (i.e., the IPFIX Message Header Version field of the
       next message in the stream), or if the end-of-file is reached
       precisely at the end of the candidate IPFIX Message, presume that
       the candidate IPFIX Message is valid, and begin reading the IPFIX
       File from the start of the candidate IPFIX Message.

   4.  If not, or if the seek reaches end-of-file or another block
       containing errors before finding the end of the candidate
       message, go back to step 1, starting the search two bytes from
       the start of the candidate IPFIX Message.

   The algorithm above will improperly identify a non-message as a
   message approximately 1 in 2^32 times, assuming random IPFIX data.
   It may be expanded to consider multiple candidate IPFIX Messages in
   order to increase reliability.

   In applications (e.g., archival storage) in which error resilience is
   very important, File Writers SHOULD use block compression algorithms,
   and MAY attempt to align IPFIX Messages within compression blocks to
   ease resynchronization after errors.  File Readers SHOULD use the
   resynchronization algorithm above to minimize data loss due to
   compression errors.

<span class="h2"><a class="selflink" id="section-11" href="#section-11">11</a>.  Recommended File Integration Strategies</span>

   This section describes methods for integrating IPFIX File data with
   other file formats.

<span class="h3"><a class="selflink" id="section-11.1" href="#section-11.1">11.1</a>.  Encapsulation of Non-IPFIX Data in IPFIX Files</span>

   At times, it may be useful to export or store non-IPFIX data inline
   in an IPFIX File or Message stream.  To do this cleanly, this data
   must be encapsulated into IPFIX Messages so that an IPFIX File Reader
   or Collecting Process can handle it without any need to interpret it.
   At the same time, this data must not be changed during transmission
   or storage.  The opaqueOctets Information Element, as defined in
   <a href="#section-8.2.17">Section 8.2.17</a>, is provided for this encapsulation.





<span class="grey">Trammell, et al.            Standards Track                    [Page 41]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-42" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Processing the encapsulated non-IPFIX data is left to a separate
   processing mechanisms that can identify encapsulated non-IPFIX data
   in an IPFIX Message Stream, but need not have any other IPFIX
   handling capability, except the ability to skip over all IPFIX
   Messages that do not encapsulate non-IPFIX data.

   The Message Checksum Options Template, described in <a href="#section-8.1.1">Section 8.1.1</a>,
   may be used as a uniform mechanism to identify errors within
   encapsulated data.

   Note that this mechanism can only encapsulate data objects up to
   65,515 octets in length.  If the space available in one IPFIX Message
   is not enough for the amount of data to be encapsulated, then the
   data must be broken into smaller segments that are encapsulated into
   consecutive IPFIX Messages.  Any additional structuring or semantics
   of the raw data is outside the scope of IPFIX and must be implemented
   within the encapsulated binary data itself.  Furthermore, the raw
   encapsulated data cannot be assumed by an IPFIX File Reader to have
   any specific format.

<span class="h3"><a class="selflink" id="section-11.2" href="#section-11.2">11.2</a>.  Encapsulation of IPFIX Files within Other File Formats</span>

   Consequently, it may also be useful to reverse the encapsulation,
   that is, to export or store IPFIX data inline within a non-IPFIX File
   or data stream.  This makes sense when the other file format is not
   compatible with the encapsulation described above in <a href="#section-11.1">Section 11.1</a>.
   Generally speaking, the encapsulation here will be specific to the
   format of the containing file.  For example, IPFIX Files may be
   embedded in XML elements using hex or Base64 encoding, or in raw
   binary files using start and end delimiters or some form of run-
   length encoding.  As there are as many potential encapsulations here
   as there are potential file formats, the specifics of each are out of
   scope for this specification.

<span class="h2"><a class="selflink" id="section-12" href="#section-12">12</a>.  Security Considerations</span>

   The Security Considerations section of [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>], on which the IPFIX
   File format is based, is largely concerned with the proper
   application of TLS and DTLS to ensure confidentiality and integrity
   when exporting IPFIX Messages.  By analogy, this document specifies
   the use of CMS [<a href="./rfc3852" title="&quot;Cryptographic Message Syntax (CMS)&quot;">RFC3852</a>] detached signatures to provide equivalent
   integrity protection to TLS and DTLS in <a href="#section-9.1">Section 9.1</a>.  However, aside
   from merely applying CMS for signatures, there are several security
   issues which much be considered in certain circumstances; these are
   covered in the subsections below.






<span class="grey">Trammell, et al.            Standards Track                    [Page 42]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-43" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="section-12.1" href="#section-12.1">12.1</a>.  Relationship between IPFIX File and Transport Encryption</span>

   The underlying protocol used to exchange the information that will be
   stored using the format proposed in this document must as well apply
   appropriate procedures to guarantee the integrity and confidentiality
   of the exported information.  Such issues are addressed in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>].
   Specifically, IPFIX Files that store data taken from an IPFIX
   Collecting Process using TLS or DTLS for transport security SHOULD be
   signed as in <a href="#section-9.1">Section 9.1</a> and SHOULD be encrypted out of band; storage
   of such flow data without encryption may present a potential breach
   of confidentiality.  Conversely, flow data considered sensitive
   enough to require encryption in storage that is later transmitted
   using IPFIX SHOULD be transmitted using TLS or DTLS for transport
   security.

<span class="h3"><a class="selflink" id="section-12.2" href="#section-12.2">12.2</a>.  End-to-End Assertions for IPFIX Files</span>

   Note that while both TLS and CMS provide the ability to sign an IPFIX
   Transport Session or an IPFIX File, there exists no method for
   protecting data integrity end-to-end in the case in which a
   Collecting Process is collocated with a File Writer.  The channel
   between the Exporting Process to Collecting Process using IPFIX is
   signed by the Exporting Process key and protected via TLS and DTLS,
   while the File is signed by the File Writer key and protected via
   CMS.  The identity of the Exporting Process is not asserted in the
   file, and the records may be modified between the Collecting Process
   and the File Writer.

   There are two potential ways to address this issue.  The first is by
   fiat, and is appropriate only when the application allows the
   Collecting-Process-to-File-Writer channel to be trusted.  In this
   case, the File Writer's signature is an implicit assertion that the
   channel to the Exporting Process was protected, that the Exporting
   Process's signature was verified, and that the data was not changed
   after collection.  For this to work, a File Writer collocated with a
   Collecting Process SHOULD NOT sign a File as specified in <a href="#section-9.1">Section 9.1</a>
   unless the Transport Session over which the data was exported was
   protected via TLS or DTLS, and the Collecting Process positively
   identified the Exporting Process by its certificate.  The File Writer
   SHOULD include the Exporting Process and Collecting Process
   certificates within the File using the Export Session Detail Options
   Template in <a href="#section-8.1.3">Section 8.1.3</a> or the Message Detail Options Template in
   <a href="#section-8.1.4">Section 8.1.4</a> to allow for later verification.

   In situations in which the Collecting Process and/or File Writer
   cannot be trusted, end-to-end integrity can then be approximated by
   collocating the File Writer with the Metering Process, and removing
   the IPFIX protocol completely from the chain.  In this case, the File



<span class="grey">Trammell, et al.            Standards Track                    [Page 43]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-44" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Writer's signature is an implicit assertion that the Metering Process
   is identified and is not tampering with the information as observed
   on the wire.

   Verification of these trust relationships is out of scope for this
   document, and should be considered on a per-implementation basis.

<span class="h3"><a class="selflink" id="section-12.3" href="#section-12.3">12.3</a>.  Recommendations for Strength of Cryptography for IPFIX Files</span>

   Note that when encrypting files for archival storage, the
   cryptographic strength is dependent on the length of time over which
   archival data is expected to be kept.  Long-term storage may require
   re-application of cryptographic protection, periodically resigning
   and reencrypting files with stronger keys.  In this case, it is
   recommended that the existing signed and/or encypted data be
   encapsulated within newer, stronger protection.  See [<a href="./rfc4810" title="&quot;Long-Term Archive Service Requirements&quot;">RFC4810</a>] for a
   discussion of this issue.

<span class="h2"><a class="selflink" id="section-13" href="#section-13">13</a>.  IANA Considerations</span>

   This document specifies the creation of several new IPFIX Information
   Elements in the IPFIX Information Element registry located at
   <a href="http://www.iana.org">http://www.iana.org</a>, as defined in <a href="#section-8.2">Section 8.2</a> above.  IANA has
   assigned the following Information Element numbers for their
   respective Information Elements as specified below:

   o  Information Element number 258 for the collectionTimeMilliseconds
      Information Element.

   o  Information Element number 274 for the collectorCertificate
      Information Element.

   o  Information Element number 275 for the exporterCertificate
      Information Element.

   o  Information Element number 259 for the exportSctpStreamId
      Information Element.

   o  Information Element number 260 for the maxExportSeconds
      Information Element.

   o  Information Element number 268 for the maxFlowEndMicroseconds
      Information Element.

   o  Information Element number 269 for the maxFlowEndMilliseconds
      Information Element.





<span class="grey">Trammell, et al.            Standards Track                    [Page 44]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-45" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   o  Information Element number 270 for the maxFlowEndNanoseconds
      Information Element.

   o  Information Element number 261 for the maxFlowEndSeconds
      Information Element.

   o  Information Element number 262 for the messageMD5Checksum
      Information Element.

   o  Information Element number 263 for the messageScope Information
      Element.

   o  Information Element number 264 for the minExportSeconds
      Information Element.

   o  Information Element number 271 for the minFlowStartMicroseconds
      Information Element.

   o  Information Element number 272 for the minFlowStartMilliseconds
      Information Element.

   o  Information Element number 273 for the minFlowStartNanoseconds
      Information Element.

   o  Information Element number 265 for the minFlowStartSeconds
      Information Element.

   o  Information Element number 266 for the opaqueOctets Information
      Element.

   o  Information Element number 267 for the sessionScope Information
      Element.

   IANA has created the media type application/ipfix for IPFIX data, as
   described by the following registration information:

   Type name:   application

   Subtype name:   ipfix

   Required parameters:   none

   Optional parameters:   none

   Encoding considerations:   IPFIX Files are binary, and therefore must
      be encoded in non-binary contexts.





<span class="grey">Trammell, et al.            Standards Track                    [Page 45]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-46" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Security considerations:   See the Security Considerations
      (<a href="#section-12">Section 12</a>) of <a href="./rfc5655">RFC 5655</a>, and the Security Considerations of
      [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>].

   Interoperability considerations:   See the "Detailed Specification"
      (<a href="#section-7">Section 7</a>) of <a href="./rfc5655">RFC 5655</a>.  The format is designed to be broadly
      interoperable, as any valid stream of IPFIX Messages over any
      transport specified in [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] MUST be recognizable as a valid
      IPFIX File.

   Published specification:   <a href="./rfc5655">RFC 5655</a>, especially <a href="#section-7">Section 7</a>, and
      [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>].

   Applications that use this media type:   Various IPFIX
      implementations (see [<a href="./rfc5153" title="&quot;IP Flow Information Export (IPFIX) Implementation Guidelines&quot;">RFC5153</a>]) support the construction of IPFIX
      File Readers and Writers.

   Additional information:

      Magic number(s):   None, although the first two bytes of any IPFIX
         File are the first two bytes of a message header, the Version
         field, which as of [<a href="./rfc5101" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information&quot;">RFC5101</a>] are always 10 in network byte
         order: 0x00, 0x0A.

      File extension(s):   .ipfix

      Macintosh file type code(s):   none

   Person &amp; email address to contact for further information:   Brian
      Trammell &lt;brian.trammell@hitachi-eu.com&gt; for the authors of <a href="./rfc5655">RFC</a>
      <a href="./rfc5655">5655</a>; Nevil Brownlee &lt;n.brownlee@auckland.ac.nz&gt; for the IPFIX
      Working Group.

   Intended usage:   LIMITED USE

   Restrictions on usage:   none

   Change controller:   Brian Trammell &lt;brian.trammell@hitachi-eu.com&gt;
      for the authors of <a href="./rfc5655">RFC 5655</a>; Nevil Brownlee
      &lt;n.brownlee@auckland.ac.nz&gt; for the IPFIX Working Group.

<span class="h2"><a class="selflink" id="section-14" href="#section-14">14</a>.  Acknowledgements</span>

   Thanks to Maurizio Molina, Tom Kosnar, and Andreas Kind for technical
   assistance with the requirements for a standard flow storage format.
   Thanks to Benoit Claise, Paul Aitken, Andrew Johnson, Gerhard Muenz,
   and Nevil Brownlee for their reviews and feedback.  Thanks to Pasi
   Eronen for pointing out [<a href="./rfc5485" title="&quot;Digital Signatures on Internet-Draft Documents&quot;">RFC5485</a>], and Russ Housley for writing it;



<span class="grey">Trammell, et al.            Standards Track                    [Page 46]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-47" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   it specifies a detached signature format, from which <a href="#section-9.1">Section 9.1</a> is
   largely drawn.  Thanks to the PRISM project for its support of this
   work.

<span class="h2"><a class="selflink" id="section-15" href="#section-15">15</a>.  References</span>

<span class="h3"><a class="selflink" id="section-15.1" href="#section-15.1">15.1</a>.  Normative References</span>

   [<a id="ref-RFC5101">RFC5101</a>]    Claise, B., "Specification of the IP Flow Information
                Export (IPFIX) Protocol for the Exchange of IP Traffic
                Flow Information", <a href="./rfc5101">RFC 5101</a>, January 2008.

   [<a id="ref-RFC5102">RFC5102</a>]    Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
                Meyer, "Information Model for IP Flow Information
                Export", <a href="./rfc5102">RFC 5102</a>, January 2008.

   [<a id="ref-RFC5610">RFC5610</a>]    Boschi, E., Trammell, B., Mark, L., and T. Zseby,
                "Exporting Type Information for IP Flow Information
                Export (IPFIX) Information Elements", <a href="./rfc5610">RFC 5610</a>,
                July 2009.

   [<a id="ref-RFC1321">RFC1321</a>]    Rivest, R., "The MD5 Message-Digest Algorithm",
                <a href="./rfc1321">RFC 1321</a>, April 1992.

   [<a id="ref-RFC1952">RFC1952</a>]    Deutsch, P., Gailly, J-L., Adler, M., Deutsch, L., and
                G. Randers-Pehrson, "GZIP file format specification
                version 4.3", <a href="./rfc1952">RFC 1952</a>, May 1996.

   [<a id="ref-RFC2119">RFC2119</a>]    Bradner, S., "Key words for use in RFCs to Indicate
                Requirement Levels", <a href="https://www.rfc-editor.org/bcp/bcp14">BCP 14</a>, <a href="./rfc2119">RFC 2119</a>, March 1997.

   [<a id="ref-RFC3852">RFC3852</a>]    Housley, R., "Cryptographic Message Syntax (CMS)",
                <a href="./rfc3852">RFC 3852</a>, July 2004.

   [<a id="ref-RFC4810">RFC4810</a>]    Wallace, C., Pordesch, U., and R. Brandner, "Long-Term
                Archive Service Requirements", <a href="./rfc4810">RFC 4810</a>, March 2007.

   [<a id="ref-RFC5280">RFC5280</a>]    Cooper, D., Santesson, S., Farrell, S., Boeyen, S.,
                Housley, R., and W. Polk, "Internet X.509 Public Key
                Infrastructure Certificate and Certificate Revocation
                List (CRL) Profile", <a href="./rfc5280">RFC 5280</a>, May 2008.

   [<a id="ref-bzip2">bzip2</a>]      Seward, J., "bzip2 (<a href="http://www.bzip.org/">http://www.bzip.org/</a>)", March 2008.








<span class="grey">Trammell, et al.            Standards Track                    [Page 47]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-48" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="section-15.2" href="#section-15.2">15.2</a>.  Informative References</span>

   [<a id="ref-RFC3917">RFC3917</a>]    Quittek, J., Zseby, T., Claise, B., and S. Zander,
                "Requirements for IP Flow Information Export (IPFIX)",
                <a href="./rfc3917">RFC 3917</a>, October 2004.

   [<a id="ref-RFC3954">RFC3954</a>]    Claise, B., "Cisco Systems NetFlow Services Export
                Version 9", <a href="./rfc3954">RFC 3954</a>, October 2004.

   [<a id="ref-RFC5153">RFC5153</a>]    Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and
                P. Aitken, "IP Flow Information Export (IPFIX)
                Implementation Guidelines", <a href="./rfc5153">RFC 5153</a>, April 2008.

   [<a id="ref-RFC5470">RFC5470</a>]    Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
                "Architecture for IP Flow Information Export", <a href="./rfc5470">RFC 5470</a>,
                March 2009.

   [<a id="ref-RFC5471">RFC5471</a>]    Schmoll, C., Aitken, P., and B. Claise, "Guidelines for
                IP Flow Information Export (IPFIX) Testing", <a href="./rfc5471">RFC 5471</a>,
                March 2009.

   [<a id="ref-RFC5472">RFC5472</a>]    Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP
                Flow Information Export (IPFIX) Applicability",
                <a href="./rfc5472">RFC 5472</a>, March 2009.

   [<a id="ref-RFC5473">RFC5473</a>]    Boschi, E., Mark, L., and B. Claise, "Reducing
                Redundancy in IP Flow Information Export (IPFIX) and
                Packet Sampling (PSAMP) Reports", <a href="./rfc5473">RFC 5473</a>, March 2009.

   [<a id="ref-SAINT2007">SAINT2007</a>]  Trammell, B., Boschi, E., Mark, L., and T. Zseby,
                "Requirements for a standardized flow storage solution",
                 in Proceedings of the SAINT 2007 workshop on Internet
                Measurement Technology, Hiroshima, Japan, January 2007.

   [<a id="ref-RFC3851">RFC3851</a>]    Ramsdell, B., "Secure/Multipurpose Internet Mail
                Extensions (S/MIME) Version 3.1 Message Specification",
                <a href="./rfc3851">RFC 3851</a>, July 2004.

   [<a id="ref-RFC4288">RFC4288</a>]    Freed, N. and J. Klensin, "Media Type Specifications and
                Registration Procedures", <a href="https://www.rfc-editor.org/bcp/bcp13">BCP 13</a>, <a href="./rfc4288">RFC 4288</a>,
                December 2005.

   [<a id="ref-RFC5485">RFC5485</a>]    Housley, R., "Digital Signatures on Internet-Draft
                Documents", <a href="./rfc5485">RFC 5485</a>, March 2009.

   [<a id="ref-pcap">pcap</a>]       "libpcap (<a href="http://www.tcpdump.org/">http://www.tcpdump.org/</a>)", October 2008.





<span class="grey">Trammell, et al.            Standards Track                    [Page 48]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-49" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h2"><a class="selflink" id="appendix-A" href="#appendix-A">Appendix A</a>.  Example IPFIX File</span>

   In this section we will explore an example IPFIX File that
   demonstrates the various features of the IPFIX File format.  This
   File contains flow records described by a single Template.  This File
   also contains a File Time Window record to note the start and end
   time of the data, and an Export Session Details record to record
   collection infrastructure information.  Each Message within this File
   also contains a Message Checksum record, as this File may be
   externally encrypted and/or stored as an archive.  The structure of
   this File is shown in Figure 2.

             +=================================================+
             | IPFIX Message                       seq. 0      |
             | +---------------------------------------------+ |
             | | Template Set (ID 2)                  1 rec  | |
             | |   Data Tmpl. ID 256                         | |
             | +---------------------------------------------+ |
             | | Options Template Set (ID 3)          3 recs | |
             | |   File Time Window Opt. Tmpl. ID 257        | |
             | |   Message Checksum Opt. Tmpl. ID 259        | |
             | |   Export Session Details Opt. Tmpl. ID 258  | |
             | +---------------------------------------------+ |
             | | Data Set (ID 259) [Message Checksum] 1 rec  | |
             | +---------------------------------------------+ |
             +=================================================+
             | IPFIX Message                       seq. 1      |
             | +---------------------------------------------+ |
             | | Data Set (ID 257) [File Time Window] 1 rec  | |
             | +---------------------------------------------+ |
             | | Data Set (ID 258) [Export Session]   1 rec  | |
             | +---------------------------------------------+ |
             | | Data Set (ID 259) [Message Checksum] 1 rec  | |
             | +---------------------------------------------+ |
             +=================================================+
             | IPFIX Message                       seq. 4      |
             | +---------------------------------------------+ |
             | | Data Set (ID 256)                   50 recs | |
             | |  contains flow data                         | |
             | +---------------------------------------------+ |
             | | Data Set (ID 259) [Message Checksum] 1 rec  | |
             | +---------------------------------------------+ |
             +=================================================+
             | IPFIX Message                       seq. 55     |
             |                    . . .                        |

                     Figure 2: File Example Structure




<span class="grey">Trammell, et al.            Standards Track                    [Page 49]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-50" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   The Template describing the data records contains a flow start
   timestamp, an IPv4 5-tuple, and packet and octet total counts.  The
   Template Set defining this is as shown in Figure 3 below:

                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 2           |          Length =  40         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Template ID = 256        |        Field Count = 8        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| flowStartSeconds      = 150 |       Field Length =  4       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| sourceIPv4Address     =   8 |       Field Length =  4       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| dest.IPv4Address      =  12 |       Field Length =  4       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| sourceTransportPort   =   7 |       Field Length =  2       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| dest.TransportPort    =  11 |       Field Length =  2       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| protocolIdentifier    =   4 |       Field Length =  1       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| octetTotalCount       =  85 |       Field Length =  4       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| packetTotalCount      =  86 |       Field Length =  4       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 3: File Example Data Template

<span class="h3"><a class="selflink" id="appendix-A.1" href="#appendix-A.1">A.1</a>.  Example Options Templates</span>

   This is followed by an Options Template Set containing the Options
   Templates required to read the File: the File Time Window Options
   Template (defined in <a href="#section-8.1.2">Section 8.1.2</a> above), the Export Session Details
   Options Template (defined in <a href="#section-8.1.3">Section 8.1.3</a> above), and the Message
   Checksum Options Template (defined in <a href="#section-8.1.1">Section 8.1.1</a> above).  This
   Options Template Set is shown in Figure 4 and Figure 5 below:













<span class="grey">Trammell, et al.            Standards Track                    [Page 50]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-51" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 3           |          Length =  80         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Template ID = 257        |        Field Count = 3        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    Scope Field Count = 1      |0| sessionScope          = 267 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length = 1        |0| minFlowStartSeconds   = 265 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length = 4        |0| maxFlowEndSeconds     = 261 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length = 4        |      Template ID = 259        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Count = 2         |    Scope Field Count = 1      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| messageScope          = 263 |       Field Length =  1       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| messageMD5Checksum    = 262 |       Field Length = 16       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 4: File Example Options Templates (Time Window and Checksum)




























<span class="grey">Trammell, et al.            Standards Track                    [Page 51]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-52" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID = 258       |         Field Count = 9       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    Scope Field Count = 1      |0| sessionScope          = 267 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  1       |0| exporterIPv4Address   = 130 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  4       |0| collectorIPv4Address  = 211 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  4       |0| exporterTransportPort = 217 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  2       |0| col.TransportPort     = 216 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  2       |0| col.TransportProtocol = 215 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  1       |0| col.ProtocolVersion   = 214 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  1       |0| minExportSeconds      = 264 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  4       |0| maxExportSeconds      = 260 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Field Length =  4       |     set padding (2 octets)    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Figure 5: File Example Options Templates, Continued (Session Details)

<span class="h3"><a class="selflink" id="appendix-A.2" href="#appendix-A.2">A.2</a>.  Example Supplemental Options Data</span>

   Following the Templates required to decode the File is the
   supplemental IPFIX Options information used to describe the File's
   contents and type information.  First comes the File Time Window
   record; it notes that the File contains data from 9 October 2007
   between 00:01:13 and 23:56:27 UTC, and appears as in Figure 6:
















<span class="grey">Trammell, et al.            Standards Track                    [Page 52]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-53" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 257         |          Length =  13         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | sessionScope  |           minFlowStartSeconds
   |       0       |         2007-10-09 00:01:13 UTC           . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |            maxFlowEndSeconds
   . . .           |         2007-10-09 23:56:27 UTC           . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |
   . . .           |
   +-+-+-+-+-+-+-+-+

                    Figure 6: File Example Time Window

   This is followed by information about how the data in the File was
   collected, in the Export Session Details record.  This record notes
   that the session stored in this File was sent via SCTP from an
   Exporter at 192.0.2.30 port 32769 to a Collector at 192.0.2.40 port
   4739, and contains messages exported between 00:01:57 and 23:57:12
   UTC on 9 October 2007; it is represented in its Data Set as in
   Figure 7:



























<span class="grey">Trammell, et al.            Standards Track                    [Page 53]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-54" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                       1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 258         |          Length =  27         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | sessionScope  |           exporterIPv4Address
   |       0       |               192.0.2.30                  . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |           collectorIPv4Address
   . . .           |               192.0.2.31                  . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |     exporterTransportPort     |   cTPort
   . . .           |             32769             |    4739   . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |   cTProtocol  |  cPVersion    |
   . . .           |      132      |     10        |           . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                minExportSeconds                   |
   . . .     2007-10-09 00:01:57 UTC               |           . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                maxExportSeconds                   |
   . . .     2007-10-09 23:57:12 UTC               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 7: File Example Export Session Details

<span class="h3"><a class="selflink" id="appendix-A.3" href="#appendix-A.3">A.3</a>.  Example Message Checksum</span>

   Each IPFIX Message within the File is completed with a Message
   Checksum record; the structure of this record within its Data Set is
   as in Figure 8:

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 259         |          Length =  24         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | messageScope  |                                               |
   |       0       |                                               |
   +-+-+-+-+-+-+-+-+                                               |
   |                       messageMD5Checksum                      |
   |           (16-byte MD5 checksum of options message)           |
   |                                                               |
   |                                                               |
   |               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               |              set padding (3 octets)           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure 8: File Example Message Checksum



<span class="grey">Trammell, et al.            Standards Track                    [Page 54]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-55" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="appendix-A.4" href="#appendix-A.4">A.4</a>.  File Example Data Set</span>

   After the Templates and supplemental Options information comes the
   data itself.  The first record of an example Data Set is shown with
   its message and set headers in Figure 9:

                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Version = 10              |         Length = 1296         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Export Time = 2007-10-09 00:01:57 UTC                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Sequence Number = 4                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Observation Domain ID = 1                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Set ID = 256           |          Length = 1254         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      flowStartSeconds                         |
   |                    2007-10-09 00:01:13 UTC                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      sourceIPv4Address                        |
   |                          192.0.2.2                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    destinationIPv4Address                     |
   |                          192.0.2.3                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      sourceTransportPort      |   destinationTransportPort    |
   |             32770             |               80              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  protocolId   |             totalOctetCount
   |       6       |                  18000                    . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |             totalPacketCount
   . . .           |                    65                     . . .
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                   |             (49 more records)
   . . .           |
   +-+-+-+-+-+-+-+-+

                      Figure 9: File Example Data Set

<span class="h3"><a class="selflink" id="appendix-A.5" href="#appendix-A.5">A.5</a>.  Complete File Example</span>

   Bringing together the examples above and adding message headers as
   appropriate, a hex dump of the first 317 bytes of the example File
   constructed above would appear as in the annotated Figure 10 below.



<span class="grey">Trammell, et al.            Standards Track                    [Page 55]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-56" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


     0:|00 0A 00 A0 47 0A B6 E5 00 00 00 00 00 00 00 01
      [^ first message header (length 160 bytes) --&gt;
    16:|00 02 00 28 01 00 00 08 00 96 00 04 00 08 00 04
      [^ data template set --&gt;
    32: 00 0C 00 04 00 07 00 02 00 0B 00 02 00 04 00 01

    48: 00 55 00 04 00 56 00 04|00 03 00 50 01 01 00 03
                              [^ opt template set --&gt;
    64: 00 01 01 0B 00 01 01 09 00 04 01 05 00 04 01 03

    80: 00 02 00 01 01 07 00 01 01 06 00 10 01 02 00 09

    96: 00 01 01 0B 00 01 00 82 00 04 00 D3 00 04 00 D9

   112: 00 02 00 D8 00 02 00 D7 00 01 00 D0 00 01 01 08

   128: 00 04 01 04 00 04 00 00|01 03 00 18 00 73 F1 12
                              [^ checksum record --&gt;
   144: D6 C7 58 BE 44 E6 60 06 4E 78 74 AE 7D 00 00 00

   176:|00 0A 00 50 47 0A B6 E5 00 00 00 01 00 00 00 01
      [^ second message header (length 80 bytes) --&gt;
   192:|01 01 00 0E 00 47 0A B6 B9 47 0C 07 1B 00|01 02
      [^ time window rec -&gt; [ session detail rec ^ --&gt;
   208: 00 1C 00 C0 00 02 1E 0C 00 02 1F 80 01 12 83 84

   224: 0A 47 0A B6 E5 47 0C 07 48 00|01 03 00 18 00 3E
           [ message checksum record ^ --&gt;
   240: 2B 37 08 CE B2 0E 30 11 32 12 4A 5F E3 AD DB 00

   256:|00 0A 05 10 47 0A B6 E5 00 00 00 06 00 00 00 01
      [^ third message header (length 1296 bytes) --&gt;
   272:|01 00 04 E6|47 0A B6 B9 C0 00 02 02 C0 00 02 03
      [^ set hdr ][^ first data rec --&gt;
   288: 80 02 00 50 06 00 00 46 50 00 00 00 41

                     Figure 10: File Example Hex Dump














<span class="grey">Trammell, et al.            Standards Track                    [Page 56]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-57" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h2"><a class="selflink" id="appendix-B" href="#appendix-B">Appendix B</a>.  Applicability of IPFIX Files to NetFlow V9 Flow Storage</span>

   As the IPFIX Message format is nearly a superset of the NetFlow V9
   packet format, IPFIX Files can be used for store NetFlow V9 data
   relatively easily.  This section describes a method for doing so.
   The differences between the two protocols are outlined in
   <a href="#appendix-B.1">Appendix B.1</a> below.  A simple, lightweight, message-for-message
   translation method for transforming V9 Packets into IPFIX Messages
   for storage within IPFIX Files is described in <a href="#appendix-B.2">Appendix B.2</a>.  An
   example of this translation method is given in <a href="#appendix-B.3">Appendix B.3</a>.

<span class="h3"><a class="selflink" id="appendix-B.1" href="#appendix-B.1">B.1</a>.  Comparing NetFlow V9 to IPFIX</span>

   With a few caveats, the IPFIX protocol is a superset of the NetFlow
   V9 protocol, having evolved from it largely through a process of
   feature addition to bring it into compliance with the IPFIX
   Requirements and the needs of stakeholders within the IPFIX Working
   Group.  This appendix outlines the differences between the two
   protocols.  It is informative only, and presented as an exploration
   of the two protocols to motivate the usage of IPFIX Files to store
   V9-collected flow data.

<span class="h4"><a class="selflink" id="appendix-B.1.1" href="#appendix-B.1.1">B.1.1</a>.  Message Header Format</span>

   Both NetFlow V9 and IPFIX use streams of messages prefixed by a
   message header, though the message header differs significantly
   between the two.  Note that in NetFlow V9 terminology, these messages
   are called packets, and messages must be delimited by datagram
   boundaries.  IPFIX does not have this constraint.  The header formats
   are detailed below:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Version Number          |            Count              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           sysUpTime                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           UNIX Secs                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Sequence Number                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Source ID                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                Figure 11: NetFlow V9 Packet Header Format





<span class="grey">Trammell, et al.            Standards Track                    [Page 57]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-58" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Version Number          |            Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Export Time                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Sequence Number                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Observation Domain ID                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure 12: IPFIX Message Header Format

   Version Number:   The IPFIX Version Number MUST be 10, while the
      NetFlow V9 Version Number MUST be 9.

   Length vs. Count:   The Count field in the NetFlow V9 packet header
      counts records in the message (including Data and Template
      Records), while the Length field in the IPFIX Message Header
      counts octets in the message.  Note that this implies that NetFlow
      V9 collectors must rely on datagram boundaries or some other
      external delimiter; otherwise, they must completely consume a
      message before finding its end.

   System Uptime:   System uptime in milliseconds is exported in the
      NetFlow V9 packet header.  This field is not present in the IPFIX
      Message Header, and must be exported using an IPFIX Option if
      required.

   Export Time:   Aside from being called UNIX Secs in the NetFlow V9
      packet header specification, the export time in seconds since 1
      January 1970 at 0000 UTC appears in both NetFlow V9 and IPFIX
      message headers.

   Sequence Number:   The NetFlow V9 Sequence Number counts packets,
      while the IPFIX Sequence Number counts records in Data Sets.  Both
      are scoped to Observation Domain.

   Observation Domain ID:   Similarly, the NetFlow V9 sourceID has
      become the IPFIX Observation Domain ID.

<span class="h4"><a class="selflink" id="appendix-B.1.2" href="#appendix-B.1.2">B.1.2</a>.  Set Header Format</span>

   Set headers are identical between NetFlow V9 and IPFIX; that is, each
   Set (FlowSet in NetFlow V9 terminology) is prefixed by a 4-byte set
   header containing the Set ID and the length of the set in octets.




<span class="grey">Trammell, et al.            Standards Track                    [Page 58]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-59" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


   Note that the special Set IDs are different between IPFIX and NetFlow
   V9.  IPFIX Template Sets are identified by Set ID 2, while NetFlow V9
   Template FlowSets are identified by Set ID 0.  Similarly, IPFIX
   Options Template Sets are identified by Set ID 3, while NetFlow V9
   Options Template FlowSets are identified by Set ID 1.

   Both protocols reserve Set IDs 0-255, and use Set IDs 256-65535 for
   Data Sets (or FlowSets, in NetFlow V9 terminology).

<span class="h4"><a class="selflink" id="appendix-B.1.3" href="#appendix-B.1.3">B.1.3</a>.  Template Format</span>

   Template FlowSets in NetFlow V9 support a subset of functionality of
   those in IPFIX.  Specifically, NetFlow V9 does not have any support
   for vendor-specific Information Elements as IPFIX does, so there is
   no enterprise bit or facility for associating a private enterprise
   number with an information element.  NetFlow V9 also does not support
   variable-length fields.

   Options Template FlowSets in NetFlow V9 are similar to Options
   Template Sets in IPFIX subject to the same caveats.

<span class="h4"><a class="selflink" id="appendix-B.1.4" href="#appendix-B.1.4">B.1.4</a>.  Information Model</span>

   The NetFlow V9 field type definitions are a compatible subset of, and
   have evolved in concert with, the IPFIX Information Model.  IPFIX
   Information Element identifiers in the range 1-127 are defined by the
   IPFIX Information Model [<a href="./rfc5102" title="&quot;Information Model for IP Flow Information Export&quot;">RFC5102</a>] to be compatible with the
   corresponding NetFlow V9 field types.

<span class="h4"><a class="selflink" id="appendix-B.1.5" href="#appendix-B.1.5">B.1.5</a>.  Template Management</span>

   NetFlow V9 has no concept of a Transport Session as in IPFIX, as
   NetFlow V9 was designed with a connectionless transport in mind.
   Template IDs are therefore scoped to an Exporting Process lifetime
   (i.e., an Exporting Process instance between restarts).  There is no
   facility in NetFlow V9 as in IPFIX for Template withdrawal or
   Template ID reuse.  Template retransmission at the Exporter works as
   in UDP-based IPFIX Exporting Processes.

<span class="h4"><a class="selflink" id="appendix-B.1.6" href="#appendix-B.1.6">B.1.6</a>.  Transport</span>

   In practice, though NetFlow V9 is designed to be transport-
   independent, it is transported only over UDP.  There is no facility
   as in IPFIX for full connection-oriented transport without datagram
   boundaries, due to the use of a record count field as opposed to a
   message length field in the packet header.  There is no support in
   NetFlow V9 for transport layer security via TLS or DTLS.




<span class="grey">Trammell, et al.            Standards Track                    [Page 59]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-60" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


<span class="h3"><a class="selflink" id="appendix-B.2" href="#appendix-B.2">B.2</a>.  A Method for Transforming NetFlow V9 Messages to IPFIX</span>

   This appendix describes a method for transforming NetFlow V9 Packets
   into IPFIX Messages, which can be used to store NetFlow V9 data in
   IPFIX Files.  A process transforming NetFlow V9 Packets into IPFIX
   Messages must handle the fact that NetFlow V9 Packets and IPFIX
   Messages are framed differently, that sequence numbering works
   differently, and that the NetFlow V9 field type definitions are only
   compatible with the IPFIX Information Model below Information Element
   identifier 128.

   For each incoming NetFlow V9 packet, the transformation process must:

   1.  Verify that the Version field of the packet header is 9.

   2.  Verify that the Sequence Number field of the packet header is
       valid.

   3.  Scan the packet to:

       1.  Verify that it contains no Templates with field types outside
           the range 1-127;

       2.  Verify that it contains no FlowSets with Set IDs between 2
           and 255 inclusive;

       3.  Verify that it contains the number of records in FlowSets,
           Template FlowSets, and Options Template FlowSets declared in
           the Count field of the packet header; and

       4.  Count the number of records in Data FlowSets for calculating
           the IPFIX Sequence Number.

   4.  Calculate a Sequence Number for each IPFIX Observation Domain by
       storing the last Sequence Number sent for each Observation Domain
       plus the count of records in Data FlowSets in the previous step
       to be sent as the Sequence Number for the IPFIX Message following
       this one within that Observation Domain.

   5.  Generate a new IPFIX Message Header with:

       1.  a Version field of 10;

       2.  a Length field with the number of octets in the IPFIX
           Message, generally available by subtracting 4 from the length
           of the NetFlow V9 packet as returned from the transport layer
           (accounting for the difference in message header lengths);




<span class="grey">Trammell, et al.            Standards Track                    [Page 60]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-61" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


       3.  the Sequence Number calculated for this message by the
           Sequence Number calculation step; and

       4.  Export Time and Observation Domain ID taken from the UNIX
           secs and Source ID fields of the NetFlow V9 packet header,
           respectively.

   6.  Copy each FlowSet from the Netflow V9 packet to the IPFIX Message
       after the header.  Replace Set ID 0 with Set ID 2 for Template
       Sets, and Set ID 1 with Set ID 3 for Options Template Sets.

   Note that this process loses system uptime information; if such
   information is required, the transformation process will have to
   export that information using IPFIX Options.  This may require a more
   sophisticated transformation process structure.

<span class="h3"><a class="selflink" id="appendix-B.3" href="#appendix-B.3">B.3</a>.  NetFlow V9 Transformation Example</span>

   The following two figures show a single NetFlow V9 packet with
   templates and the corresponding IPFIX Message, exporting a single
   flow record representing 60,303 octets sent from 192.0.2.2 to
   192.0.2.3.  This would be the third packet exported in Observation
   Domain 33 from the NetFlow V9 exporter, containing records starting
   with the 12th record (packet and record sequence numbers count from
   0).

   The ** symbol in the IPFIX example shows those fields that required
   modification from the NetFlow V9 packet by the transformation
   process.






















<span class="grey">Trammell, et al.            Standards Track                    [Page 61]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-62" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Version = 9          |         Count = 2             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               Uptime = 3750405 ms (1:02:30.405)               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Export Time = 1171557627 epoch sec (2007-02-15 16:40:27)    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Sequence Number = 2                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 Observation Domain ID = 33                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Set ID = 0          |       Set Length = 20         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID = 256       |       Field Count = 3         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | IPV4_SRC_ADDR           =   8 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | IPV4_DST_ADDR           =  12 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | IN_BYTES                =   1 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 256         |       Set Length = 16         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         IPV4_SRC_ADDR                         |
   |                           192.0.2.2                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         IPV4_DST_ADDR                         |
   |                           192.0.2.3                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           IN_BYTES                            |
   |                             60303                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 13: Example NetFlow V9 Packet















<span class="grey">Trammell, et al.            Standards Track                    [Page 62]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-63" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


                       1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | **       Version = 10         | **      Length = 52           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Export Time = 1171557627 epoch sec (2007-02-15 16:40:27)    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | **                   Sequence Number = 11                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Observation Domain ID = 33                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | **         Set ID = 2         |       Set Length = 20         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID = 256       |       Field Count  = 3        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| sourceIPv4Address      =  8 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| destinationIPv4Address = 12 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| octetDeltaCount        =  1 |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 256         |       Set Length = 16         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       sourceIPv4Address                       |
   |                           192.0.2.2                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     destinationIPv4Address                    |
   |                           192.0.2.3                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        octetDeltaCount                        |
   |                             60303                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

              Figure 14: Corresponding Example IPFIX Message

















<span class="grey">Trammell, et al.            Standards Track                    [Page 63]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-64" ></span>
<span class="grey"><a href="./rfc5655">RFC 5655</a>                      IPFIX Files                   October 2009</span>


Authors' Addresses

   Brian Trammell
   Hitachi Europe
   c/o ETH Zurich
   Gloriastrasse 35
   8092 Zurich
   Switzerland
   Phone: +41 44 632 70 13
   EMail: brian.trammell@hitachi-eu.com

   Elisa Boschi
   Hitachi Europe
   c/o ETH Zurich
   Gloriastrasse 35
   8092 Zurich
   Switzerland
   Phone: +41 44 632 70 57
   EMail: elisa.boschi@hitachi-eu.com

   Lutz Mark
   Fraunhofer IFAM
   Wiener Str. 12
   28359 Bremen
   Germany
   Phone: +49 421 2246206
   EMail: lutz.mark@ifam.fraunhofer.de

   Tanja Zseby
   Fraunhofer Institute for Open Communication Systems
   Kaiserin-Augusta-Allee 31
   10589 Berlin
   Germany
   Phone: +49 30 3463 7153
   EMail: tanja.zseby@fokus.fraunhofer.de

   Arno Wagner
   ETH Zurich
   Gloriastrasse 35
   8092 Zurich
   Switzerland
   Phone: +41 44 632 70 04
   EMail: arno@wagner.name








Trammell, et al.            Standards Track                    [Page 64]
</pre>