Data class
==========

.. default-domain:: lua

.. class:: Data

    The Data class contains the visibility data, flag masks and meta data.
    
    It may hold information for multiple polarizations, channels and timesteps,
    such that it contains the data for a particular observing field,
    spectral window (band) and antenna-pair (or single dish).
    The data object thus holds a dynamic spectrum for each polarization.
    
    The visibility data can consist of complex values, or single-float
    values representing the real, imaginary, amplitude or phase of the
    visibilities (see :meth:`get_complex_state`).
    
    The Data class is a Lua ``userdata`` object.
    Assigning one Data variable to another will therefore cause both variables
    to point to the same object, and changing one will also change the other.
    To create an independent object, :meth:`copy` can be used.
    
    Internally, AOFlagger uses a copy-on-write-like mechanism for the
    visibility sets, flag masks and meta data. It is not required for users
    of the class to know about these implementation details, but the
    relevant thing to know is that copy or assignment operations are generally
    fast and don't increase memory usage.

Method summary
^^^^^^^^^^^^^^

* Copy & Modification
    - :meth:`~Data.clear_mask`
    - :meth:`~Data.copy`
    - :meth:`~Data.flag_nans`
    - :meth:`~Data.flag_zeros`
    - :meth:`~Data.invert_mask`
    - :meth:`~Data.join_mask`
    - :meth:`~Data.set_mask`
    - :meth:`~Data.set_mask_for_channel_range`
    - :meth:`~Data.set_masked_visibilities`
    - :meth:`~Data.set_polarization_data`
    - :meth:`~Data.set_visibilities`
    - :meth:`~Data.__div`
    - :meth:`~Data.__sub`
* Conversion
    - :meth:`~Data.convert_to_complex`
    - :meth:`~Data.convert_to_polarization`
* Meta-data
    - :meth:`~Data.get_antenna1_index`
    - :meth:`~Data.get_antenna1_name`
    - :meth:`~Data.get_antenna2_index`
    - :meth:`~Data.get_antenna2_name`
    - :meth:`~Data.get_baseline_angle`
    - :meth:`~Data.get_baseline_distance`
    - :meth:`~Data.get_baseline_vector`
    - :meth:`~Data.get_complex_state`
    - :meth:`~Data.get_frequencies`
    - :meth:`~Data.get_polarizations`
    - :meth:`~Data.get_times`
    - :meth:`~Data.has_metadata`
    - :meth:`~Data.is_auto_correlation`
    - :meth:`~Data.is_complex`


Detailed descriptions
^^^^^^^^^^^^^^^^^^^^^

    .. method:: Data.clear_mask(data)
    
        Clear the flag mask. Unflags all visibilities (sets all flags to
        false).
        
        :param data: Data for which the mask is cleared. 
        :type data: :class:`Data`
    
    .. method:: Data.convert_to_complex(data, new_state)
    
        Make a new :class:`Data` object with a different complex-value state.
        Complex input data (data with :meth:`get_complex_state` ==
        ``"complex"``) can be converted to real, imaginary, amplitude or
        phase values.
        Amplitude data (:meth:`get_complex_state` == ``"amplitude"``)
        can also be converted back to complex.
        In that case the phases become zero. Other conversions are not
        implemented and will cause an error.
        
        The complex state of an :class:`Data` object is stored internally and
        can be acquired by calling :meth:`get_complex_state`.
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :param new_state: ``"complex"``, ``"real"``, ``"imaginary"``,
            ``"amplitude"`` or ``"phase"``.
        :type new_state: string
        :return: New object with :meth:`get_complex_state` == ``new_state``
        :rtype: :class:`Data`
       
    .. method:: Data.convert_to_polarization(data, new_polarization)
    
        Make a new :class:`Data` object by converting the polarization.
        If the input data does not hold the polarimetric data to convert
        to the requested polarization, an error is thrown. For example,
        converting to ``"i"`` from data for which
        :meth:`get_polarizations` == ``{"xx","yy"}`` is possible, but
        converting to ``"q"`` from data with
        :meth:`get_polarizations` == ``{"ll"}`` is not.
        
        This method can also be used to extract a single polarization
        from the set of available polarizations, e.g.
        
        .. code-block:: lua
        
            xxdata = data:convert_to_polarization("xx")
            
        for data with :meth:`get_polarizations`
        == ``{"xx", "xy", "yx", "yy"}``.
        
        :param data: input data (unchanged).
        :type data: :class:`Data`
        :param new_polarization: ``"i"``, ``"q"``, ``"u"``, ``"v"``,
            ``"xx"``, ``"xy"``, ``"yx"``, ``"yy"``, ``"rr"``, ``"rl"``,
            ``"lr"`` or ``"ll"``.
        :type new_polarization: string
        
    .. method:: Data.copy(data)
    
        Make a value copy of the data.
        
        :param data: Source data.
        :type data: :class:`Data`
        :return: Value copy of input data.
        :rtype: :class:`Data`
    
    .. method:: Data.flag_nans(data)
    
        Flag visibilities that are 'not a number' (nan) or hold overflow.
        Each polarization is independently searched for nans, and its
        mask is updated for that polarization (this is different from
        :meth:`Data.flag_zeros`).
        
        :since: AOFlagger 3.1.
        
    .. method:: Data.flag_zeros(data)
    
        Flag visibilities that are exactly zero. This corrects for
        correlators that output zeros during faults, such as network problems.
        
        It flags samples when the sum of visibilities over polarizations is
        zero. When it is necessary to flag the polarizations independently, the
        statement should be placed inside a loop, e.g.:
        
        .. code-block:: lua
        
            for _,polarization in ipairs(data.get_polarizations()) do
              pol_data = data:convert_to_polarization(polarization)
              flag_zeros(pol_data)
              data:set_polarization_data(polarization, pol_data)
            end
    
        :param data: Data (modified inplace).
        :type data: :class:`Data`
    
    .. method:: Data.get_antenna1_index(data)
    
        Get first antenna index of the two correlated antennas.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Index of first antenna
        :rtype: integer
    
    .. method:: Data.get_antenna1_name(data)
    
        Get name of first antenna of the two correlated antennas.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Name of first antenna
        :rtype: string
    
    .. method:: Data.get_antenna2_index(data)
    
        Get second antenna index of the two correlated antennas.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Index of second antenna
        :rtype: integer
        
    .. method:: Data.get_antenna2_name(data)
    
        Get name of second antenna of the two correlated antennas.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Name of second antenna
        :rtype: string
    
    .. method:: Data.get_baseline_angle(data)
    
        Get angle of this baseline. This is that angle between
        the line from antenna2 to antenna1 and North.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).
        
        :param data: Input data (unchanged)
        :type data: :class:`Data`
        :return: Baseline angle in radians
        :rtype: number
    
    .. method:: Data.get_baseline_distance(data)
    
        Get distance of the antenna1-antenna2 baseline in meters.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Baseline distance in meters
        :rtype: number
    
    .. method:: Data.get_baseline_vector(data)
    
        Get a table with items ``x``, ``y`` and ``z`` that form the three-
        dimensional vector between antennas 1 and 2.
        Throws an error of the antenna metadata is not available
        (see :meth:`has_metadata`).
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Baseline vector in meters
        :rtype: table
    
    .. method:: Data.get_complex_state(data)
    
        Get the state that the visibilities represent. This can be ``"phase"``,
        ``"amplitude"``, ``"real"``, ``"imaginary"`` or ``"complex"``. When the
        data is complex, each visibility consists of two number. Conversions
        can be performed with :meth:`convert_to_complex`.
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: Complex state of data.
        :rtype: string
    
    .. method:: Data.get_frequencies(data)
    
        Get the frequencies of the channels.
        Throws an error of the spectral window metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: List that maps channel nr to frequency in Hz.
        :rtype: table
    
    .. method:: Data.get_polarizations(data)
    
        Get the list of polarizations provided by the data.
        See :meth:`convert_to_polarization` for the list of possible
        polarization names.
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: List that maps polarization nr to a string.
        :rtype: table
    
    .. method:: Data.get_times(data)
    
        Get the time of each timestep in these data.
        Throws an error of the time metadata is not available
        (see :meth:`has_metadata`).

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: List that maps timestep nr to MJD time in s.
        :rtype: table
    
    .. method:: Data.has_metadata(data)
    
        Returns whether metadata is completely present.
        Not all data formats (or simulations) provide all metadata
        items and some of the other methods (e.g. :meth:`get_times`) may throw
        an error because of this. If this function returns ``true``, all these
        functions will succeed.

        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: ``true`` in case all metadata is available, ``false`` otherwise.
        :rtype: boolean
    
   .. method:: Data.invert_mask(data)
   
        Changes masked values to be unmasked and vice versa.
        
        :param data: Destination data (modified in place)
        :type data: :class:`Data`
    
   .. method:: Data.is_auto_correlation(data)
    
        Determine whether this baseline is an auto-correlation.
        This is the case if :meth:`get_antenna1_index` ==
        :meth:`get_antenna2_index`. Unlike the ``get_antenna*``
        functions, this method won't throw an error when no meta-data is
        available. ``false`` is returned in that case.
        
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: ``true`` when this is an auto-correlation.
        :type: boolean
    
    .. method:: Data.is_complex(data)
    
        :param data: Input data (unchanged).
        :type data: :class:`Data`
        :return: ``true`` when :meth:`get_complex_state` == ``"complex"``
        :type: boolean
    
    .. method:: Data.join_mask(first_data, second_data)
    
        Join two masks together. A flag will be set when it is set in either or
        both of the input data sets.
        
        :param first_data: First mask and destination of operation.
        :type first_data: :class:`Data`
        :param second_data: Second mask (unchanged).
        :type second_data: :class:`Data`
            
    .. method:: Data.set_mask(destination_data, mask_data)
    
        Assign the mask of one :class:`Data` object to another.

        :param first_data: Destination data (changed inplace).
        :type first_data: :class:`Data`
        :param second_data: Source data (unchanged).
        :type second_data: :class:`Data`
    
    .. method:: Data.set_mask_for_channel_range(destination_data, mask_data, freq_start, freq_end)
    
        Partially assign the mask of one :class:`Data` object to another.
        The flag mask of channels within the given frequency range are copied
        from ``mask_data`` to ``destination_data``.
        
        This can for example be useful when a certain channel range should not
        be flagged, by partially copying the initial flags to the mask
        produced by the RFI detection.
        
        :param first_data: Destination data (changed inplace).
        :type first_data: :class:`Data`
        :param second_data: Source data (unchanged).
        :type second_data: :class:`Data`
        :param freq_start: Frequency range start in MHz
        :type freq_start: number
        :param freq_end: Frequency range end in MHz
        :type freq_end: number
    
    .. method:: Data.set_polarization_data(destination_data, polarization, source_data)
    
        Replace one polarization of a :class:`Data` object with some other data.
        The ``source_data`` should have only one polarization. The typical
        use-case for this method is to loop over polarizations and modify
        them one by one, e.g.:
        
        .. code-block:: lua
        
            for _,polarization in ipairs(data:get_polarizations()) do 
              pol_data = input:convert_to_polarization(polarization)
              -- Change pol_data here...
              data:set_polarization_data(polarization, pol_data)
            end
            
        Note that the two data sets should have the same complex state
        (see :meth:`get_complex_state()`). This method copies both the
        mask and the visibilities. The meta-data is unchanged.
    
        :param destination_data: Destination data (changed inplace).
        :type destination_data: :class:`Data`
        :param polarization: Name of polarization to change
        :type polarization: string 
        :param source_data: Source data (unchanged).
        :type source_data: :class:`Data`
       
    .. method:: Data.set_visibilities(destination_data, visibility_data)
    
        Assign the visibility data from one :class:`Data` object to another.
        The flagmask and meta-data are unchanged. The two sets should have
        the same number of polarizations and the same complex state.
        
        :param destination_data: Destination data (changed inplace).
        :type destination_data: :class:`Data`
        :param visibility_data: Source data (unmodified).
        :type visibility_data: :class:`Data`
    
    .. method:: Data.set_masked_visibilities(destination_data, visibility_data)
    
        Conditionally assign the visibility data from one :class:`Data` object
        to another. Only visibilities that are set to `true` in the destination mask
        are altered. The flagmask and meta-data are unchanged. The two sets should have
        the same number of polarizations and the same complex state.
        
        :param destination_data: Destination data (changed inplace).
        :type destination_data: :class:`Data`
        :param visibility_data: Source data (unmodified).
        :type visibility_data: :class:`Data`
    
    .. method:: Data.__gc(data)
    
        Internal method for garbage collection function of the Data class.
        This will free the allocated data when those data are no longer 
        used by other Data objects. Note that AOFlagger will immediately
        clear data when the :meth:`execute` function is finished, even when
        the class hasn't been garbage collected yet. 
        
        TODO add a method ``set_persistent`` to disable this.
    
        :param destination_data: Garbage collected data.
        :type destination_data: :class:`Data`
        
    .. method:: Data.__div(lhs_data, rhs_data)
    
        Special Lua method for the division operator. Performs
        element-wise division of :class:`Data`
        objects. For example, with ``data`` and ``rms`` both of
        type ``Data``, this would assign ``corrected`` to the element-wise
        divided data:
        
        .. code-block:: lua
        
            corrected = data / rms
        
        The parameters ``lhs_data`` and ``rhs_data`` are required to have the same number
        of polarizations and are required to have the same complex state.
        The output data will have the same meta-data and masks as ``lhs_data``.
        
        :param lhs_data: Left-hand side data (not modified)
        :type lhs_data: :class:`Data`
        :param rhs_data: Right-hand side data (not modified)
        :type rhs_data: :class:`Data`
        :returns: Left / right
        :rtype: :class:`Data`
        
    .. method:: Data.__sub(lhs_data, rhs_data)
    
        Lua-special method for element-wise subtraction of :class:`Data`
        objects, e.g.:
        
        .. code-block:: lua
 
            data = data - filtered_data
            
        For information about the requirements and produced meta data, see :meth:`__div`.
                   
        :param lhs_data: Left-hand side data (not modified)
        :type lhs_data: :class:`Data`
        :param rhs_data: Right-hand side data (not modified)
        :type rhs_data: :class:`Data`
        :returns: Left - right
        :rtype: :class:`Data`