2.1.

Data Division and FITS Format

The Hitomi data are organized by sequence number and placed in the archive after processing. A sequence number may contain the entire observation of an object if shorter than 1 day. Observations longer than a day are divided into different sequences on the boundary between days from the start time of the observation. Each observation contains both pointing and incoming slew data for all instruments, subsystems, and housekeeping (HK). Figure 1 shows the different time boundaries. The science data files for each instrument are divided into slew and pointing data and may be further divided by the instrument mode or filter. The data from the instrument and spacecraft HK, as well as from the attitude and orbit, are instead not separated by slew and pointing. If an observation spans more than one sequence, then only the first sequence contains the slew data. The science files within a sequence have a size limited to 2 Gb.

Fig. 1

An observation contains all of the data collected between T_SA and T_EA, which indicate the start and end of the attitude. The T_SS and T_ES values indicate the start and the end of the slew and the T_SO and T_EO values are the start and end of the actual pointing. Within the sequence, only the science data are divided into separate files that contain only slew data or only pointing data.

All Hitomi telemetry data within a sequence are reformatted into FITS without loss of information, using either the BINTABLE or IMAGE extensions or the FITS primary header. The science data are organized in FITS BINTABLE extensions using the EVENT format, where each row contains columns with information specific to an event to characterize the coordinates, the time, and the energy of the event. The HK data are time-ordered FITS BINTABLE tables recording several parameters related to the instruments and subsystems. The files generated from the telemetry are first named FITS files (FFF) and after calibration are named second FITS files (SFF and SFFa). In addition to the events associated with the x-ray source, the science file for each of the instruments contains different types of signal related either to the calibration sources or electronic indicators that are used to derive corrections. The SXS and SXI files contain columns with either a single element or fixed array in one row with information related to one event. The FFF for the HXI and SGD instead contains variable length arrays. The x-ray interaction with the HXI or SGD detectors at any one time produces an ensemble of signals. The number of signals in one interaction (hereafter “occurrence”) is variable and stored in variable length array columns. The path of these signals (belonging to an occurrence) within the detectors is reconstructed to identify an event by the calibration software. The reconstructed events for the HXI and SGD are stored in the SFFa, which do not have variable length array columns.

2.2.

Software and Calibration Data

The calibration and analysis of the Hitomi data require mission-specific tasks, as well as multimission tasks included within HEASoft.¹¹ The Hitomi-specific tasks consist of 78 tools to support various stages of the calibration of the instruments and their subsystems, the data filtering, and to perform simulations for all instruments. Each task is built as an FTOOL (this is the standard within HEASoft) and uses a standard parameter file as interface, the CFITSIO library to access and write FITS files, and a common makefile to build the code. Each of the Hitomi tasks is dedicated to a specific function and is written in C or C++ or in Perl.

The tasks to calibrate the events detected by the different instruments include the assignment of time, energy, and coordinates for each event. The calibration of the subsystems is for the MXS, (used for time-dependent gain correction of the SXS data), the SXS anticoincidence signal, and the CAMS (used to calculate the coordinates for the HXI). There are tasks to calculate an ancillary response file [ARF, the effective area (EA) of the telescope which also includes the detector properties], an instrument response matrix file necessary for the spectral analysis, and also tasks to calculate an exposure map for imaging analysis.¹² Several Perl scripts facilitate the data calibration and data filtering. A raytracing code—to model the soft and hard x-ray telescopes,¹³ as well as some general simulation tasks, to generate events for a specific source, and to model the sky background¹⁴—has also been developed. The analysis of the Hitomi data makes use of the standard multimission software packages existing within HEASoft: XSELECT, to extract events, light curves and/or spectra filtered for a region and/or specific pulse phases or source intensity; XSPEC,¹⁵ XRONOS,¹⁶ and XIMAGE¹⁷ for spectral, timing, and imaging analysis.

The Hitomi software uses the HEASARC CALDB to access the Hitomi calibration information.¹⁸ The Hitomi calibration data are in FITS files and include all the prelaunch information as well as the postlaunch updates. The calibration files are designed to accommodate dependencies either on time or specific mode or other parameters to allow software to retrieve the correct file via the CALDB metadata that lists these dependences. The updates of the calibration information do not require changes in the software but only updates of the CALDB files and metadata.

2.3.

Data Flow: Prepipeline, Pipeline and Archive

The processing data flow consists of two steps: the prepipeline (run at ISAS Japan) that decodes the telemetry data into FITS file and organizes the observations into sequences and the pipeline (run at GSFC/USA) that calibrates and cleans the data and transfers the data to the archives (Fig. 2). In the prepipeline, the Hitomi telemetry data are divided into observations, using the time boundaries, as shown in Fig. 1, and stored in files with a FITS wrapper named raw package telemetry (RPT). The FFF are created by decoding the telemetry from the RPT. The FFF with the additional support files, such as those for the attitude and orbit, are packaged together and labeled with a sequence number (Table 1). These files have the same structure and the same number of columns as the final FITS files, but the prepipeline only populates the columns with the telemetered values (to maintain the original information without any conversion) leaving blank the columns to store the calibrated values. An exception to this is the TIME column containing the time values. This column is calculated using the tasks related to time and included in the Hitomi software package to ensure that the attitude and orbit data are also time-tagged consistently. The pipeline populates the remaining columns in the FITS file with the calibrated information for position and energy (the SFF) and creates high level data product files. The data processing in the pipeline outputs: (a) the unfiltered calibrated data (SFF file also level $1/1\mathrm{a}$), where the information for all events is maintained, i.e., no events are discarded; (b) the good time intervals (GTIs), which exclude unwanted times either derived from orbital or instrument HK parameters; (c) the filtered or cleaned files (level 2), where the data are screened for unwanted event flags and selected within the GTIs; (d) additional files to monitor the instrument performance (trend data); (e) the attitude corrected for aberration; (f) extended orbital information not included in the orbit file; (g) a filter file containing HK information necessary for screening; and (h) the database file containing high level information for each observation used by the archive. The tasks used to process the data in the pipeline as well as to calculate the timing information used in the prepipeline are all part of the Hitomi package so that users may always recalibrate or rescreen the data if new calibration information is available or special screening is needed in their analysis.

Fig. 2

The data processing flow: prepipeline for data preparation, pipeline for data processing, and archive.

Table 1

Data generated by the prepipeline are the RPT and FFF files. Data generated by the pipeline are the levels 1, 1a, 2, and 3 files and they populate the archive.

The pipeline data processing flow consists of a general part applicable to all instruments and a specific part for each individual instrument. The general part applies the aberration correction to the attitude; calculates the average pointing based on the time interval when the telescope was on source; derives the times for the South Atlantic Anomaly, the satellite day and night, cutoff rigidity, and other orbital quantities necessary for screening the data; and calculates observation-dependent quantities, as for the optical axis, that are stored as keywords in the header of the final FITS files. The data processing for the individual instruments involves several steps to calculate the coordinates and the energy of the events, set quality flags for the events with a quality status, calibrate the subsystem (MXS, SXS anticoincidence detector and CAMS) and derived quantities needed either for checking the instrument performance or to be used for further postprocessing, for example, to derive the non x-ray background (NXB). After the science data are calibrated, the processing applies screening criteria to calculate the GTIs and removes events flagged as not useful for the science analysis.

The pipeline processes the data by sequence and, at the end of each run, the files within a sequence are packaged and sent to the HEASARC (USA) and DARTS (Japan) archives, as is the case for the trend data. The data are organized in the archive by observation and by trend.¹⁹

The observation data are divided into directories, one for each sequence. Under each sequence directory, there are directories for each instrument and one for the auxiliary data. The CAMS data and the MXS data are included within the HXI and SXS directories, respectively. The different science data levels are stored in subdirectories within each of the instrument directories. Each sequence contains all the science, HK, and auxiliary files for all the instruments and subsystems.

The trend data are divided into instrument directories with one subdirectory dedicated to each trend type. The trend data consist of either HK information with no further processing or quantities calculated in the pipeline for monitoring special instrument characteristics.

During the 38 days of operation, Hitomi observed six astronomical objects and made several blank sky observations. These data have been divided into 42 sequences. These observations were taken during the commissioning phase and not all instruments were operational for all observations. The SXS started operating soon after launch and collected data from February 23, 2016, the SXI from March 6, 2016, the HXI1 and HXI2 from March 11 and March 14, 2016, respectively, and last, the SGD1 and SGD2 from March 20 and 22, respectively.

3.1.

Time Assignment

The time assignment framework is general to all science instruments and HK. Hitomi has a GPS receiver on board that dispatches the time to all subsystems as a time indicator (TI in units of $2^{-6}\mathrm{s}$). The instruments’ electronics assign the fine time to the science data. If the GPS is off, the time is assigned similarly to how it was done for the Suzaku satellite, where the TI is calibrated using the time assigned to the telemetry packets, the receiving time on ground, and the temperature–frequency relation of the on-board clock. Terada et al.²⁰ contain a detailed description of the on-board time assignment.

The main tasks in the Hitomi package that assign time are “ahmktim” and “ahtime.” Additional instrument-dependent tasks are used to precalculate the instrument fine time. ahmktim computes a table (TIM table) of TIME versus TI (the lower 32 bit of the TI value, L32TI) used by ahtime. The calculation uses the combination of the GPS flags that indicate when the GPS is on, off, in transition or in an illegal state. If the GPS is on, the TIME versus TI is obtained using the values of the time to when the packets are tagged (stored in a column named S_TIME) and using the L32TI column. If the GPS is off, TIME versus TI is calculated using the closest GPS time to derive the temperature of the on-board clock in the HK and the frequency–temperature relation obtained from the clock ground calibration. ahtime computes the time for all science data and HK with a three-step procedure. The first step derives the fine time resolution that is assigned internally by the instruments either using support instrument dependent look-up tables or keywords in the header of the FITS file. The second step derives, for a given L32TI, the corresponding time using the TIM table. The third step adds a delay, when appropriate, between the on-board computer and the instruments. For HK, data steps 1 and 3 are skipped because there is no need for fine time resolution. The times are stored in the column TIME, which contains seconds since an epoch set to 2014-01-01 00:00:00 UTC. The epoch is written in the header of all FITS files as an MJD value in the keywords MJDREFI and MJDREFF. The time system is terrestrial time (TT), and the MJD value of the epoch corresponds to MJD 56658.0007775925926 (TT).

3.2.

Coordinates Assignment

Hitomi has three instruments at the focal plane of the telescopes: the SXI, SXS (soft energy telescopes), and the HXI (high energy telescopes). The coordinates for these instruments are described by five different coordinate systems and their values are stored in the FITS columns named $\mathrm{PIXEL}/\mathrm{RAW}n$, $\mathrm{ACT}n$, $\mathrm{DET}n$, $\mathrm{FOC}n$, where $n$ is either $X$ or $Y$, and the columns $X/Y$ that represent the SKY system²¹ (see below). Since the SGDs are not imaging detectors, SGD files do not have any of the above columns for coordinates, however, the $XYZ$ position in the detector is provided for any reconstructed event. The transformation of coordinates for all instruments is defined in calibration files, one for each instrument (the telescope definition, teldef). The SXI, SXS, and HXI coordinates are calculated by a common mission independent task coordevt that uses the information from the CAMS to account for the movements of the EOB, where the HXIs are located when transforming from RAW to ACT, the attitude to calculate $X$ and $Y$, and the teldef for all other coordinate transformations. The characteristics for each instrument, such as pixel size, number of pixels, size of the FoV expressed in the different coordinate systems together with the definition on whether the coordinates are defined as a look-up/down system with respect to the detectors, are listed in Table 2.

Table 2

Instrument coordinate system characteristics.

The RAW coordinates comprise the basic coordinate system. For the SXS, they correspond to the telemetered pixel number where the event landed. This value is stored in the 1-d column PIXEL and ranges from 0 to 35, where one of the pixels (pixel 12) is the calibration source pixel located outside of the array that collects events from the sky. For the SXI, the RAW coordinates correspond to the telemetered event location into a segment of the CCD. Each CCD has two segments and the SXI has four CCDs for a total of eight segments. The $\mathrm{RAW}Y$ values range from 0 to 639 and the $\mathrm{RAW}X$ values from 0 to 319. For the HXI, the RAW coordinates are calculated after the event is reconstructed using the telemetered strip location, and they are affected by the movements of the EOB, where the HXIs are located.

The ACT coordinates are derived from the RAW coordinates. The SXS ACT coordinates represent a 2-d look-down linearized coordinate system starting from the values stored in the PIXEL column. The SXI ACT coordinates correspond to the pixel locations in one SXI CCD. They range from 1 to 640 in the $X$ and $Y$ dimensions. The SXI RAW-to-ACT conversion depends on the window mode and readout node. The HXI ACT coordinates are calculated from the RAW coordinates; the movement of the EOB with respect to the main satellite body is removed using the CAMS time-dependent misalignments.

The DET coordinates are derived from the ACT coordinates. The SXI DET coordinates combine all four CCDs into a single-system frame, accounting for any misalignments among them. The ACT-to-DET conversion changes the configuration from look-down (from sky looking down at the detector) to look-up (from the detector looking up at the sky) for all instruments. Figure 3 shows the DET coordinates for all instruments overlaid.

Fig. 3

SXI look-up view of the DET system with the HXIs and SXS overlaid. The FoV centers of the two HXI sensors and SXS coincide with the SXI pointing position. For each instrument, the origins of the coordinate system are indicated with color-coded circles. The large gray circles represent the placement of the SXI calibration sources.

The FOC system combines all of the HXI, SXI, and SXS individual DET coordinates into a common system with the same pixel size and image size, accounting for small misalignments among them. Since the SXI has the largest FoV and smallest detector pixel size, the FOC coordinates for all sensors adopt the pixel scale and range for the SXI.

The $X$ and $Y$ coordinates (or SKY) are derived from the FOC coordinates using the satellite attitude, and they differ from the $\mathrm{FOC}X$ and $\mathrm{FOC}Y$ according to the position angle or roll. The $X$ and $Y$ columns are in units of pixel. Each pixel value is associated with a position in the sky via the FITS WCS keywords. The SKY system defined by the WCS keywords is a tangent plane projection and oriented such that declination ($\delta$) increases in the $+Y$ direction and right ascension ($\alpha$) increases in the $-X$ direction. Figure 4 shows the in-flight sky images for the G21.5-0.9 source for each individual detector as well as the overlay of the HXIs and SXS onto the SXI frame.

Fig. 4

(a) Each panel shows the image of in an individual detector in sky coordinates. The bright red spot in the SXS image is the calibration pixel. (b) SXI image with contour overlay of the HXI1, HXI2, and SXS. In this image, the SXI has the area discriminator activated so that a small fraction of the FoV is exposed to the x-ray source.

3.3.

Energy Assignment

The HXI, SGD, SXI, and SXS spectra are derived using the FITS column pulse invariant (PI) that corresponds to the linearized pulse height (channel), where each channel has equal energy width (see Table 3). The PI is calculated with algorithms appropriate for each instrument due to the different physics of the x-ray interaction with the detector. Many intermediate quantities are necessary before computing the PI value, and all tasks for the SXI, SXS, HXI, and SGD record the intermediate values in additional columns of the event file.

Table 3

For each instrument, PI_CHAN gives the PI channel range and PI_BIN gives the size in energy of the channel in eV units.

The SXS telemetered “energy” information is recorded in the FITS columns PHA for all event grade types (high, med, and low grade both primary and secondary), where the grade information is stored in the column ITYPE. The PHA of the mid secondary grade events are first corrected by the task “sxssecor” and written in the column PHA2 that otherwise stores the telemetered PHA for primary events. The PI is calculated using “sxsgain” and “sxspha2pi” tasks. sxsgain calculates the SXS time-dependent energy scale correction using a calibration line and the appropriate temperature-dependent gain coefficients. The calibration lines originate either from the calibration pixel that is always operating or from the MXS calibration sources if they are turned on. Using the results of sxsgain and the SXS temperature-dependent gain file, sxspha2pi calculates the PI values and stores them in the PI column. The SXS anticoincidence data are also corrected for gain using a simple linear correction implemented in the task sxsanticopi. The first observation of the Perseus cluster was obtained when the SXS was not yet in thermal equilibrium that was reached on March 4. For the observations prior to that date, the gain is adjusted with an additional pixel-to-pixel correction, derived using an observation subsequent to that of Perseus conducted using an Fe55 source, on a filter wheel, that illuminated all pixels, and applied by the task “sxsperseus.”

The SXI telemetered energy is stored in the “PHAS” column that contains the $3\times 3\text{pixel}$ array of telemetered charge around the event center. “sxipi” corrects the PHAS values in the $3\times 3$ array for the video temperature part of the CCD electronics, for the charge trail, and for the charge transfer inefficiency. Using the corrected $3\times 3$ array, sxipi assigns the grade and the pulse height for each event and populates the GRADE and PHA columns. In the last step, the gain is applied to each event and the computed PI value is stored in the PI column.

For the HXI and SGD, the telemetered pulse height corresponds to the signals detected by the instrument. The x-ray photon, when interacting with the detector, may create more than one signal. The ensemble of all signals constitutes an occurrence. Therefore, for one interaction, there are many signals that may contribute to the final PI calculation. At the first step, hxisgdpha gain corrects the PHA of each individual signal, deriving an energy. The PI is calculated using hxievtid and sgdevtid tasks for the HXI and SGD, respectively. These tasks determine whether the signals, due to the interaction of the x-ray with the detector parts, are consistent with a valid event by comparing the detected signals with the allowed template patterns of signals. If the pattern within an occurrence is consistent with an event, hxievtid and sgdevtid assign the PI to the reconstructed event, considering the energies of the signals within the occurrence, and the RAW coordinates.

4.1.

SXI and SXS ARF and HXI Response Generator

There is a common framework to calculate the telescope EA.¹² aharfgen is a script that calls different tasks to calculate the ARF for the SXS and SXI (ahsxtarfgen), and RSP for the HXI (hxirspeffimg). Although the scheme for creating response functions differs among instruments, there are commonalities applicable to the three instruments. (a) The landing position in the focal plane of an event coming from a fixed position in sky varies with the attitude motions. This variation is accounted for by the outputs of the exposure calculation. ahexpmap calculates an histogram of time spent at different off-axis positions and, for the SXI and SXS, also calculates the fractional exposure of each detector pixel for each off-axis bin. (b) The telescope EA is calculated by the ray-tracing code¹³ (xrtraytrace), which is essentially a Monte Carlo simulation of photons injected into the telescope aperture. The run is made for each off-axis angle using a precalculated energy grid. The output is an event file containing the path of each photon through the telescope and the final impact coordinates on the focal plane. (c) The ray-tracing energy grid is coarser than the final energy grid of the ARF, because it utilizes an optimized precalculated grid with fine energy sampling in regions, where telescope or detector edges dominate and coarse sampling, where no features are expected due to the telescope or the detector. (d) There are four options to account for the spatial distribution of the x-ray source: point source, extended source flat within a circle, extended source described by a beta model, or a spatial distribution from an image obtained by a different mission. Events generated by the ray-tracing are filtered through the source region selection and propagated down to the detector. Any object encountered by the event down to the detector is accounted for at this stage. These are the filters for SXS and SXI and the baffle for the HXI. For the SXI and SXS, the final ARF calculation also includes the QE that is position dependent for the SXI but not for the SXS, the contamination and the exposure information. For the HXI, the code outputs an RSP file because the intrinsic detector characteristics are also time-and-position dependent. The LSF and QE are different for each of the five layers present in a single HXI unit. The HXI is located on the EOB that moves with respect to the main satellite body, where the telescopes are located. The EOB movements are tracked by the CAMS and recorded in the CAMS data. Therefore, the net response matrix is a weighted combination of the five responses. The weights depend on the relative time-and-position dependent x-ray source counts, which in turn depend on the movements of the EOB, with respect to the telescope and on the telescope EA.

4.2.

SXI and SXS RMF Generator

The RMF for the SXS and SXI is calculated by the sxsmkrmf and sxirmf tasks, respectively.¹² The SXS RMF depends on the grade distribution and the pixels selected to create the spectrum. sxsmkrmf first calculates an histogram of grades for the pixels of interest and then calculates the RMF for each pixel and grade, and averages the individual RMF according to the histogram. By default, the SXS RMF input and output grids result in an RMF with 32,768 channels of width 0.5 eV. The RMF for the SXI depends on the extraction region and sxirmf uses the WMAP, a coarse-binned image of the extraction region with the spatial counts distribution, to determine the weight distribution for the LSF from different positions on the detector. By default, an RMF is generated with an input energy grid in the range 0.200 to 23.974 keV, and with 5900 output channels of width 2 eV up to $\sim 12\mathrm{keV}$ and 500 channels up to $\sim 24\mathrm{keV}$.

4.3.

Flat Field

The counts images derived from the science events for the SXI, SXS, and HXI may be corrected for telescope and detector effects using the flat field image.

The SXI and SXS flat fields are calculated with ahexpmap by requesting an efficiency map as output. This is the same task that computes the exposure distribution at different off-axis angle necessary for the ARF calculation. The efficiency is calculated using several calibration quantities, the attitude information, and an input energy band. The calibration included in the efficiency calculation for both instruments are the SXT vignetting, the detector QE, and the time-dependent contamination column density. For the SXS only, the effect of the optical blocking filter, the gate valve, and any additional filter on the filter wheel is also considered. For the SXI, the optical blocking is already included in the QE. The contributions from these quantities, for either a single energy or for a range of energies, are multiplied together for each detector pixel considering the time-dependent variation of the attitude.

The HXI hxirspeffimg task calculates either the response or the flat field, the latter is achieved by requesting as output an efficiency map. Similarly to the SXI and SXS, the efficiency for the HXI is calculated using the vignetting from the hard x-ray telescope, the QE for each layer of the detector, the attitude information (output of ahexpmap), and an input energy range. However, for the HXI, the flat field also corrects for the movements of the EOB using the CAMS data. The efficiency map values for all instruments are coded as unitless numbers that range between 0 and 1.

4.4.

Non-x-ray Background

The background in all Hitomi detectors includes contributions from x-rays that are focused by the telescope optics, referred to as x-ray background (XRB) and nonfocused background components from particles and internal detector effects collectively referred to as the NXB. There are various techniques to account for the XRB and NXB backgrounds in the spectral fitting as well as different methods to correct for the XRB (see also Hitomi analysis guide at Ref. 10). For the HXI, SXI, and SXS, the NXB is instead obtained using products of the pipeline. For each sequence and instrument, the pipeline creates event files, where the events are selected to belong to the night earth data when x-rays are blocked from the detector view. However, to derive spectra with sufficient statistics to characterize the NXB, night earth event data from several observations are combined together over a time period longer than a typical observation and archived together with all necessary support files. To derive the appropriate NXB spectrum for a given observation, additional filtering may be needed. The tasks hxinxbgen, sxinxbgen, and sxsnxbgen allow for the filtering of the NXB for the region of the x-ray source of interest, to select NXB time interval that includes the same geomagnetic cut-off orbital rigidity as the observation, and to perform additional filtering or calculation specific to each detector.