Introduction¶
GNIRS-Pype, for now, uses Python 2.7. Please keep this in mind.
Running from the Command Line¶
GBIRS-Pype, for now, is started simply by running the gnirsPipeline script from the command line.
Running from Python¶
Python Programmers: even though GNIRS-Pype’s API has not been defined yet, you can still run GNIRS-Pype Pipeline, Steps, Routines, and Tasks from a Python interpeter by importing them.
For example, to run the combine 2D spectra from the Python interpreter:
This is currently under development.
You might be able to figure out what imports you need by checking the tops of the relevant scripts.
Examples of Running from the Command Line¶
Starting a Data Reduction from the Beginning
This is currently under development.
Starting a Data Reduction from a Specified Point
You can run each step, one at a time, from the command line like so. You need a gnirs.cfg file in your current working directory to run individual steps. Each step requires the general config section and its unique and/or additional config sections to be populated.
You can also run an individual step by turning them on or off in gnirsPipeline config and running the gnirsPipeline.
gnirsSort: To only sort and copy the GNIRS raw data or create symbolic links in between the qorking directories use a gnirs.cfg file like this:
# GNIRS-Pype configuration file.
[defaults]
gnirs-pipeVersion = 1.0.0
manualMode = False
overwrite = False
[ScienceDirectories]
[TelluricDirectories]
[CalibrationDirectories]
[gnirsPipeline]
getData = False
sort = True
checkData = False
calibrationReduction = False
scienceReduction = False
telluricReduction = False
combineSpectra2D = False
extractSpectra1D = False
telluricCorrection = False
fluxCalibration = False
combineOrdersXD = False
calculateSpectrumSNR = False
writeDataSheet = False
[interactive]
nsprepareInter = False
nsflatInter = False
nscombineInter = False
nssdistInter = False
nswavelengthInter = False
nsfitcoordsInter = False
nsextractInter = False
hLineInter = False
continuumInter = False
telluricInter = False
tempInter = False
[getData]
rawPath = rawData/
program =
[sort]
proprietaryCookie =
telluricTimeThreshold = 5400.0 ; units [seconds]
[checkData]
# No input parameters required
[calibrationReduction]
Start = 1
Stop = 5
cleanir_IRflats = False
cleanir_QHflats = False
cleanir_arcs = False
cleanir_pinholes = False
[scienceReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 20
checkPeaksMatch = True
[telluricReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 50
[combineSpectra2D]
# No input parameters required
[extractSpectra1D]
useApall = True
extractionApertureRadius = 4
checkPeaksMatch = True
toleranceOffset = 5
extractionFullSlit = False
extractionStepwise = False
extractionStepSize = 6 ; units [pixels]
[telluricCorrection]
Start = 1
Stop = 5
hLineMethod = vega
[fluxCalibration]
Start = 1
Stop = 6
telluricRA =
telluricDEC =
telluricSpectralType =
telluricMagnitude =
telluricTemperature =
[combineOrdersXD]
# no input parameters required
[calculateSpectrumSNR]
# This section is currently under development.
Start = 1
Stop = 1
[writeDataSheet]
# no input parameters required
[TelluricRegions] ; regions to use when tweaking the Telluric line removal
3 = *
4 = 14200:18000
5 = 11200:13400,14200:15200
6 = 11000:12000
7 = 9000:10000
8 = 8500:9500
[HLineRegions] ; regions to use when tweaking the H line removal
3 = 21400:21900
4 = 15500:18000
5 = 12750:12900
6 = 10700:11200
7 = 9000:10500
8 = 8500:9300
[ContinuumRegions] ; regions to use when fitting the Telluric continuum
3 = *
4 = *
5 = 11200:13300,14300:15200
6 = 9500:10900,11700:12600
7 = 8500:9200,9800:10700
8 = 7500:9500
And run the gnirsPipeline with:
This section is currently under development.
gnirsBaselineCalibration: To only reduce calibrations use a gnirs.cfg file like this:
# GNIRS-Pype configuration file.
[defaults]
gnirs-pipeVersion = 1.0.0
manualMode = False
overwrite = False
[ScienceDirectories]
[TelluricDirectories]
[CalibrationDirectories]
[gnirsPipeline]
getData = False
sort = False
checkData = False
calibrationReduction = True
scienceReduction = False
telluricReduction = False
combineSpectra2D = False
extractSpectra1D = False
telluricCorrection = False
fluxCalibration = False
combineOrdersXD = False
calculateSpectrumSNR = False
writeDataSheet = False
[interactive]
nsprepareInter = False
nsflatInter = False
nscombineInter = False
nssdistInter = False
nswavelengthInter = False
nsfitcoordsInter = False
nsextractInter = False
hLineInter = False
continuumInter = False
telluricInter = False
tempInter = False
[getData]
rawPath = rawData/
program =
[sort]
proprietaryCookie =
telluricTimeThreshold = 5400.0 ; units [seconds]
[checkData]
# No input parameters required
[calibrationReduction]
Start = 1
Stop = 5
cleanir_IRflats = False
cleanir_QHflats = False
cleanir_arcs = False
cleanir_pinholes = False
[scienceReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 20
checkPeaksMatch = True
[telluricReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 50
[combineSpectra2D]
# No input parameters required
[extractSpectra1D]
useApall = True
extractionApertureRadius = 4
checkPeaksMatch = True
toleranceOffset = 5
extractionFullSlit = False
extractionStepwise = False
extractionStepSize = 6 ; units [pixels]
[telluricCorrection]
Start = 1
Stop = 5
hLineMethod = vega
[fluxCalibration]
Start = 1
Stop = 6
telluricRA =
telluricDEC =
telluricSpectralType =
telluricMagnitude =
telluricTemperature =
[combineOrdersXD]
# no input parameters required
[calculateSpectrumSNR]
# This section is currently under development.
Start = 1
Stop = 1
[writeDataSheet]
# no input parameters required
[TelluricRegions] ; regions to use when tweaking the Telluric line removal
3 = *
4 = 14200:18000
5 = 11200:13400,14200:15200
6 = 11000:12000
7 = 9000:10000
8 = 8500:9500
[HLineRegions] ; regions to use when tweaking the H line removal
3 = 21400:21900
4 = 15500:18000
5 = 12750:12900
6 = 10700:11200
7 = 9000:10500
8 = 8500:9300
[ContinuumRegions] ; regions to use when fitting the Telluric continuum
3 = *
4 = *
5 = 11200:13300,14300:15200
6 = 9500:10900,11700:12600
7 = 8500:9200,9800:10700
8 = 7500:9500
And run the gnirsPipeline with:
This section is currently under development.
gnirsReduce Science: To only reduce science data use a config.cfg file like this: Make sure to populate ScienceDirectories, TelluricDirectories and CalibrationDirectories before running!
# GNIRS-Pype configuration file.
[defaults]
gnirs-pipeVersion = 1.0.0
manualMode = False
overwrite = False
[ScienceDirectories]
[TelluricDirectories]
[CalibrationDirectories]
[gnirsPipeline]
getData = False
sort = False
checkData = False
calibrationReduction = False
scienceReduction = True
telluricReduction = False
combineSpectra2D = False
extractSpectra1D = False
telluricCorrection = False
fluxCalibration = False
combineOrdersXD = False
calculateSpectrumSNR = False
writeDataSheet = False
[interactive]
nsprepareInter = False
nsflatInter = False
nscombineInter = False
nssdistInter = False
nswavelengthInter = False
nsfitcoordsInter = False
nsextractInter = False
hLineInter = False
continuumInter = False
telluricInter = False
tempInter = False
[getData]
rawPath = rawData/
program =
[sort]
proprietaryCookie =
telluricTimeThreshold = 5400.0 ; units [seconds]
[checkData]
# No input parameters required
[calibrationReduction]
Start = 1
Stop = 5
cleanir_IRflats = False
cleanir_QHflats = False
cleanir_arcs = False
cleanir_pinholes = False
[scienceReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 20
checkPeaksMatch = True
[telluricReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 50
[combineSpectra2D]
# No input parameters required
[extractSpectra1D]
useApall = True
extractionApertureRadius = 4
checkPeaksMatch = True
toleranceOffset = 5
extractionFullSlit = False
extractionStepwise = False
extractionStepSize = 6 ; units [pixels]
[telluricCorrection]
Start = 1
Stop = 5
hLineMethod = vega
[fluxCalibration]
Start = 1
Stop = 6
telluricRA =
telluricDEC =
telluricSpectralType =
telluricMagnitude =
telluricTemperature =
[combineOrdersXD]
# no input parameters required
[calculateSpectrumSNR]
# This section is currently under development.
Start = 1
Stop = 1
[writeDataSheet]
# no input parameters required
[TelluricRegions] ; regions to use when tweaking the Telluric line removal
3 = *
4 = 14200:18000
5 = 11200:13400,14200:15200
6 = 11000:12000
7 = 9000:10000
8 = 8500:9500
[HLineRegions] ; regions to use when tweaking the H line removal
3 = 21400:21900
4 = 15500:18000
5 = 12750:12900
6 = 10700:11200
7 = 9000:10500
8 = 8500:9300
[ContinuumRegions] ; regions to use when fitting the Telluric continuum
3 = *
4 = *
5 = 11200:13300,14300:15200
6 = 9500:10900,11700:12600
7 = 8500:9200,9800:10700
8 = 7500:9500
And run the gnirsPipeline with:
This is currently under development.
gnirsFluxCalibrate: To only do a flux calibration use a gnirs.cfg file like this: Make sure to populate scienceDirectoryList, telluricDirectoryList and calibrationDirectoryList before running!
# GNIRS-Pype configuration file.
[defaults]
gnirs-pipeVersion = 1.0.0
manualMode = False
overwrite = False
[ScienceDirectories]
[TelluricDirectories]
[CalibrationDirectories]
[gnirsPipeline]
getData = False
sort = False
checkData = False
calibrationReduction = False
scienceReduction = False
telluricReduction = False
combineSpectra2D = False
extractSpectra1D = False
telluricCorrection = False
fluxCalibration = True
combineOrdersXD = False
calculateSpectrumSNR = False
writeDataSheet = False
[interactive]
nsprepareInter = False
nsflatInter = False
nscombineInter = False
nssdistInter = False
nswavelengthInter = False
nsfitcoordsInter = False
nsextractInter = False
hLineInter = False
continuumInter = False
telluricInter = False
tempInter = False
[getData]
rawPath = rawData/
program =
[sort]
proprietaryCookie =
telluricTimeThreshold = 5400.0 ; units [seconds]
[checkData]
# No input parameters required
[calibrationReduction]
Start = 1
Stop = 5
cleanir_IRflats = False
cleanir_QHflats = False
cleanir_arcs = False
cleanir_pinholes = False
[scienceReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 20
checkPeaksMatch = True
[telluricReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 50
[combineSpectra2D]
# No input parameters required
[extractSpectra1D]
useApall = True
extractionApertureRadius = 4
checkPeaksMatch = True
toleranceOffset = 5
extractionFullSlit = False
extractionStepwise = False
extractionStepSize = 6 ; units [pixels]
[telluricCorrection]
Start = 1
Stop = 5
hLineMethod = vega
[fluxCalibration]
Start = 1
Stop = 6
telluricRA =
telluricDEC =
telluricSpectralType =
telluricMagnitude =
telluricTemperature =
[combineOrdersXD]
# no input parameters required
[calculateSpectrumSNR]
# This section is currently under development.
Start = 1
Stop = 1
[writeDataSheet]
# no input parameters required
[TelluricRegions] ; regions to use when tweaking the Telluric line removal
3 = *
4 = 14200:18000
5 = 11200:13400,14200:15200
6 = 11000:12000
7 = 9000:10000
8 = 8500:9500
[HLineRegions] ; regions to use when tweaking the H line removal
3 = 21400:21900
4 = 15500:18000
5 = 12750:12900
6 = 10700:11200
7 = 9000:10500
8 = 8500:9300
[ContinuumRegions] ; regions to use when fitting the Telluric continuum
3 = *
4 = *
5 = 11200:13300,14300:15200
6 = 9500:10900,11700:12600
7 = 8500:9200,9800:10700
8 = 7500:9500
And run the gnirsPipeline with:
This is currently under development.
gnirsCombineOrdersXD Orders Combining: To only combine sifferent spectral orders use a gnirs.cfg file like this: Make sure to populate scienceDirectoryList, telluricDirectoryList and calibrationDirectoryList before running!
# GNIRS-Pype configuration file.
[defaults]
gnirs-pipeVersion = 1.0.0
manualMode = False
overwrite = False
[ScienceDirectories]
[TelluricDirectories]
[CalibrationDirectories]
[gnirsPipeline]
getData = False
sort = False
checkData = False
calibrationReduction = False
scienceReduction = False
telluricReduction = False
combineSpectra2D = False
extractSpectra1D = False
telluricCorrection = False
fluxCalibration = False
combineOrdersXD = True
calculateSpectrumSNR = False
writeDataSheet = False
[interactive]
nsprepareInter = False
nsflatInter = False
nscombineInter = False
nssdistInter = False
nswavelengthInter = False
nsfitcoordsInter = False
nsextractInter = False
hLineInter = False
continuumInter = False
telluricInter = False
tempInter = False
[getData]
rawPath = rawData/
program =
[sort]
proprietaryCookie =
telluricTimeThreshold = 5400.0 ; units [seconds]
[checkData]
# No input parameters required
[calibrationReduction]
Start = 1
Stop = 5
cleanir_IRflats = False
cleanir_QHflats = False
cleanir_arcs = False
cleanir_pinholes = False
[scienceReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 20
checkPeaksMatch = True
[telluricReduction]
Start = 1
Stop = 5
cleanir = False
radiationCorrectionMethod = fixpix
radiationThreshold = 50
[combineSpectra2D]
# No input parameters required
[extractSpectra1D]
useApall = True
extractionApertureRadius = 4
checkPeaksMatch = True
toleranceOffset = 5
extractionFullSlit = False
extractionStepwise = False
extractionStepSize = 6 ; units [pixels]
[telluricCorrection]
Start = 1
Stop = 5
hLineMethod = vega
[fluxCalibration]
Start = 1
Stop = 6
telluricRA =
telluricDEC =
telluricSpectralType =
telluricMagnitude =
telluricTemperature =
[combineOrdersXD]
# no input parameters required
[calculateSpectrumSNR]
# This section is currently under development.
Start = 1
Stop = 1
[writeDataSheet]
# no input parameters required
[TelluricRegions] ; regions to use when tweaking the Telluric line removal
3 = *
4 = 14200:18000
5 = 11200:13400,14200:15200
6 = 11000:12000
7 = 9000:10000
8 = 8500:9500
[HLineRegions] ; regions to use when tweaking the H line removal
3 = 21400:21900
4 = 15500:18000
5 = 12750:12900
6 = 10700:11200
7 = 9000:10500
8 = 8500:9300
[ContinuumRegions] ; regions to use when fitting the Telluric continuum
3 = *
4 = *
5 = 11200:13300,14300:15200
6 = 9500:10900,11700:12600
7 = 8500:9200,9800:10700
8 = 7500:9500
And run the gnirsPipeline with:
This is currently under development.
Preparing the .cfg Input File¶
GNIRS-Pype reads data reduction parameters with the Python 2.7 built-in ConfigParser. See https://docs.python.org/2.7/library/configparser.html for full documentation on the parser.
Interactive Input Preparation¶
This is currently under development.
An Example Input File¶
GNIRS-Pype includes a default configuration file. As of v1.0.1, it looks like this:
TODO(Viraja): Updated example
Data Reduction Examples¶
Observations of NGC 4736; GN-2011A-Q-126¶
This is currently under development.
Example 2¶
This is currently under development.
Example 3¶
This is currently under development.
Tutorials¶
H Line Removal¶
This is currently under development.
Custom Telluric Corrections¶
This is currently under development.
Combining XD Orders¶
This is currently under development.
Known Issues¶
gnirsPipeline.py¶
gnirsGetData.py¶
sort_data.py¶
make_lists.py¶
link_cals.py¶
gnirsCheckData.py¶
gnirsBaselineCalibration.py¶
gnirsReduce.py¶
gnirsCombineSpectra2D.py¶
gnirsExtractSpectra1D.py¶
gnirsTelluric.py¶
gnirsFluxCalibrate.py¶
gnirsCombineOrdersXD.py¶
gnirsCalculateSpectramSNR.py¶
gnirsWriteDataSheet.py¶
Maintaining GNIRS-Pype¶
Documentation¶
Right now there exists four forms of documentation.
Paper
README.rst
.rst Files in the docs/ directory
This file, others like it in the docs/ directory and the README are written in reStructuredText. This markup language integrates well with Python’s automatic documentation builder (we used Sphinx) and Github as well as being human readable. You can read more about reStructuredText here.
Comments and DocStrings in Source Code
Tests¶
This is a TODO. Currently we do not have automated tests.
Pipeline Structure¶
This is currently under development.
GNIRS-Pype is built at the lowest level from Python and IRAF subroutines. It is built so that it is relatively easy to change the implementation of the underlying tasks.
Updates¶
To update GNIRS-Pype, do …
This is currently under development.
Version Numbers
GNIRS-Pype uses semantic versioning(see http://semver.org/). This means version numbers come in
MAJOR.MINOR.PATCH
In brief, when releasing a version of Nifty that is not backward-compatible with old test recipes, or changes break the public API, it is time to increment the MAJOR version number.
Code Conventions¶
For naming variables and functions, a mix of camelCase and lower_case_with_underscores was used.
Code style was influenced by the Google Python Style Guide.
GNIRS-Pype uses the Google docstring style. Examples of docstrings can be found here.
Future Work¶
Throughout the code, vkhatu has placed many TODO notes. These are things that should be reviewed at some point.
Future work:
- Support other GNIRS gratings (10l/mm and 111l/mm)
- Implement more telluric and flux calibration methods.
- Implement instrument signature removal routine.
- Implement differential atmospheric refraction correction routine.
- Implement full automatic Gemini (on server) reduction routine.
- Python 3 compatability (if possible)
- Incorporate more command line options
- Compiling as a self-contained executable
- Full AstroConda integration
Changelog¶
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
Unreleased¶
All in-development changes will be tracked here.
- Adding unit tests for each step and integration test for pipeline.
1.0.0 - 2019-08-19¶
This is currently under development.
API¶
Note: I didn’t have time to implement this using Sphinx automodule. GNIRS-Pype has fairly good docstrings and you can use individual steps, routines and tasks by importing them. This is a TODO.