spec2nexus.spec

Library of classes to read the contents of a SPEC data file.

How to use spec2nexus.spec

spec2nexus.spec provides Python support to read the scans in a SPEC data file. (It does not provide a command-line interface.) Here is a quick example how to use spec:

1
2
3
4
5
6
7
8
9
from spec2nexus.spec import SpecDataFile

specfile = SpecDataFile('data/33id_spec.dat')
print 'SPEC file name:', specfile.specFile
print 'SPEC file time:', specfile.headers[0].date
print 'number of scans:', len(specfile.scans)

for scanNum, scan in specfile.scans.items():
    print scanNum, scan.scanCmd

For one example data file provided with spec2nexus.spec, the output starts with:

How to read one scan

Here is an example how to read one scan:

1
2
3
4
5
6
from spec2nexus.spec import SpecDataFile

specfile = SpecDataFile('data/33id_spec.dat')
specscan = specfile.getScan(5)
print specscan.scanNum
print specscan.scanCmd

which has this output:

5
ascan  del 84.3269 84.9269  30 1

The data columns are provided in a dictionary. Using the example above, the dictionary is specscan.data where the keys are the column labels (from the #L line) and the values are from each row. It is possible to make a default plot of the last column vs. the first column. Here’s how to find that data:

1
2
3
4
x_label = specscan.L[0]          # first column from #L line
y_label = specscan.L[-1]         # last column from #L line
x_data = specscan.data[x_label]  # data for first column
y_data = specscan.data[y_label]  # data for last column

Get a list of the scans

The complete list of scan numbers from the data file is obtained (sorting is necessary since the list of dictionary keys is returned in a scrambled order):

all_scans = sorted(specfile.scans.keys())

SPEC data files

The SPEC data file format is described in the SPEC manual. [1] This manual is taken as a suggested starting point for most users. Data files with deviations from this standard are produced at some facilities.

[1]SPEC manual: http://www.certif.com/spec_manual/user_1_4_1.html

Assumptions about data file structure

These assumptions are used to parse SPEC data files:

  1. SPEC data files are text files organized by lines. The lines can be categorized as: control lines, data lines, and blank lines.

    line type description
    control contain a # character in the first column followed by a command word [2]
    data generally contain a row of numbers (the scan data)
    special data containing MCA data [3]

  2. Lines in a SPEC data file start with a file name control line, then series of blocks. Each block may be either a file header block or a scan block. (Most SPEC files have only one header block. A new header block is created if the list of positioners is changed in SPEC without creating a new file. SPEC users are encouraged to always start a new data file after changing the list of positioners.) A block consists of a series of control, data, and blank lines.

    SPEC data files are composed of a sequence of a single file header block and zero or more scan blocks. [4]

  3. A SPEC data file always begins with this control lines: #F, such as:

    #F samplecheck_7_17_03

  4. A file header block begins with these control lines in order: #E #D #C, such as:

    #E 1058427452
#D Thu Jul 17 02:37:32 2003
#C psic  User = epix

  5. A scan block begins with these command lines in order: #S #D, such as:

    #S 78  ascan  del 84.6484 84.8484  20 1
#D Thu Jul 17 08:03:54 2003
[2]See Example of Control Lines
[3]See Example of MCA data lines
[4]It is very unusual to have more than one file header block in a SPEC data file.

Control lines (keys) defined by SPEC

Here is a list [5] of keys (command words) from the comments in the file.mac (SPEC v6) macro source file:

command word description
#C comment line
#D date current date and time in UNIX format
#E num the UNIX epoch (seconds from 00:00 GMT 1/1/70)
#F name name by which file was created
#G1 … geometry parameters from G[] array (geo mode, sector, etc)
#G2 … geometry parameters from U[] array (lattice constants, orientation reflections)
#G3 … geometry parameters from UB[] array (orientation matrix)
#G4 … geometry parameters from Q[] array (lambda, frozen angles, cut points, etc)
#I num a normalizing factor to apply to the data
#j% … mnemonics of counter (% = 0,1,2,… with eight counters per row)
#J% … names of counters (each separated by two spaces)
#L s1 … labels for the data columns
#M num data was counted to this many monitor counts
#N num [num2] number of columns of data [ num2 sets per row ]
#o% … mnemonics of motors (% = 0,1,2,… with eight motors per row)
#O% … names of motors (each separated by two spaces)
#P% … positions of motors corresponding to above #O/#o
#Q a reciprocal space position (H K L)
#R user-defined results from a scan
#S num scan number
#T num data was counted for this many seconds
#U user defined
#X a temperature
#@MCA fmt this scan contains MCA data (array_dump() format, as in "%16C")
#@CALIB a b c coefficients for x[i] = a + b * i + c * i * i for MCA data
#@CHANN n f l r MCA channel information (number_saved, first_saved, last_saved, reduction coef)
#@CTIME p l r MCA count times (preset_time, elapsed_live_time, elapsed_real_time)
#@ROI n f l MCA ROI channel information (ROI_name, first_chan, last_chan)
[5]Compare with Supplied spec plugin modules

Example of Control Lines

The command word of a control line may have a number at the end, indicating it is part of a sequence, such as these control lines (see Control lines (keys) defined by SPEC for how to interpret):

Example of MCA data lines

Lines with MCA array data begin with the @A command word. (If such a data line ends with a continuation character \, the next line is read as part of this line.)

This is an example of a 91-channel MCA data array with trivial (zero) values:

1
2
3
4
5
6
@A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0

Several MCA spectra may be written to a scan. In this case, a number follows @A indicating which spectrum, such as in this example with four spectra:

1
2
3
4
     @A1 0 0 0 0 0 0 35 0 0 35
     @A2 0 0 0 0 0 0 0 35 0 35
     @A3 0 0 35 35 0 0 0 0 0 0
     @A4 0 0 0 0 0 35 35 0 35 0

Supported header keys (command words)

The SPEC data file keys recognized by spec are listed in Supplied spec plugin modules.

source code summary

classes

SpecDataFile contents of a SPEC data file
SpecDataFileHeader contents of a spec data file header (#E) section
SpecDataFileScan contents of a spec data file scan (#S) section

methods

strip_first_word return everything after the first space on the line from the spec data file
is_spec_file test if a given file name is a SPEC data file

exceptions

SpecDataFileNotFound data file was not found
SpecDataFileCouldNotOpen data file could not be opened
SpecDataFileNotFound data file was not found
DuplicateSpecScanNumber multiple use of scan number in a single SPEC data file
UnknownSpecFilePart unknown part in a single SPEC data file

dependencies

os OS routines for NT or Posix depending on what system we’re on.
re Support for regular expressions (RE).
sys This module provides access to some objects used or maintained by the interpreter and to functions that interact strongly with the interpreter.

internal structure of spec2nexus.spec.SpecDataFileScan

The internal variables of a Python class are called attributes. It may be convenient, for some, to think of them as variables.

scan attributes

parent:obj - instance of spec2nexus.spec.SpecDataFile
scanNum:int - SPEC scan number
scanCmd:str - SPEC command line
raw:str - text of scan, as reported in SPEC data file

scan attributes (variables) set after call to plugins

These attributes are only set after the scan’s interpret() method is called. This method is called automatically when trying to read any of the following scan attributes:

comments:[str] - list of all comments reported in this scan
data:{label,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.data_lines_postprocessing()
data_lines:[str] - raw data (and possibly MCA) lines with comment lines removed
date:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Date
G:{key,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Geometry
I:float - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NormalizingFactor
header:obj - instance of spec2nexus.spec.SpecDataFileHeader
L:[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Labels
M:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Monitor
positioner:{key,number} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners.postprocess
N:[int] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NumColumns
P:[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners
Q:[number] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_HKL
S:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Scan
T:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_CountTime
V:{key,number|str} - written by spec2nexus.plugins.unicat_spec2nexus.UNICAT_MetadataValues
column_first:str - label of first (ordinate) data column
column_last:str - label of last (abscissa) data column

internal use only - do not modify

These scan attributes are for internal use only and are not part of the public interface. Do not modify them or write code that depends on them.

postprocessors:{key,obj} - dictionary of postprocessing methods
h5writers:{key,obj} - dictionary of methods that write HDF5 structure
__lazy_interpret__:
 bool - Is lazy (on-demand) call to interpret() needed?
__interpreted__:
 bool - Has interpret() been called?

source code documentation

Provides a set of classes to read the contents of a SPEC data file.

author:Pete Jemian
email:jemian@anl.gov

SpecDataFile() is the only class users will need to call. All other spec classes are called from this class. The read() method is called automatically.

The user should create a class instance for each spec data file, specifying the file reference (by path reference as needed) and the internal routines will take care of all that is necessary to read and interpret the information.

is_spec_file(filename) test if a given file name is a SPEC data file
is_spec_file_with_header(filename) test if a given file name is a SPEC data file
SpecDataFile(filename) contents of a SPEC data file
SpecDataFileHeader(buf[, parent]) contents of a spec data file header (#E) section
SpecDataFileScan(header, buf[, parent]) contents of a spec data file scan (#S) section

Note that the SPEC geometry control lines (#G0 #G1 …) have meanings that are unique to specific diffractometer geometries including different numbers of values. Consult the geometry macro file for specifics.

Examples

Get the first and last scan numbers from the file:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> print(spec_data.fileName)
path/to/my/spec_data.dat
>>> print('first scan: ', spec_data.getFirstScanNumber())
1
>>> print('last scan: ', spec_data.getLastScanNumber())
22

Get plottable data from scan number 10:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> scan10 = spec_data.getScan(10)
>>> x_label = scan10.L[0]
>>> y_label = scan10.L[-1]
>>> x_data = scan10.data[x_label]
>>> y_data = scan10.data[y_label]

Try to read a file that does not exist:

>>> spec_data = spec.SpecDataFile('missing_file')
Traceback (most recent call last):
  ...
spec.SpecDataFileNotFound: file does not exist: missing_file

Classes and Methods

exception spec2nexus.spec.DuplicateSpecScanNumber[source]

multiple use of scan number in a single SPEC data file

exception spec2nexus.spec.NotASpecDataFile[source]

content of file is not SPEC data (first line must start with #F)

class spec2nexus.spec.SpecDataFile(filename)[source]

contents of a SPEC data file

dissect_file() divide (SPEC data file text) buffer into sections
getFirstScanNumber() return the first scan
getLastScanNumber() return the last scan
getMaxScanNumber() return the highest numbered scan
getMinScanNumber() return the lowest numbered scan
getScan([scan_number]) return the scan number indicated, None if not found
getScanCommands([scan_list]) return all the scan commands as a list, with scan number
getScanNumbers() return a list of all scan numbers sorted by scan number
getScanNumbersChronological() return a list of all scan numbers sorted by date
read() Reads and parses a spec data file
refresh() update (refresh) the content if the file is updated
update_available Has the file been updated since the last time it was read?
dissect_file()[source]

divide (SPEC data file text) buffer into sections

internal: A block starts with either #F | #E | #S

RETURNS

[block]
list of blocks where each block is one or more lines of text with one of the above control lines at its start
getFirstScanNumber()[source]

return the first scan

getLastScanNumber()[source]

return the last scan

getMaxScanNumber()[source]

return the highest numbered scan

getMinScanNumber()[source]

return the lowest numbered scan

getScan(scan_number=0)[source]

return the scan number indicated, None if not found

getScanCommands(scan_list=None)[source]

return all the scan commands as a list, with scan number

getScanNumbers()[source]

return a list of all scan numbers sorted by scan number

getScanNumbersChronological()[source]

return a list of all scan numbers sorted by date

read()[source]

Reads and parses a spec data file

refresh()[source]

update (refresh) the content if the file is updated

returns previous last_scan or None if file not updated

update_available

Has the file been updated since the last time it was read?

Reference file modification time is stored after file is read in read() method.

EXAMPLE USAGE

Open the SPEC data file (example):

sdf = spec.SpecDataFile(filename)

then, monitor (continuing example):

if sdf.update_available:
myLastScan = sdf.last_scan sdf.read() plot_scan_and_newer(myLastScan) # new method myLastScan = sdf.last_scan
exception spec2nexus.spec.SpecDataFileCouldNotOpen[source]

data file could not be opened

class spec2nexus.spec.SpecDataFileHeader(buf, parent=None)[source]

contents of a spec data file header (#E) section

interpret() interpret the supplied buffer with the spec data file header
addPostProcessor(label, func) add a function to be processed after interpreting all lines from a header
addH5writer(label, func) add a function to be processed when writing the scan header
getLatestScan()
addH5writer(label, func)[source]

add a function to be processed when writing the scan header

Parameters:
  • label (str) – unique label by which this writer will be known
  • func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)[source]

add a function to be processed after interpreting all lines from a header

Parameters:
  • label (str) – unique label by which this postprocessor will be known
  • func (obj) – function reference of postprocessor

The postprocessors will be called at the end of header interpretation.

interpret()[source]

interpret the supplied buffer with the spec data file header

exception spec2nexus.spec.SpecDataFileNotFound[source]

data file was not found

class spec2nexus.spec.SpecDataFileScan(header, buf, parent=None)[source]

contents of a spec data file scan (#S) section

get_macro_name() name of the SPEC macro used for this scan
interpret() interpret the supplied buffer with the spec scan data
add_interpreter_comment(comment) allow the interpreter to communicate information to the caller
get_interpreter_comments() return the list of comments
addPostProcessor(label, func) add a function to be processed after interpreting all lines from a scan
addH5writer(label, func) add a function to be processed when writing the scan data
addH5writer(label, func)[source]

add a function to be processed when writing the scan data

Parameters:
  • label (str) – unique label by which this writer will be known
  • func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)[source]

add a function to be processed after interpreting all lines from a scan

Parameters:
  • label (str) – unique label by which this postprocessor will be known
  • func (obj) – function reference of postprocessor

The postprocessors will be called at the end of scan data interpretation.

add_interpreter_comment(comment)[source]

allow the interpreter to communicate information to the caller

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_interpreter_comments()[source]

return the list of comments

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_macro_name()[source]

name of the SPEC macro used for this scan

interpret()[source]

interpret the supplied buffer with the spec scan data

exception spec2nexus.spec.UnknownSpecFilePart[source]

unknown part in a single SPEC data file

spec2nexus.spec.is_spec_file(filename)[source]

test if a given file name is a SPEC data file

Parameters:filename (str) – path/to/possible/spec/data.file

filename is a SPEC file if it contains at least one #S control line

spec2nexus.spec.is_spec_file_with_header(filename)[source]

test if a given file name is a SPEC data file

Parameters:filename (str) – path/to/possible/spec/data.file

filename is a SPEC file only if the file starts [6] with these control lines in order:

  • #F - original filename
  • #E - the UNIX epoch (seconds from 00:00 GMT 1/1/70)
  • #D - current date and time in UNIX format
  • #C - comment line (the first one provides the filename again and the user name)

such as:

#F LNO_LAO
#E 1276730676
#D Wed Jun 16 18:24:36 2010
#C LNO_LAO  User = epix33bm
[6]SPEC manual, Standard Data File Format, http://www.certif.com/spec_manual/user_1_4_1.html