Processing guidelines
Cambridge CS
Using the data viewer (Graph plot)
Quick Start
Cambridge CS can be called with the following command:
cambridgecs [OPTION] [Script file]
The full options available can be seen in the help text:
cambridgecs -h
To launch the gui:
cambridgecs -g
To load an existing script file into the gui:
cambridgecs -g script.scr
To run an existing script in automatic mode (command line only):
cambridgecs -a script.scr
Processing data
In non-uniformly sampled experiments, the first (direct/acquisition) dimension is always fully sampled. Consequently, this dimension should be processed using the Fourier Tranform, with traditional tools. Cambridge CS is able to process the direct dimension within the software package, but can also interface with NMRPipe (see below for details).
Subsequently the indirect dimensions are processed. In essence, the processing software replaces the missing points with zeros and then the CS reconstruction attempts to 'fill-in' these points with the correct values, thus generating the ideal spectrum. The result can then be used for subsequent processing in the Cambridge CS package (using Azara) or for export to NMRPipe.
In the following, full processing (i.e. direct and indirect dimensions with post-processing in Cambridge CS) is described.
Processing Overview
Cambridge CS allows a script-based or GUI-based processing approach. The GUI generates a processing script and starts the reconstruction, subsequently opening a data viewer to display the result. When using the GUI, at any time the 'Open Raw Script' button shows the current state of the script file enabling the command syntax to be viewed. Commands may also be written in the scripting window and the 'Sync text' button updates the GUI.
There are four sections to the GUI/data-processing scripts.
Data parameters - information about the data e.g. file size, location etc.
Pre-processing - direct dimension processing (optional)
Reconstruction - indirect dimension reconstruction (required)
Post-processing - post-processing commands available in the Azara package e.g. baseline correction (optional)
Test data is available in the test_data folder.
2D processing example
# General properties: directory ~/data/CS/CScode/test_data/2D outputFile 102_cs_nus threads 1 rawDirectory 102 nusList vclist direct-dim: npts 1024 end_script indirect-dims: npts 80 ndim 1 end_script # Pre-processing parameters: pre_processing_script: complex brukerGroupPhase 71.625 gaussian_sw 20.0 1.0 10000.0 zerofill 1 fft phase 103.0 40.0 1.0 reduce range 220 481 end_script # Reconstruction parameters: reconstruction_script: reconstruction_flag IHT stop_flag l2 dim 2: zerofill 1 phase 180.0 -360.0 mask_ppmm sinebell 90.0 end_script end_script # Azara post processing instructions: post_processing_script: script_com 2: base_poly 8.0 0.0 end_script script_com 1: base_poly2 8.0 1.0 1.0 261.0 end_script end_script
Important points:
The first section contains general data properties.
# General properties: directory ~/data/CS/CScode/test_data/2D outputFile 102_cs_nus threads 1 rawDirectory 102 nusList vclist direct-dim: npts 1024 end_script indirect-dims: npts 80 ndim 1 end_script
All data is treated as a pseudo-2D and the total number of real and imaginary points in the direct and all indirect dimensions is given. The number of indirect dimensions, ndim, must also be specified. Other important fields in this section are:
directory - directory containing the data; for Bruker data this is assumed to contain the data folder
rawDirectory - for Bruker data this is the folder containing the unprocessed data i.e. ser file
threads - the number of cpu threads to use. By default this is set to 1. '0' uses the maximum number of available threads.
outputFile - prefix for all output data
nusList - sampling list specifying the recorded points
The pre-processing script contains a series of commands to process the direct dimension.
# Pre-processing parameters: pre_processing_script: complex brukerGroupPhase 71.625 gaussian_sw 20.0 1.0 10000.0 zerofill 1 fft phase 103.0 40.0 1.0 reduce range 220 481 end_script
Commands are performed in a linear manner i.e. order matters. These commands follow the Azara processing style with the addition of the brukerGroupPhase command which takes the GRPDLY parameter found in Bruker acqus files. It is important to include the reduce command (after the Fourier transform) in order to discard the imaginary part which is not required for further processing.
The reconstruction script carries out reconstruction of the indirect dimensions.
# Reconstruction parameters: reconstruction_script: reconstruction_flag IHT stop_flag l2 dim 2: zerofill 1 phase 180.0 -360.0 mask_ppmm sinebell 90.0 end_script end_script
Since the indirect dimensions are processed together, the order of commands is not important here. The type of reconstruction is specified first, either FT or various CS algorithms available with the software (currently IHT, IST, IRL1). In the example given here, iterative hard thresholding (IHT) is used. A choice of stopping criteria (stop_flag) is available. The default, l2, works well. Commands for each of the indirect dimensions again follow the Azara style. Various window functions are available, zerofilling and data manipulation options (mask_ppmm, conjugate etc.) to handle different forms of data recording. It is important to set the correct phasing for each of the indirect dimensions. Additionally if (0,0) phase is used, scaleT0 should be set to 0.5 to correct the first point scaling.
The best way to determine the correct indirect dimension parameters is to process with the FT first. This is set by changing Reconstruction type (reconstruction_flag) to FT. The data can then be processed quickly and the phasing etc. checked before subsequently reconstructing the data using CS.
Finally the post-processing section specifies any data processing required after reconstruction.
# Azara post processing instructions: post_processing_script: script_com 2: base_poly 8.0 0.0 end_script script_com 1: base_poly2 8.0 1.0 1.0 261.0 end_script end_script
In the example given above, the output from the CS reconstruction is the frequency domain data and only baseline correction is applied, again in the Azara syntax. It is also possible to use the time-domain data (useTimeData); since the missing points have been reconstructed, full FT processing (using Azara commands) can be applied at this stage, in a similar manner to the direct dimension processing above. Available options can be seen in the GUI and further details in the manual or on the Azara website.
Higher dimensions
To extend to more indirect dimensions, the ndim option is changed in the first section (number of indirect dimensions) and additional processing parameters need to be specified in the reconstruction script section for each additional dimension. If setting up the script using the GUI, increasing ndim will automatically increase the number of dimensions available in the reconstruction script.
An example of a 3D processing script is shown below:
# General properties: directory ~/data/CS/CScode/test_code/3D/HNCO outputFile 550_cs threads 1 rawDirectory 550 nusList vclist direct-dim: npts 1024 end_script indirect-dims: npts 1400 ndim 2 interlace 2 end_script# Pre-processing parameters: pre_processing_script: complex brukerGroupPhase 71.625 sinebell 60.0 zerofill 3 fft phase 103.0 32.0 0.0 reduce range 1050 1661 end_script# Reconstruction parameters: reconstruction_script: reconstruction_flag IHT stop_flag l2 dim 2: zerofill 2 phase 90.0 0 sinebell 60.0 end_script dim 3: zerofill 1 phase 0 0 sinebell 60.0 end_script end_script# Azara post processing instructions: post_processing_script: script_com 2: reverse base_poly 6.0 0.0 end_script script_com 3: reverse base_poly 6.0 0.0 end_script script_com 1: base_poly2 8.0 1.0 1.0 612.0 end_script end_script
Running the reconstruction
Once the script has been set up it can be run by pressing the button 'Process current script' in the GUI. This will automatically save the parameters in a .scr file with the prefix set by the outputFile option. Alternatively, the script can be saved manually using the 'Save file' button and then code run from the command line using:
cambridgecs -a script_name.scr
This is useful for longer reconstructions.
If processing via the GUI, press 'Process current script' to start the reconstruction and the GUI will automatically open a window showing the reconstruction progress. Once the reconstruction is completed, the standalone 'Graph plot' GUI is opened to display the reconstructed data.
Using the GUI
Calling the GUI with
cambridgecs -gresults in a blank GUI window shown below:
The four sections of the script, 'Data parameters', 'Pre-processing script', 'Reconstruction script' and 'Post-processing script', make up the four quadrants of the GUI.
The various elements of the script can be built up by filling in the text boxes, adding parameters from the 'Extra Params' dropdown menus or, in the Pre- and Post-processing sections, by clicking the green cross to reveal the list of available commands. Options from the 'Extra Params' sections can be removed at any point by unticking the option in the dropdown menu, while options added in the Pre- and Post-processing sections will have a red cross next to them which allows them to be removed.
At any time the script can be viewed by clicking 'Open script editor'. Options can be edited in the script editor and clicking the 'Sync text' button at the bottom of the script editor will sync any changes with the GUI.
A completed script in the GUI will look as follows (shown for the 2D example given above):
Once the script is completed press the 'Process current script' button to start the reconstruction. Starting the reconstruction automatically saves the script with the prefix given by the 'Output filename' parameter followed by '.scr'. The script can also be saved to a different filename using the 'Save file' button.
A saved script can be loaded directly into the GUI using the 'Load file' button, or by calling the programme with the script as an argument:
cambridgecs -g script.scr
Using the Data Viewer (Graph plot)
The standalone data viewer, 'Graph plot' enables the user to quickly view the output from Cambridge CS reconstructions. The Graph plot window looks as follows:
The viewer opens automatically if processing in the Cambridge CS GUI or can be launch from the command line as follows:
graphPlot spectrum.spc.par
where spectrum.spc.par is an Azara style parameter file (which is used for the output format of Cambridge CS).
The viewer enables a number of simple spectrum manipulations:
Interfacing with NMRPipe
CambridgeCS can be used in conjunction with NMRPipe to reconstruct undersampled data. Use the script nusExpand.tcl from NMRPipe to replace the missing points with zeros (more details on NMRPipe NUS processing can be found here). In this example, ser is the undersampled, raw Bruker data, vclist is the list of sampled points, '-sampleCount' is the number of sampled points and ser_full is the zero-padded data.
nusExpand.tcl -mode bruker -sampleCount 350 -off 1 \ -in ../550/ser -out ../550/ser_full -sample vclist
The data can then be converted using a standard bruk2pipe script. For example:
bruk2pipe -in ../550/ser_full \ -bad 0.0 -ext -noaswap -AMX -decim 16 -dspfvs 12 -grpdly -1 \ -xN 1024 -yN 64 -zN 96 \ -xT 512 -yT 32 -zT 48 \ -xMODE DQD -yMODE Echo-AntiEcho -zMODE Complex \ -xSW 10000.000 -ySW 2530.36 -zSW 1666.667 \ -xOBS 500.132 -yOBS 50.684 -zOBS 125.780 \ -xCAR 4.773 -yCAR 120.376 -zCAR 175.972 \ -xLAB HN -yLAB 15N -zLAB 13C \ -ndim 3 -aq2D Complex \ -out ./fid/test%03d.fid -verb -ov
Subsequently, the data can be processed in the direct dimension. The output should be a single file. Note that CambridgeCS processes the data 'in place' so there is no need to transpose the data after the direct dimension FFT, i.e. the direct dimension remains the first dimension.
xyz2pipe -in fid/test%03d.fid -x \| nmrPipe -fn SOL \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \| nmrPipe -fn ZF -zf 3 \| nmrPipe -fn FT \| nmrPipe -fn PS -p0 151 -p1 0.0 -di \| nmrPipe -fn EXT -x1 1050Pts -xn 1660Pts -sw \ -ov -out test.nus
At this point CambridgeCS can be used to reconstruct the data in the indirect dimensions. A script file is generated, this time omitting the 'Pre-processing parameters', since the direct dimension is already processed. The 'nmrPipe' flag and 'head' size (512) are included, as well as the 'max_points' in dim2 and dim3. The script can be run via the GUI, or from the command line.
# General properties: directory ~/data/CS/CScode/test_code/3D/HNCO/NMRPipe outputFile 550_pipe_cs threads 1 dataInput ~/data/CS/CScode/test_code/3D/HNCO/NMRPipe/test.nus rawDirectory ../550 nusList ../NMRPipe/vclist_coordinate nmrPipe head 512 direct-dim: npts 611 end_script indirect-dims: npts 6144 ndim 2 end_script# Reconstruction parameters: reconstruction_script: reconstruction_flag IHT stop_flag l2 dim 2: zerofill 2 phase 0 0 scaleT0 0.5 max_points 64 sinebell 60.0 end_script dim 3: zerofill 1 phase 0 0 scaleT0 0.5 max_points 96 sinebell 60.0 end_script end_script
The script generates an output file *_recon.spc.par which can be viewed with Graph Plot. In addition a file *.ft1 is created, which contains reconstructed time-domain data in the indirect dimension(s). This can be further processed with nmrPipe. For example:
nmrPipe -in 550_pipe_cs.ft1 -verb \| nmrPipe -fn TP \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \| nmrPipe -fn ZF -zf 2 -auto \| nmrPipe -fn FT \| nmrPipe -fn PS -p0 0 -p1 0 -di \| nmrPipe -fn TP \| pipe2xyz -out ft2/test%05d.ft2 -x -ovxyz2pipe -in ft2/test%05d.ft2 -z -verb \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 \| nmrPipe -fn ZF -zf 1 -auto \| nmrPipe -fn FT -neg \| nmrPipe -fn PS -p0 0 -p1 0 -di \| pipe2xyz -out recon/test%03d.ft3 -z -ov \#Output as a single file for viewing in CCPN Analysisxyz2pipe -in recon/test%03d.ft3 -x -verb \| nmrPipe -ov -out 550_pipe_cs.ft3/bin/rm -rf *.dat#Create 2D projections of the data.proj3D.tcl -in recon/test%03d.ft3/bin/rm -rf ft2
The scripts can easily be combined into a single processing script, with the command
set dSize = `getParm -in test.nus -parm NDSIZE -dim CUR_XDIM -fmt %.0f`
used to find the number of points in the direct dimension of the experiment after direct dimension processing. This can be passed to the reconstruction with the flag --npts0:
cambridgecs -a ./550_pipe_cs.scr --npts0 $dSize
The full script is shown below:
#!/bin/cshxyz2pipe -in fid/test%03d.fid -x \| nmrPipe -fn SOL \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \| nmrPipe -fn ZF -zf 3 \| nmrPipe -fn FT \| nmrPipe -fn PS -p0 151 -p1 0.0 -di \| nmrPipe -fn EXT -x1 1050Pts -xn 1660Pts -sw \ -ov -out test.nusset dSize = `getParm -in test.nus -parm NDSIZE -dim CUR_XDIM -fmt %.0f`echo "Direct dimension has $dSize points"cambridgecs -a ./550_pipe_cs.scr --npts0 $dSizenmrPipe -in 550_pipe_cs.ft1 -verb \| nmrPipe -fn TP \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 2 -c 0.5 \| nmrPipe -fn ZF -zf 2 -auto \| nmrPipe -fn FT \| nmrPipe -fn PS -p0 0 -p1 0 -di \| nmrPipe -fn TP \| pipe2xyz -out ft2/test%05d.ft2 -x -ovxyz2pipe -in ft2/test%05d.ft2 -z -verb \| nmrPipe -fn SP -off 0.5 -end 0.98 -pow 1 -c 0.5 \| nmrPipe -fn ZF -zf 1 -auto \| nmrPipe -fn FT -neg \| nmrPipe -fn PS -p0 0 -p1 0 -di \| pipe2xyz -out recon/test%03d.ft3 -z -ov \xyz2pipe -in recon/test%03d.ft3 -x -verb \| nmrPipe -ov -out 550_pipe_cs.ft3/bin/rm -rf *.datproj3D.tcl -in recon/test%03d.ft3/bin/rm -rf ft2