StaMPS-InSAR research based on ENVISAT using LiDO3

Intro

This post is about working steps of StaMPS-InSAR using 13 ENVISAT images in LiDO3.

StamPS-InSAR

StaMPS (Stanford Method for Persistent Scatterers) is a software package that implements an InSAR persistent scatterer (PS) method developed to work even in terrains devoid of man-made structures and/or undergoing non-steady deformation. StaMPS/MTI (Multi-Temporal InSAR) is an extended version of StaMPS that also includes a small baseline method and a combined multi-temporal InSAR method. The original development of StaMPS was undertaken at Stanford University, but subsequent development of StaMPS and StaMPS/MTI has taken place at the University of Iceland, Delft University of Technology, and the University of Leeds. There are also contributions from users of the package based at other institutions1.

StaMPS PS analysis is performed within MATLAB, using the main function stamps(), which takes the first and last StaMPS-step as main arguments. To run StaMPS complete with default arguments, use stamps(1,8)2.

Train

The Toolbox for Reducing Atmospheric InSAR Noise - TRAIN - is developed in an effort to include current state of the art tropospheric correction methods into the default InSAR processing chain. Initial development was performed at the University of Leeds. The toolbox consists of a combination of command line scripts, shell scripts, and matlab scripts. More information on software is provided in Chapter 3. TRAIN is independent of the used InSAR processor, as long as the data convention is followed. The toolbox is compatible with the StaMPS software. Further initial efforts have been put to include TRAIN into the default -rate processing chain3.

Configuration

  • edit the StaMPS config file (e.g. via vim ./StaMPS/StaMPS_CONFIG.bash) to contain the correct paths to StAMPS, SNAPHU and triangle

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # set environment variables:
    export STAMPS="/home/username/StaMPS"
    export SNAP2STAMPS="/home/username/snap2stamps"
    ...
    #if triangle and snaphu are not installed through the repositories (i.e. compiled locally):
    export TRIANGLE_BIN="/home/username/software/triangle/bin"
    export SNAPHU_BIN="/home/username/software/snaphu/bin"
    ...
    export MATLABPATH=$STAMPS/matlab:`echo $MATLABPATH`
    ...
    # use points not commas for decimals, and give dates in US english
    export LC_NUMERIC="en_US.UTF-8"
    export LC_TIME="en_US.UTF-8"
    ...
    export SAR_TAPE="/dev/rmt/0mn"
    ...
    export PATH=${PATH}:$STAMPS/bin:$MATLABPATH:$SNAP2STAMPS/bin

  • source the StaMPS config file upon opening terminal by adding it to your .bashrc

    1
    2
    # source StaMPS
    source /home/username/StaMPS/StaMPS_CONFIG.bash

  • do the same thing for your TRAIN (APS_toolbox) config file

    • adjust path in the APS_CONFIG file
    1
    export APS_toolbox="/home/username/TRAIN"

    -source it upon opening a terminal

    1
    2
    # add TRAIN config
    source /home/username/TRAIN/APS_CONFIG.sh

  • install dependencies

gwak and tsch was already installed in LiDO3, thank god!

Process

Pre-stamps

In Stamps official tutorial there is ‘INSAR_master_date’ folder, after search answers in STEP forum, we found the perfect resolution,

I am about to start my step 4- PS processing the ‘PSI’ folder which i obtained from stamps export in windows environment has four folders in it ‘dem’, ‘diff0’, ‘geo’ and ‘rslc’. The following are my queries:

Can i proceed or should i rename any of the folders to proceed? In which path should i place this PSI folder?

  • Answer4

    1-move the ‘PSI’ folder to your Linux directory,

    2- use the shell to create a new folder, (LINUX)

    mkdir ~/INSAR_master_date

    3- copy the folder ‘PSI’ into this folder ‘INSAR_master_date’

    cp -/PSI -/INSAR_master_date

    Now you’re ready to start as explained in the previous post.

Pre-Matlab Processing

1
2
3
$ module avail
$ module matlab
$ matlab

Working in LiDO3 with ThinLinc Client

cd /work/smyumeng/ENCISAT_project

1
2
3
4
5
6
7
8
$ source /work/smyumeng/StaMPS/StaMPS_CONFIG.bash
$ module add matlab
$ mt_prep_snap 20070602 /work/smyumeng/ENVISAT_project/INSAR_20070602 0.4 3 2 50 200
# $ mt_prep_snap 20070602 /work/smyumeng/ENVISAT_project/INSAR_20070602 0.42 2 2 50 200

working...


Matlab

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>> ps_info
  1  17-Jun-2006     0 m
  2  22-Jul-2006   757 m
  3  26-Aug-2006   391 m
  4  04-Nov-2006  -805 m
  5  09-Dec-2006    18 m
  6  17-Feb-2007  -245 m
  7  28-Apr-2007  -168 m
  8  02-Jun-2007   -87 m
  9  07-Jul-2007  -261 m
 10  11-Aug-2007  -251 m
 11  15-Sep-2007   151 m
 12  20-Oct-2007  -277 m
 13  02-Feb-2008  -142 m
Number of stable-phase pixels: 1364212

Step 1: Load Data

Converts the data into the formats required for PS processing and stores them in matlab workspaces.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
>> stamps(1,1)

STAMPS: ########################################
STAMPS: ####### StaMPS/MTI Version 4.0b6 #######
STAMPS: #######  Beta version, Jun 2018  #######
STAMPS: ########################################

STAMPS: Will process current directory only

STAMPS: ########################################
STAMPS: ################ Step 1 ################
STAMPS: ########################################
STAMPS: Directory is PATCH_1

...
Rotating by -10.9782 degrees
PS_LOAD_INITIAL_GAMMA: Finished
STAMPS: Finished

Step 2: Estimate phase noise

This is an iterative step that estimates the phase noise value for each candidate pixel in every interferogram.

Error in step2

Index in position 1 is invalid. Array indices must be positive integers or logical values.

Error in ps_est_gamma_quick (line 236) ph_grid(grid_ij(i,1),grid_ij(i,2),:)=ph_grid(grid_ij(i,1),grid_ij(i,2),:)+shiftdim(ph_weight(i,:),-1);

Error in stamps (line 326)
ps_est_gamma_quick(est_gamma_parm);

Solution: 

You may try with this function I have modified time ago. I hope it works for you. [ps_load_initial_gamma.m](https://forum.step.esa.int/uploads/default/original/2X/6/66b076f51da0e68543126e37dccc38937f5806d6.m). It should be used on the stamps step 1. So before you override the current one on the stamps_v4.1b/matlab/ please make a backup. Then please run again the command stamps(1,1) so you re-run step1 to use this function Let me know if this helps[5].

1
2
3
4
5
6
7
8
>> stamps(2,2)

...working

PS_EST_GAMMA_QUICK: gamma_change_change=0.000000
GETPARM: gamma_change_convergence=0.005 
GETPARM: gamma_max_iterations=3 
PS_EST_GAMMA_QUICK: Finished

Step 3: PS selection

Pixels are selected on the basis of their noise characteristics. This step also estimates the percentage of random (non-PS) pixels in a scene from which the density per km2 can be obtained.

Processing is controlled by the following parameters:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
>> stamps(3,3)

STAMPS: ########################################
STAMPS: ####### StaMPS/MTI Version 4.0b6 #######
STAMPS: #######  Beta version, Jun 2018  #######
STAMPS: ########################################

working...

Warning: Not enough random phase pixels to set gamma threshold - using default threshold of 0.3 
> In ps_select (line 184)
In stamps (line 356) 
PS_SELECT: Initial gamma threshold: 0.300 at D_A=0.07 to 0.300 at D_A=0.40
PS_SELECT: 0 PS selected initially
PS_SELECT: Reestimation gamma threshold: 0.300 at D_A=0.07 to 0.300 at D_A=0.40
PS_SELECT: 0 ps selected after re-estimation of coherence
***No PS points left. Updating the stamps log for this****
PS_SELECT: Finished

Step 4:

No Aavailable PS points in every batch (1-6), still finding solution. If can not solve, then qiut ENVISAT PS processing in LiDO3.

Update: After setting clap_beta to 0.6, batch_3 resulted 6006 PS selected initally, so we reset parameter and redo stamps(1,5)

1

Step5-8

Matlab plot

After stamps(1,8), we copy the file folder /work/smyumeng/ENVISAT_PS/ (except subfolder /INSAR_20070602) to windows H:\StaMPS_result\ENVISAT_PS_1 using WinSCP for further processing.

To equalize the source .bash command in Windows, we use addpath in matlab.

1
2
3
4
5
6
7
>> getparm
'getparm' is not found in the current folder or on the MATLAB path, but exists in:
    D:\Software\StaMPS-master\matlab

# Change the MATLAB current folder or add its folder to the MATLAB path.

>> addpath 'D:\Software\StaMPS-master\matlab'

Error 1

1
2
3
4
5
6
7
8
9
>> ps_plot('v-do', 'ts');
This is Envisat, oscilator drift is being removed... 
Deramping computed on the fly. 
**** z = ax + by+ c
7224 ref PS selected
Unrecognized function or variable 'nanmean'.

Error in ps_plot (line 1579)
        ph_uw=ph_uw-repmat(nanmean(ph_uw(ref_ps,:),1),n_ps,1);

Solution

Install all matlab toolbox!

Slurm

This section is about test run for slurm to control batch jobs and cluster resources6.

Queue max. wlltime remake
short 02:00:00 /
med 08:00:00 /
long 2-00:00:00 /
ultralong 28-00:00:00 no GPU or “non-bolcking” nodes
testpart 02:00:00 use when instructed by LiDO3 administrators 
1
$ cat stamps_batch_script.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash -l
#SBATCH --partition=med
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=3
#SBATCH --time=0:30
#SBATCH --mem-per-cpu=100
#SBATCH --job-name=stampsbatch
#SBATCH --output=/work/smyumeng/project/ENVISAT/stampsbatch.out.txt
#SBATCH --constraint=cstd01
srun 



1
2
3
$ sbatch my_submit_script.sh

$ scontrol show job 13891

SBATCH to run jobs

Script dir: /work/smyumeng/project/script

  • Create script to atuo run stamps jobs in LiDO.
1
2
[smyumeng@gw02 scripts]$ echo > stamps_testrun.sh #create .sh file
[smyumeng@gw02 scripts]$
  • Edit stamps_testrun.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/bin/bash -l
#SBATCH --job-name=StampsTestRun
#SBATCH --time=0-1:00:00
#SBATCH --partition=short
#SBATCH --constraint=cquad01|cgpu01
# ask for ten compute cores on one compute node
#SBATCH --nodes=6 --ntasks-per-node=1 --cpus-per-task=20
#SBATCH --nodes=1 --ntasks-per-node=1 --cpus-per-task=20 #One node for ENVISAT
# memory requirement per core in megabytes
#SBATCH --mem-per-cpu=2G
#SBATCH --output=/work/smyumeng/tmp/slurm_job
#SBATCH --error=/work/smyumeng/tmp/slurm_job
# send mail when jobs starts, end, fails, gets requeued etc.
#SBATCH --mail-type=ALL
#SBATCH --mail-user=yuchi.meng@tu-dortmund.de


source /work/smyumeng/StaMPS/StaMPS_CONFIG.bash 
# cd /work/smyumeng/ENCISAT_project/
module add matlab
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_1;stamps(1,5);quit'
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_2;stamps(1,5);quit'
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_3;stamps(1,5);quit'
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_4;stamps(1,5);quit'
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_5;stamps(1,5);quit'
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project/PATCH_6;stamps(1,5);quit'
echo "executing SLURM_JOB_ID $SLURM_JOB_ID (SLURM_TASK_PID $SLURM_TASK_PID) on $SLURM_NODENAME"
sleep 10
echo "worker ($SLURM_NODENAME):stop"
wait

Error:

1
2
3
4
5
6
7
8
9
10
STAMPS: Will process current directory only
STAMPS: ########################################
STAMPS: ################ Step 1 ################
STAMPS: ########################################
STAMPS: Directory is scripts

PS_LOAD_INITIAL: Starting
PS_LOAD_INITIAL: Loading data into matlab...
{Error using load
Unable to read file '../day.1.in'. No such file or directory.

Solution:

1
missset the working dir in matlab command

1
2
module add matlab
srun matlab -nodisplay -nosplash -r 'cd /work/smyumeng/ENVISAT_project;stamps(1,5);quit