Filtering procedures for DATAΒΆ

Here a few scripts to reduce size of GCM data.

We decided to provide ACMAD with three GMC datasets:

  • ERAINTERIM (size ~ 331 GB)
  • HadGEM2 (size ~ 5.1 TB)
  • MPI-ESM-MR (size ~ 5TB)

We need to resize the size of such datasets..

Here a script how to this on Erainterim:

#!/bin/bash
# a small script to reduce datasets
climadods="http://clima-dods.ictp.it/data/d9/ERAIN150/"
starty=1979
stopy=1989

 for (( year = $starty; year < $stopy; year ++ ))
 do
 mkdir -p $year
 for var in air hgt rhum uwnd vwnd
   do
       for time in 00 06 12 18
           do
             get -c $climadods/$year/$var.$year.$time.nc
           done
             ncpdq -U  $var.$year.$time.nc  $var.$year.$time.unpacked.nc
             ncks -d latitude,-60.0,50.0 -d longitude,320.0,45.0 $var.$year.$time.unpacked.nc $var.$year.$time.unpack-africa.nc
             ncpdq  $var.$year.$time.unpack-africa.nc  $var.$year.$time.africa.nc
          done
        done

In the above script we are filtering out a box of 320- 45 degree of longitude and -60 50 degre of latitude, a box we consider ok for Africa simulation.

Two other scripts to download and filter HadGEM2 and MPI-ESM-MR data sets are here included:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
#!/bin/bash
# a small script to reduce datasets for HadGEM2
# written by S.Cozzini Seaquest-Infotel srl 

#TODO:
#  -add reduction function by means of nco operators
#  -more sanity checks 
#  -some command line options.. 
#  -better coding

# data server at ICTP: please check it ! 
climadods="http://clima-dods.ictp.it/data/d8/cordex/HadGEM2/"

#### data types ############
# data types from 2005 on. RCP85 

#datatyp="RCP85/"
#basename="ev_HadGEM2-ES_rcp85_r1i1p1_"

#  RF is from 1949-2005: 
datatyp="RF/"
basename="ev_HadGEM2-ES_historical_r1i1p1_"


######  range of years.. ##############
starty=1990
stopy=1991


## nothing to be changed below this line #

for var in hus_6hrL ta_6hrL ua_6hrL va_6hrL 

do 
 dir=${var%%_*}
 echo $dir 
 path=$datatyp"/"$dir
 if [ ! -d "$path" ] ; 
  then 
   echo " dir not existent; let us create it\n" 
   mkdir -p $path 
 fi   
 for (( year = $starty; year < $stopy; year ++ ))
 do
  year1=`expr $starty + 1`  
  filename=$var$basename$starty"12010600-"$year1"03010000.nc" 
#  filename2=$var$basename$year1"03010600-"$year1"06010000.nc" 
#  filename3=$var$basename$year1"06010600-"$year1"09010000.nc" 
#  filename4=$var$basename$year1"09010600-"$year1"12010000.nc" 
  echo $filename  "  \n"
  local_file=$path"/"$filename
  if [ ! -f "$local_file" ] ;
   then 
    echo " file not found locally.. downloading..\n" 
    remotepath=$climadods$datatyp$dir"/"$filename 
    echo "get this"  $remotepath  
    wget $remotepath -O  $path"/"$filename
  fi 
# here goes reduction procedure..
    

  done
done

#special variable psl
# different name and also one full year of data in one single file..

  
path=$datatyp"/psl"
 if [ ! -d "$path" ] ; 
  then 
   echo " dir not existent; let us create it\n" 
   mkdir -p $path 
 fi   
  for (( year = $starty; year < $stopy; year ++ ))
  do
   year1=`expr $starty + 1`
   filename="psl_6hrPl"$basename$starty"12010600-"$year1"12010000.nc"
   echo $filename
   local_file=$path"/"$filename
   if [ ! -f "$local_file" ] ;
     then
     echo " file not found locally.. downloading..\n" 
     remotepath=$climadods$path"/"$filename
     echo "get this"  $remotepath  
     wget $remotepath -O  $path"/"$filename
   fi
  done

The following is for MPI-ESM-MR dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
#!/bin/bash
# a small script to reduce datasets for MPI-ESM 
# written by S.Cozzini Seaquest-Infotel srl 

#TODO:
#  -add reduction function by means of nco operators
#  -more sanity checks 
#  -some command line options.. 
#  -better coding

# data server at ICTP: please check it ! 
climadods="http://clima-dods.ictp.it/data/d8/cordex/MPI-ESM-MR/"

#### data types ############
# data types from 2005 on. RCP85 

#datatyp="RCP85/"
#basename="ev_HadGEM2-ES_rcp85_r1i1p1_"

#  RF is from 1949-2005: 
datatyp="RF/"
basename="_6hrLev_MPI-ESM-MR_historical_r1i1p1_"


######  range of years.. ##############
starty=1990
stopy=1991


## nothing to be changed below this line #

# Please note that few data that should be downloaded in any case:

# wget  fixed dir: http://clima-dods.ictp.it/data/d8/cordex/MPI-ESM-MR/fixed  
# wget  SST dir  : http://clima-dods.ictp.it/data/d8/cordex/MPI-ESM-MR/SST
# wget  http://clima-dods.ictp.it/data/d8/cordex/MPI-ESM-MR/ta/ta_6hrLev_MPI-ESM-MR_historical_r1i1p1_197001010000-197002010000.nc


for var in aps hus ta ua va 
do 
 path=$datatyp"/"$var
 if [ ! -d "$path" ] ; 
  then 
   echo " dir not existent; let us create it\n" 
   mkdir -p $path 
 fi   
 for (( year = $starty; year < $stopy; year ++ ))
  do
    year1=`expr $starty + 1`  
    for month in $(seq 1 12); 
     do 
       mb=`printf %02d $month`
       begin=$year$mb"010000-"
       month1=`expr $month + 1`
       echo $mb $month $month1
       em=`printf %02d $month1`
       end=$year$em"010000"
       if [ "$month" eq "12" ] ;
         then  
           end=$year1"01010000"
         fi
       filename=$var$basename$begin$end".nc" 
       echo $filename  "  \n"
       local_file=$path"/"$filename
         if [ ! -f "$local_file" ] ;
          then 
            echo " file not found locally.. downloading..\n" 
            remotepath=$climadods$datatyp$var"/"$filename 
            echo "get this"  $remotepath  
            wget $remotepath -O  $path"/"$filename
        fi 
# here goes reduction procedure..
    done    
  done
done