Extracting SST time series at multiple lat, lon locations using CDO - cdo-climate

Background: I am working with satellite observations of SST (time, lat, lon) from the CoRTAD SST dataset (netCDF file). I have a set of (lon,lat) coordinates following the coastal contour of Portugal (called below midshelf locations). I want to extract the SST time series at each of these midshelf locations, average them and subtract the SST at the same latitudes but a fixed longitude to give a coastal SST index.
The midshelf lon,lats were determined first from a nautical chart, which were then linearly interpolated to the lon,lats in the CoRTAD grid.
How can this be done using CDO?
The lon,lats from the nautical map are given below.
midshelf lon,lats from the nautical map:
-8.000 43.860
-9.000 43.420
-9.350 43.220
-9.388 42.893
-9.000 42.067
-8.935 41.308
-9.000 40.692
-9.278 40.000
-9.324 39.550
-9.518 39.387
-9.777 38.883
-9.285 38.378
-8.909 38.293
-8.951 38.000
-8.965 37.953
-8.917 37.833
-8.913 37.667
-8.915 37.500
-8.975 37.333
-9.017 37.167
-9.045 37.000

So here is my attempt to answer the question as it was stated in the comments (i.e. you wanted an index which was the midshelf locations averaged and then subtracting the same latitude SST sampled at Longitude=9E). I assume the locations are stored pair-wise in a text file called "locations.txt" as in your question above. The loop part of the answer is from one of this question's solutions.
# first loop over the pairs of indices in the text files.
while read -r -a fields; do
for ((i=0; i < ${#fields[#]}; i += 2)); do
# precise lon/lat for sampled mid-shelf
cdo remapnn,"lon=${fields[i]}/lat=${fields[i+1]}" in.nc pt_${i}.nc
# same lat but lon=9E (change as wanted)
cdo remapnn,"lon=9/lat=${fields[i+1]}" in.nc 9E_${i}.nc
done
done < example.txt
# now take the ensemble average over the points.
cdo ensmean pt_*.nc mid_shelf_sst.nc
cdo ensmean 9E_*.nc mid_shelf_9E.nc
# and calculate the index
cdo sub mid_shelf_sst.nc mid_shelf_9E.nc SST_index.nc

Related

How do I get two coefficients from a set of regressions plotted on the same chart?

I am estimating a model in Stata 16 over several subsamples. I want a chart comparing two coefficients of interest over the different subsamples, with axis labels showing which subsample it comes from.
Is there a way to combine both of these on the same panel, with the mileage estimates in one colour and the trunk space in another?
The closest I can get using coefplot is a tiled plot with a set of coefficients of one variable in one panel, and the coefficients for the other variable in another panel (see toy example below). Any idea how to get both on the same panel?
webuse auto
forval v=2/5 {
reg price trunk mpg if rep78==`v'
est store reg_`v'
}
coefplot reg_2 || reg_3 || reg_4 || reg_5, keep(trunk mpg) bycoefs vertical
There's likely a more elegant way to do this with coefplot, but until someone posts that solution: you can use matrices to brute force coefplot into behaving the way you'd like. Specifically, define as many matrices as you have unique covariates, with each matrix's dimension being #specs x 3. Each row will contain the covariate's estimated coefficient, lower CI, and upper CI for a particular model specification.
This works because coefplot assigns the same color to all quantities associated with plot (as defined by coefplot's help file). plot is usually a stored model from estimates store, but by using the matrix trick, we've shifted plot to be equivalent to a specific covariate, giving us the same color for a covariate across all the model specifications. coefplot then looks to the matrix's rows to find its "categorical" information for the labeled axis. In this case, our matrix's rows correspond to a stored model, giving us the specification for our axis labels.
// (With macros for the specification names + # of coefficient
// matrices, for generalizability)
clear *
webuse auto
// Declare model's covariates
local covariates trunk mpg
// Estimate the various model specifs
local specNm = "" // holder for gph axis labels
forval v=2/5 {
// Estimate the model
reg price `covariates' if rep78==`v'
// Store specification's name, for gph axis labels
local specNm = "`specNm' reg_`v'"
// For each covariate, pull its coefficient + CIs for this model, then
// append that row vector to a new matrix containing that covariate's
// b + CIs across all specifications
matrix temp = r(table)
foreach x of local covariates{
matrix `x' = nullmat(`x') \ (temp["b","`x'"], temp["ll","`x'"], temp["ul","`x'"])
}
}
// Store the list of 'new' covariate matrices, along w/the
// column within this matrix containing the coefficients
global coefGphList = ""
foreach x of local covariates{
matrix rownames `x' = `specNm'
global coefGphList = "$coefGphList matrix(`x'[,1])"
}
// Plot
coefplot $coefGphList, ci((2 3)) vertical

How can compute "Mean Absolute Error (MAE)" by CDO

I did not find as a new student how to calculate "Mean Absolute Error (MAE)" by CDO!
Can you assist us with how to compute by CDO!
if you have your observations (or analysis?) in obs.nc and model in model.nc then you can calculate the MAE in the following way
# calculate the difference
cdo sub model.nc obs.nc diff.nc
# absolute value
cdo abs diff.nc abs.nc
# time mean
cdo timmean abs.nc MAE.nc
or piping it all in one line:
cdo timmean -abs -sub model.nc obs.nc MAE.nc
If instead of the temporal mean you need the spatial MAE then of course you replace timmean with fldmean

How can I generate a square wave plot of a pulse train of multiple signals from the data in a csv file (in Linux)?

For instance, given the data in a text file:
10:37:18.459 1
10:37:18.659 0
10:37:19.559 1
How could this be displayed as an image that looked like a square wave that correctly represented the high time and low time? I am trying both gnuplot and scipy. The result should ultimately include more than one sensor, and all plots would have to be displayed above one another so as to show a time delta.
The code in the following link creates a square wave from the formulas listed,
link to waveforms. How can the lower waveform (pwm) be driven by the numbers above if they were in a file (to show a high state for 200 ms, then a low state for 100 ms, and finally a high state)?
If I understood your question correctly you want to plot a step function based on timedata. To avoid further guessing please specify in more detail.
In gnuplot there is the plotting style with steps. Check help steps.
Code:
### display waveform as steps
reset sesion
$Data <<EOD
10:37:18.459 1
10:37:18.659 0
10:37:19.559 1
10:37:19.789 0
10:37:20.123 1
10:37:20.456 0
10:37:20.789 1
EOD
set yrange [-0.05:1.2]
myTimeFmt = "%H:%M:%S" # input time format
set format x "%M:%.1S" time # output time format on x axis
plot $Data u (timecolumn(1,myTimeFmt)):2 w steps lc rgb "red" lw 2 ti "my square wave"
### end of code
Result:
The answer I ended up with was:
file_info = os.stat( self.__outfile)
if file_info.st_size:
x,y,z,a = np.genfromtxt( self.__outfile, delimiter=',',unpack=True )
fig = plt.figure(self.__outfile)
ax = fig.add_subplot(111)
fig.canvas.draw()
test_array = [(datetime.datetime.utcfromtimestamp(e2).strftime('%d_%H:%M:%S.%f')).rstrip('0') for e2 in x]
plt.xticks(x, test_array)
l1, = plt.plot(x,y, drawstyle='steps-post')
l2, = plt.plot(x,a-2, drawstyle='steps-post')
l3, = plt.plot(x,z-4, drawstyle='steps-post')
ax.grid()
ax.set_xlabel('Time (s)')
ax.set_ylabel('HIGH/LOW')
ax.set_ylim((-6.5,1.5))
ax.set_title('Sensor Sequence')
fig.autofmt_xdate()
ax.legend([l1,l2, l3],['sprinkler','lights', 'alarm'], loc='lower left')
plt.show()
I had a input file that had convertDateToFloat values in it. That was passed in to this function. The name is perhaps misleading (__outfile), but on the previous function, it was the output.

can we use data generator for regression? (Keras, python)

I want to augmente my data using data generator in keras as below:
datagen = ImageDataGenerator(
featurewise_center=True, # set input mean to 0 over the dataset
samplewise_center=True, # set each sample mean to 0
featurewise_std_normalization=True, # divide inputs by std of the dataset
samplewise_std_normalization=True, # divide each input by its std
zca_whitening=True, # apply ZCA whitening
rotation_range=0, # randomly rotate images in the range (degrees, 0 to 180)
rescale=1./255,
shear_range=0.2,
zoom_range=0,
width_shift_range=0, # randomly shift images horizontally (fraction of total width)
height_shift_range=0, # randomly shift images vertically (fraction of total height)
horizontal_flip=True, # randomly flip images
vertical_flip=True) # randomly flip images
but I use this network for regression not classification. I have my doubts that datagenerator set new output values or not? Because If I used 0 or 1 classification problem then datagenerator could flip or rotate data without changing output but in here it should change output like input. Does that do this?
Thanks in advance.

subplots only plotting 1 plot using pandas

I am trying to get two plots on one figure using matplotlib's subplots() command. I want the two plots to share an x-axis and have one legend for the whole plot. The code I have right now is:
observline = mlines.Line2D([], [], color=(1,0.502,0),\
markersize=15, label='Observed',linewidth=2)
wrfline=mlines.Line2D([], [], color='black',\
markersize=15, label='WRF',linewidth=2)
fig,axes=plt.subplots(2,1,sharex='col',figsize=(18,10))
df08.plot(ax=axes[0],linewidth=2, color=(1,0.502,0))\
.legend(handles=[observline,wrfline],loc='lower center', bbox_to_anchor=(0.9315, 0.9598),prop={'size':16})
axes[0].set_title('WRF Model Comparison Near %.2f,%.2f' %(lat,lon),fontsize=24)
axes[0].set_ylim(0,360)
axes[0].set_yticks(np.arange(0,361,60))
df18.plot(ax=axes[1],linewidth=2, color='black').legend_.remove()
plt.subplots_adjust(hspace=0)
axes[1].set_ylim(0,360)
axes[1].set_yticks(np.arange(0,361,60))
plt.ylabel('Wind Direction [Degrees]',fontsize=18,color='black')
axes[1].yaxis.set_label_coords(-0.05, 1)
plt.xlabel('Time',fontsize=18,color='black')
#plt.savefig(df8graphfile, dpi = 72)
plt.show()
and it produces four figures, each with two subplots. The top is always empty. The bottom is filled for three of them with my 2nd dataframe. The indices for each dataframe is a datetimeindex in the format YYYY-mm-DD HH:MM:SS. The data is values from 0-360 nearly randomly across the whole time series, which is for two months.
Here is an example of each figure produced: