Related
I have an import query (table a) and an imported Excel file (table b) with records I am trying to match it up with.
I am looking for a method to replicate this type of SQL in M:
SELECT a.loc_id, a.other_data, b.stk
FROM a INNER JOIN b on a.loc_id BETWEEN b.from_loc AND b.to_loc
Table A
| loc_id | other data |
-------------------------
| 34A032B1 | ... |
| 34A3Z011 | ... |
| 3DD23A41 | ... |
Table B
| stk | from_loc | to_loc |
--------------------------------
| STKA01 | 34A01 | 34A30ZZZ |
| STKA02 | 34A31 | 34A50ZZZ |
| ... | ... | ... |
Goal
| loc_id | other data | stk |
----------------------------------
| 34A032B1 | ... | STKA01 |
| 34A3Z011 | ... | STKA02 |
| 3DD23A41 | ... | STKD01 |
All of the other queries I can find along these lines use numbers, dates, or times in the BETWEEN clause, and seem to work by exploding the (from, to) range into all possible values and then filtering out the extra rows. However I need to use string comparisons, and exploding those into all possible values would be unfeasable.
Between all the various solutions I could find, the closest I've come is to add a custom column on table a:
Table.SelectRows(
table_b,
(a) => Value.Compare([loc_id], table_b[from_loc]) = 1
and Value.Compare([loc_id], table_b[to_loc]) = -1
)
This does return all the columns from table_b, however, when expanding the column, the values are all null.
This is not very specific "After 34A01 could be any string..." in trying to figure out how your series progresses.
But maybe you can just test for how a value "sorts" using the native sorting function in PQ.
add custom column with table.Select Rows:
= try Table.SelectRows(TableB, (t)=> t[from_loc]<=[loc_id] and t[to_loc] >= [loc_id])[stk]{0} otherwise null
To reproduce with your examples:
let
TableB=Table.FromColumns(
{{"STKA01","STKA02"},
{"34A01","34A31"},
{"34A30ZZZ","34A50ZZZ"}},
type table[stk=text,from_loc=text,to_loc=text]),
TableA=Table.FromColumns(
{{"34A032B1","34A3Z011","3DD23A41"},
{"...","...","..."}},
type table[loc_id=text, other data=text]),
//determine where it sorts and return the stk
#"Added Custom" = Table.AddColumn(#"TableA", "stk", each
try Table.SelectRows(TableB, (t)=> t[from_loc]<=[loc_id] and t[to_loc] >= [loc_id])[stk]{0} otherwise null)
in
#"Added Custom"
Note: if the above algorithm is too slow, there may be faster methods of obtaining these results
I am new to image processing. We have a requirement to get circle centers with sub pixel accuracy from an image. I have used median blurring to reduce the noise. A portion of the image is shown below. The steps I followed for getting circle boundaries is given below
Reduced the noise with medianBlur
Applied OTSU thresholding with threshold API
Identified circle boundaries with findContours method.
I get different results when used different kernel size for medianBlur. I selected medianBlur to keep edges. I tried kernel size 3, 5 and 7. Now I am confused to use the right kernel size for medianBlur.
How can I decide the right kernel size?
Is there any scientific approach to decide the right kernel size for medianBlur?
I will give you two suggestions here for how to find the centroids of these disks, you can pick one depending on the level of precision you need.
First of all, using contours is not the best method. Contours depend a lot on which pixels happen to fall within the object on thresholding, noise affects these a lot.
A better method is to find the center of mass (or rather, the first order moments) of the disks. Read Wikipedia to learn more about moments in image analysis. One nice thing about moments is that we can use pixel values as weights, increasing precision.
You can compute the moments of a binary shape from its contours, but you cannot use image intensities in this case. OpenCV has a function cv::moments that computes the moments for the whole image, but I don't know of a function that can do this for each object separately. So instead I'll be using DIPlib for these computations (I'm an author).
Regarding the filtering:
Any well-behaved linear smoothing should not affect the center of mass of the objects, as long as the objects are far enough from the image edge. Being close to the edge will cause the blur to do something different on the side of the object closest to the edge compared to the other sides, introducing a bias.
Any non-linear smoothing filter has the ability to change the center of mass. Please avoid the median filter.
So, I recommend that you use a Gaussian filter, which is the most well-behaved linear smoothing filter.
Method 1: use binary shape's moments:
First I'm going to threshold without any form of blurring.
import diplib as dip
a = dip.ImageRead('/Users/cris/Downloads/Ef8ey.png')
a = a(1) # Use green channel only, simple way to convert to gray scale
_, t = dip.Threshold(a)
b = a<t
m = dip.Label(b)
msr = dip.MeasurementTool.Measure(m, None, ['Center'])
print(msr)
This outputs
| Center |
- | ----------------------- |
| dim0 | dim1 |
| (px) | (px) |
- | ---------- | ---------- |
1 | 18.68 | 9.234 |
2 | 68.00 | 14.26 |
3 | 19.49 | 48.22 |
4 | 59.68 | 52.42 |
We can now apply a smoothing to the input image a and compute again:
a = dip.Gauss(a,2)
_, t = dip.Threshold(a)
b = a<t
m = dip.Label(b)
msr = dip.MeasurementTool.Measure(m, None, ['Center'])
print(msr)
| Center |
- | ----------------------- |
| dim0 | dim1 |
| (px) | (px) |
- | ---------- | ---------- |
1 | 18.82 | 9.177 |
2 | 67.74 | 14.27 |
3 | 19.51 | 47.95 |
4 | 59.89 | 52.39 |
You can see there's some small change in the centroids.
Method 2: use gray scale moments:
Here we use the error function to apply a pseudo-threshold to the image. What this does is set object pixels to 1 and background pixels to 0, but pixels around the edges retain some intermediate value. Some people refer to this as a "fuzzy thresholding". These two images show the normal ("hard") threshold, and the error function clip ("fuzzy threshold"):
By using this fuzzy threshold, we retain more information about the exact (sub-pixel) location of the edges, which we can use when computing the first order moments.
import diplib as dip
a = dip.ImageRead('/Users/cris/Downloads/Ef8ey.png')
a = a(1) # Use green channel only, simple way to convert to gray scale
_, t = dip.Threshold(a)
c = dip.ContrastStretch(-dip.ErfClip(a, t, 30))
m = dip.Label(a<t)
m = dip.GrowRegions(m, None, -2, 2)
msr = dip.MeasurementTool.Measure(m, c, ['Gravity'])
print(msr)
This outputs
| Gravity |
- | ----------------------- |
| dim0 | dim1 |
| (px) | (px) |
- | ---------- | ---------- |
1 | 18.75 | 9.138 |
2 | 67.89 | 14.22 |
3 | 19.50 | 48.02 |
4 | 59.79 | 52.38 |
We can now apply a smoothing to the input image a and compute again:
a = dip.Gauss(a,2)
_, t = dip.Threshold(a)
c = dip.ContrastStretch(-dip.ErfClip(a, t, 30))
m = dip.Label(a<t)
m = dip.GrowRegions(m, None, -2, 2)
msr = dip.MeasurementTool.Measure(m, c, ['Gravity'])
print(msr)
| Gravity |
- | ----------------------- |
| dim0 | dim1 |
| (px) | (px) |
- | ---------- | ---------- |
1 | 18.76 | 9.094 |
2 | 67.87 | 14.19 |
3 | 19.50 | 48.00 |
4 | 59.81 | 52.39 |
You can see the differences are smaller this time, because the measurement is more precise.
In the binary case, the differences in centroids with and without smoothing are:
array([[ 0.14768417, -0.05677508],
[-0.256 , 0.01668085],
[ 0.02071882, -0.27547569],
[ 0.2137167 , -0.03472741]])
In the gray-scale case, the differences are:
array([[ 0.01277204, -0.04444567],
[-0.02842993, -0.0276569 ],
[-0.00023144, -0.01711335],
[ 0.01776011, 0.01123299]])
If the centroid measurement is given in µm rather than px, it is because your image file contains pixel size information. The measurement function will use this to give you real-world measurements (the centroid coordinate is w.r.t. the top-left pixel). If you do not desire this, you can reset the image's pixel size:
a.SetPixelSize(1)
The two methods in C++
This is a translation to C++ of the code above, including a display step to double-check that the thresholding produced the right result:
#include "diplib.h"
#include "dipviewer.h"
#include "diplib/simple_file_io.h"
#include "diplib/linear.h" // for dip::Gauss()
#include "diplib/segmentation.h" // for dip::Threshold()
#include "diplib/regions.h" // for dip::Label()
#include "diplib/measurement.h"
#include "diplib/mapping.h" // for dip::ContrastStretch() and dip::ErfClip()
int main() {
auto a = dip::ImageRead("/Users/cris/Downloads/Ef8ey.png");
a = a[1]; // Use green channel only, simple way to convert to gray scale
dip::Gauss(a, a, {2});
dip::Image b;
double t = dip::Threshold(a, b);
b = a < t; // Or: dip::Invert(b,b);
dip::viewer::Show(a);
dip::viewer::Show(b); // Verify that the segmentation is correct
dip::viewer::Spin();
auto m = dip::Label(b);
dip::MeasurementTool measurementTool;
auto msr = measurementTool.Measure(m, {}, { "Center"});
std::cout << msr << '\n';
auto c = dip::ContrastStretch(-dip::ErfClip(a, t, 30));
dip::GrowRegions(m, {}, m, -2, 2);
msr = measurementTool.Measure(m, c, {"Gravity"});
std::cout << msr << '\n';
// Iterate through the measurement structure:
auto it = msr["Gravity"].FirstObject();
do {
std::cout << "Centroid coordinates = " << it[0] << ", " << it[1] << '\n';
} while(++it);
}
Not sure why I'm having a difficult time with this, it seems so simple considering it's fairly easy to do in R or pandas. I wanted to avoid using pandas though since I'm dealing with a lot of data, and I believe toPandas() loads all the data into the driver’s memory in pyspark.
I have 2 dataframes: df1 and df2. I want to filter df1 (remove all rows) where df1.userid = df2.userid AND df1.group = df2.group. I wasn't sure if I should use filter(), join(), or sql For example:
df1:
+------+----------+--------------------+
|userid| group | all_picks |
+------+----------+--------------------+
| 348| 2|[225, 2235, 2225] |
| 567| 1|[1110, 1150] |
| 595| 1|[1150, 1150, 1150] |
| 580| 2|[2240, 2225] |
| 448| 1|[1130] |
+------+----------+--------------------+
df2:
+------+----------+---------+
|userid| group | pick |
+------+----------+---------+
| 348| 2| 2270|
| 595| 1| 2125|
+------+----------+---------+
Result I want:
+------+----------+--------------------+
|userid| group | all_picks |
+------+----------+--------------------+
| 567| 1|[1110, 1150] |
| 580| 2|[2240, 2225] |
| 448| 1|[1130] |
+------+----------+--------------------+
EDIT:
I've tried many join() and filter() functions, I believe the closest I got was:
cond = [df1.userid == df2.userid, df2.group == df2.group]
df1.join(df2, cond, 'left_outer').select(df1.userid, df1.group, df1.all_picks) # Result has 7 rows
I tried a bunch of different join types, and I also tried different
cond values:
cond = ((df1.userid == df2.userid) & (df2.group == df2.group)) # result has 7 rows
cond = ((df1.userid != df2.userid) & (df2.group != df2.group)) # result has 2 rows
However, it seems like the joins are adding additional rows, rather than deleting.
I'm using python 2.7 and spark 2.1.0
Left anti join is what you're looking for:
df1.join(df2, ["userid", "group"], "leftanti")
but the same thing can be done with left outer join:
(df1
.join(df2, ["userid", "group"], "leftouter")
.where(df2["pick"].isNull())
.drop(df2["pick"]))
I ran a regression in Stata:
reg y I.ind1990#I.year, nocons r
Then I exported the coefficient vector from Stata using
matrix x = e(b)
esttab matrix(x) using "xx.csv", replace plain
and loaded it in Python and pandas using
df = pd.read_csv('xx.csv', skiprows=1, index_col=[0]).T.dropna()
df.index.name = 'interaction'
df = df.reset_index()
ind1990 and year are numeric. But I have some odd values in my csv (year and ind are manually pulled out of interaction):
interaction y1 ind year
0 0b.ind1990#2001b.year 0.000000 0b 2001b
1 0b.ind1990#2002.year 0.320578 0b 2002
2 0b.ind1990#2003.year 0.304471 0b 2003
3 0b.ind1990#2004.year 0.271429 0b 2004
4 0b.ind1990#2005.year 0.295347 0b 2005
I believe that 0b is how Stata translates missing values aka NIU. But I can't make sense of the other non-numeric values.
Here's what I get for years (and there is both b and o as unexpected suffix:
array(['2001b', '2002', '2003', '2004', '2005', '2006', '2007', '2008',
'2009', '2010', '2011', '2012', '2013', '2014', '2015', '2004o',
'2008o', '2012o', '2003o', '2005o', '2006o', '2007o', '2009o',
'2010o', '2011o', '2013o', '2014o', '2015o', '2002o'], dtype=object)
and for ind1990 (where 0b is apparently NIU, but there are also o suffixes that I can't make sense of:
array(['0b', '10', '11', '12', '20', '31', '32', '40', '41', '42', '50',
'60', '100', '101', '102', '110', '111', '112', '120', '121', '122',
'122o', '130', '130o', '132', '140', '141', '142', '150', '151',
'152', '152o', '160', '161', '162', '171', '172', '180', '181',
'182', '190', '191', '192', '200', '201', '201o', '210', '211',
'220', '220o', '221', '221o', '222', '222o', '230', '231', '232',
'241', '242', '250', '251', '252', '261', '262', '270', '271',
'272o', '272'], dtype=object)
What do the b and o suffixes mean at the end of values of the interaction column?
This isn't an answer, but it won't go well as a comment and it may clarify the question.
The example here isn't reproducible without #FooBar's data. Here is another one that (a) Stata users can reproduce and (b) Python users can, I think, import:
. sysuse auto, clear
(1978 Automobile Data)
. regress mpg i.foreign#i.rep78, nocons r
note: 1.foreign#1b.rep78 identifies no observations in the sample
note: 1.foreign#2.rep78 identifies no observations in the sample
Linear regression Number of obs = 69
F(7, 62) = 364.28
Prob > F = 0.0000
R-squared = 0.9291
Root MSE = 6.1992
-------------------------------------------------------------------------------
| Robust
mpg | Coef. Std. Err. t P>|t| [95% Conf. Interval]
--------------+----------------------------------------------------------------
foreign#rep78 |
Domestic#2 | 19.125 1.311239 14.59 0.000 16.50387 21.74613
Domestic#3 | 19 .8139726 23.34 0.000 17.37289 20.62711
Domestic#4 | 18.44444 1.520295 12.13 0.000 15.40542 21.48347
Domestic#5 | 32 1.491914 21.45 0.000 29.01771 34.98229
Foreign#1 | 0 (empty)
Foreign#2 | 0 (empty)
Foreign#3 | 23.33333 1.251522 18.64 0.000 20.83158 25.83509
Foreign#4 | 24.88889 .8995035 27.67 0.000 23.09081 26.68697
Foreign#5 | 26.33333 3.105666 8.48 0.000 20.1252 32.54147
-------------------------------------------------------------------------------
. matrix b = e(b)
. esttab matrix(b) using b.csv, plain
(output written to b.csv)
The file b.csv looks like this:
"","b","","","","","","","","",""
"","0b.foreign#1b.rep78","0b.foreign#2.rep78","0b.foreign#3.rep78","0b.foreign#4.rep78","0b.foreign#5.rep78","1o.foreign#1b.rep78","1o.foreign#2o.rep78","1.foreign#3.rep78","1.foreign#4.rep78","1.foreign#5.rep78"
"y1","0","19.125","19","18.44444","32","0","0","23.33333","24.88889","26.33333"
Stata's notation here is accessible to non-Stata users. See enter link description here
I don't use esttab (a user-written Stata program) or Python (that's ignorance, not prejudice), so I can't comment beyond that.
I am comparing weights of credit rating determinants across Moody's and S&P's.
The goal of doing the bioprobit analysis is then to do a test whether the beta coefficients are the same between Moody's and S&P.
I wanna do this based on a Wald test, but I need the covariance matrix of the Beta’s. Could you please help me with the code for Stata how to get the covariance matrix??
Variables entering the model are S&Prat Mrat GDP Inflation Ratio etc
Thanks in advance
Based on #Nick Cox:
Example from Stata data (you need to install bioprobit which is user written command)
sysuse auto
bioprobit headroom foreign price length mpg turn
. bioprobit headroom foreign price length mpg turn
group(forei |
gn) | Freq. Percent Cum.
------------+-----------------------------------
1 | 52 70.27 70.27
2 | 22 29.73 100.00
------------+-----------------------------------
Total | 74 100.00
initial: log likelihood = -148.5818
rescale: log likelihood = -148.5818
rescale eq: log likelihood = -147.44136
Iteration 0: log likelihood = -147.44136
Iteration 1: log likelihood = -147.43958
Iteration 2: log likelihood = -147.43958
Bivariate ordered probit regression Number of obs = 74
Wald chi2(4) = 22.61
Log likelihood = -147.43958 Prob > chi2 = 0.0002
------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
headroom |
price | -.0000664 .0000478 -1.39 0.164 -.00016 .0000272
length | .0347597 .013096 2.65 0.008 .009092 .0604274
mpg | -.0118916 .0354387 -0.34 0.737 -.0813502 .0575669
turn | -.0333833 .0554614 -0.60 0.547 -.1420857 .0753191
-------------+----------------------------------------------------------------
foreign |
price | .0003981 .0001485 2.68 0.007 .0001071 .0006892
length | -.0585548 .0284639 -2.06 0.040 -.114343 -.0027666
mpg | -.0306867 .0543826 -0.56 0.573 -.1372745 .0759012
turn | -.3471526 .1321667 -2.63 0.009 -.6061946 -.0881106
-------------+----------------------------------------------------------------
athrho |
_cons | .053797 .3131717 0.17 0.864 -.5600082 .6676022
-------------+----------------------------------------------------------------
/cut11 | 2.72507 2.451108 -2.079014 7.529154
/cut12 | 3.640296 2.445186 -1.152181 8.432772
/cut13 | 4.227321 2.443236 -.561334 9.015975
/cut14 | 4.792874 2.452694 -.0143182 9.600067
/cut15 | 5.586825 2.480339 .7254488 10.4482
/cut16 | 6.381491 2.505192 1.471404 11.29158
/cut17 | 7.145783 2.529663 2.187735 12.10383
/cut21 | -21.05768 6.50279 -33.80292 -8.312449
-------------+----------------------------------------------------------------
rho | .0537452 .3122671 -.5079835 .5834004
------------------------------------------------------------------------------
LR test of indep. eqns. : chi2(1) = 0.03 Prob > chi2 = 0.8636
# results that are in `Stata's memory`
ereturn list
scalars:
e(rc) = 0
e(ll) = -147.4395814769408
e(converged) = 1
e(rank) = 17
e(k) = 17
e(k_eq) = 11
e(k_dv) = 2
e(ic) = 2
e(N) = 74
e(k_eq_model) = 1
e(df_m) = 4
e(chi2) = 22.60944901065799
e(p) = .0001515278365065
e(ll_0) = -147.4543291018424
e(k_aux) = 8
e(chi2_c) = .0294952498030625
e(p_c) = .8636405133599019
macros:
e(chi2_ct) : "LR"
e(depvar) : "headroom foreign"
e(predict) : "bioprobit_p"
e(cmd) : "bioprobit"
e(chi2type) : "Wald"
e(vce) : "oim"
e(opt) : "ml"
e(title) : "Bivariate ordered probit regression"
e(ml_method) : "d2"
e(user) : "bioprobit_d2"
e(crittype) : "log likelihood"
e(technique) : "nr"
e(properties) : "b V"
matrices:
e(b) : 1 x 17
e(V) : 17 x 17
e(gradient) : 1 x 17
e(ilog) : 1 x 20
functions:
e(sample)
#You need to use mat list e(V) to display the variance covariance matrix
mat list e(V)
symmetric e(V)[17,17]
headroom: headroom: headroom: headroom: foreign: foreign: foreign: foreign:
price length mpg turn price length mpg turn
headroom:price 2.280e-09
headroom:length -1.431e-07 .00017151
headroom:mpg 3.991e-07 .00018914 .0012559
headroom:turn 4.426e-07 -.00050302 .00027186 .00307597
foreign:price 1.124e-10 -4.999e-09 2.093e-08 2.079e-08 2.205e-08
foreign:length -5.846e-09 8.021e-06 9.950e-06 -.0000249 -2.087e-06 .00081019
foreign:mpg 1.712e-08 .00001035 .00006387 .00001352 1.254e-06 .0006546 .00295746
foreign:turn 1.145e-08 -.00002418 .00001022 .00015562 -.00001083 -.00028103 -.0001411 .01746805
athrho:_cons 2.360e-07 -.00004531 .0000684 .00005575 -2.010e-06 .00043717 -.00147713 -.00449239
cut11:_cons .0000134 .01507955 .07578798 .03653671 1.039e-06 .00068972 .00401168 .00211706
cut12:_cons .00001374 .01514192 .07570527 .03630636 9.488e-07 .0007133 .00386727 .00165474
cut13:_cons .00001393 .01520261 .07550433 .03603257 9.668e-07 .0007088 .00386171 .00165557
cut14:_cons .00001363 .01539981 .07532214 .03582323 1.042e-06 .00068687 .00392914 .00189195
cut15:_cons .00001264 .01584186 .07541396 .03541453 1.101e-06 .00068091 .0040106 .00209853
cut16:_cons .00001148 .01611862 .07562328 .03535426 1.052e-06 .00069849 .00401805 .00206701
cut17:_cons .00001055 .01602514 .07547739 .03620485 9.866e-07 .00069868 .00399718 .00207143
cut21:_cons 4.412e-07 .00073781 .00377201 .00190456 -.00058242 .13231539 .18778679 .51179829
athrho: cut11: cut12: cut13: cut14: cut15: cut16: cut17:
_cons _cons _cons _cons _cons _cons _cons _cons
athrho:_cons .09807649
cut11:_cons -.0064343 6.0079319
cut12:_cons .00229188 5.9652808 5.9789347
cut13:_cons .00187855 5.9546524 5.9639617 5.9694026
cut14:_cons -.00310632 5.9724552 5.9793328 5.9820512 6.0157096
cut15:_cons -.00783593 6.0300908 6.03522 6.0360956 6.0667389 6.1520838
cut16:_cons -.00756313 6.0745198 6.0789515 6.0788816 6.1081885 6.1880183 6.275988
cut17:_cons -.00673882 6.0811477 6.0851101 6.0844209 6.1128719 6.1897756 6.2679698 6.3991936
cut21:_cons -.13478036 .30582954 .28918756 .28844026 .29527602 .30401845 .30575462 .30503648
cut21:
_cons
cut21:_cons 42.286275
# If you want to use variance covariance matrix of first four variables
mat kk=e(V)
mat kkk=kk[1..4,1..4]
mat list kkk
symmetric kkk[4,4]
headroom: headroom: headroom: headroom:
price length mpg turn
headroom:price 2.280e-09
headroom:length -1.431e-07 .00017151
headroom:mpg 3.991e-07 .00018914 .0012559
headroom:turn 4.426e-07 -.00050302 .00027186 .00307597