Android Maps v2 API...How to calculate distance between two locations (Lat, long) - android-maps-v2

In my "native" Android app I'm trying to calculate the distance in feet and/or meters (as the crow flies) between two locations on a map. Ideally there would be a method that would take two LatLng values (as that is what I have readily available) as input and return the distance in meters and/or yards. As noted above I'm using Android Maps v2 API.
I've looked at MANY postings (20-30) regarding calculating distances and haven't found any that have helped me resolve my issue. Again this is a native Android application and I'm using Maps v2 API.
Not sure if this is the best approach but I'm currently trying to use the Location method:
distanceBetween(double startLatitude, double startLongitude, double endLatitude, double endLongitude, float[] results)
But unfortunately the app crashes every time because I get a null value for results. I have verified via the debugger that the latitude and longitude dummy values that I'm passing in are valid and that the value of results is null (hence the problem). Here is the LogCat:
04-04 23:48:04.770: D/onMapClick(17970): lat/lng: (32.90843038503306,-117.22537234425546)
04-04 23:48:08.710: D/dalvikvm(17970): threadid=1: still suspended after undo (sc=1 dc=1 s=Y)
04-04 23:48:08.710: D/dalvikvm(17970): GC_FOR_MALLOC freed 7010 objects / 641712 bytes in 121ms
04-04 23:48:24.980: D/AndroidRuntime(17970): Shutting down VM
04-04 23:48:24.980: W/dalvikvm(17970): threadid=1: thread exiting with uncaught exception (group=0x40020950)
04-04 23:48:25.100: E/AndroidRuntime(17970): FATAL EXCEPTION: main
04-04 23:48:25.100: E/AndroidRuntime(17970): java.lang.IllegalArgumentException: results is null or has length < 1
04-04 23:48:25.100: E/AndroidRuntime(17970): at android.location.Location.distanceBetween(Location.java:394)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.example.googleplacesandmaps.PlacesMapActivity$1.onMapClick(PlacesMapActivity.java:259)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.google.android.gms.maps.GoogleMap$6.onMapClick(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.google.android.gms.internal.t$a.onTransact(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at android.os.Binder.transact(Binder.java:249)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.google.android.gms.maps.internal.IOnMapClickListener$Stub$Proxy.onMapClick(IOnMapClickListener.java:93)
04-04 23:48:25.100: E/AndroidRuntime(17970): at maps.i.s.b(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at maps.y.v.c(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at maps.y.bf.onSingleTapConfirmed(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at maps.d.v.onSingleTapConfirmed(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at maps.d.j.handleMessage(Unknown Source)
04-04 23:48:25.100: E/AndroidRuntime(17970): at android.os.Handler.dispatchMessage(Handler.java:99)
04-04 23:48:25.100: E/AndroidRuntime(17970): at android.os.Looper.loop(Looper.java:123)
04-04 23:48:25.100: E/AndroidRuntime(17970): at android.app.ActivityThread.main(ActivityThread.java:4627)
04-04 23:48:25.100: E/AndroidRuntime(17970): at java.lang.reflect.Method.invokeNative(Native Method)
04-04 23:48:25.100: E/AndroidRuntime(17970): at java.lang.reflect.Method.invoke(Method.java:521)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:893)
04-04 23:48:25.100: E/AndroidRuntime(17970): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:651)
04-04 23:48:25.100: E/AndroidRuntime(17970): at dalvik.system.NativeStart.main(Native Method)
Here is the method I'm currently using to try and calculate the distance:
Location.distanceBetween(Double.parseDouble(user_latitude), Double.parseDouble(user_longitude),
fxdLatitude, fxdLongitude, results);
The results was defined as a Class variable: private float[] results = null;
Not sure what i'm doing wrong here.
Again I'd rather have a method that would take two LatLng's as input parameters because they are both readily available...but perhaps more importantly I only have one of the locations available as separate Longitude, Latitude values...the second location is only available as a LatLng value because I get it via onMapClick (when user taps screen). So to use the method distanceBetween I'd need someway to convert this LatLng value into a Latitude/Longitude pair to be able to use it. Anybody know how to do that?
Hopefully there is a simple way of calculating the distances and a simple way to "extract" the latitude/longitude from a LatLng value.

What about making Location object from the two ?
private float distanceBetween(LatLng latLng1, LatLng latLng2) {
Location loc1 = new Location(LocationManager.GPS_PROVIDER);
Location loc2 = new Location(LocationManager.GPS_PROVIDER);
loc1.setLatitude(latLng1.latitude);
loc1.setLongitude(latLng1.longitude);
loc2.setLatitude(latLng2.latitude);
loc2.setLongitude(latLng2.longitude);
return loc1.distanceTo(loc2);
}

You can use following function that accepts two LatLng for calculating distance between them. But before you use this make sure your LatLng variables are not null.
public static double distance(LatLng StartP, LatLng EndP) {
double lat1 = StartP.latitude;
double lat2 = EndP.latitude;
double lon1 = StartP.longitude;
double lon2 = EndP.longitude;
double dLat = Math.toRadians(lat2-lat1);
double dLon = Math.toRadians(lon2-lon1);
double a = Math.sin(dLat/2) * Math.sin(dLat/2) +
Math.cos(Math.toRadians(lat1)) * Math.cos(Math.toRadians(lat2)) *
Math.sin(dLon/2) * Math.sin(dLon/2);
double c = 2 * Math.asin(Math.sqrt(a));
return 6366000 * c;
}

private float[] results = null; is wrong.
You need to use private float[] results = new float[1]; and read the distance with results[0].

In Kotlin, you can:
Define a method to convert LatLng to Location:
fun LatLng.toLocation() = Location(LocationManager.GPS_PROVIDER).also {
it.latitude = latitude
it.longitude = longitude
}
Call it on each LatLng and use Location#distanceTo(...) method:
val distance = coords1.toLocation().distanceTo(coords2.toLocation())
Define an extension method for LatLngBounds:
fun LatLngBounds.computeDistance(): Float {
return northeast.toLocation().distanceTo(southwest.toLocation())
}

Yes, I think MaciejGórski is right.
The computed distance is stored in results[0]. If results has length 2 or greater, the initial bearing is stored in results[1]. If results has length 3 or greater, the final bearing is stored in results[2].

Related

How To Troubleshot This Exception :No match found for function signature IF(<BOOLEAN>, <NUMERIC>, <NUMERIC>)

I have a table defined 2 filed:integer id,varchar name
I can execute common query like this right:
ResultSet resultSet = statement.executeQuery("select \"id\",\"name\" from test.test01 where \"id\" in (1,2)");
Now,I want use "if" condition in select statement,execute this query:
ResultSet resultSet = statement.executeQuery("select \"id\",IF(\"id\">=1, 100, 200) as myid,\"name\" from test.test01 where \"id\" in (1,2)");
Caught Exception:
java.sql.SQLException: Error while executing SQL "select "id",IF("id">=1, 100, 200) as myid,"name" from test.test01 where "id" in (1,2)": From line 1, column 13 to line 1, column 33: No match found for function signature IF(<BOOLEAN>, <NUMERIC>, <NUMERIC>)
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at org.example.Client.main(Client.java:40)
Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 13 to line 1, column 33: No match found for function signature IF(<BOOLEAN>, <NUMERIC>, <NUMERIC>)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:932)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:917)
at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5266)
at org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1948)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:326)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6277)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6264)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1862)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1847)
at org.apache.calcite.sql.SqlAsOperator.deriveType(SqlAsOperator.java:133)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6277)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6264)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1862)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1847)
at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:463)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4409)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3652)
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1100)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1071)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:247)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1046)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:752)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:586)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:257)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:648)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:514)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:484)
at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:234)
at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:623)
at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:674)
at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 2 more
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature IF(<BOOLEAN>, <NUMERIC>, <NUMERIC>)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:599)
... 39 more
How To Troubleshot this problem, is there any thing wrong?
————————
Solution1:add connection config like below
URL url = Client.class.getResource("/model.json");
String str = URLDecoder.decode(url.toString(), "UTF-8");
Properties info = new Properties();
info.put("model", str.replace("file:", ""));
info.put("fun","hive");
Connection connection = DriverManager.getConnection("jdbc:calcite:", info);
Solution2:use case-when instead:
ResultSet resultSet = statement.executeQuery("select \"id\",\"id\"*10 as myid1,case when \"id\">1 then 100 else 200 end as myid2,\"name\" from test.test01 where \"id\" in (1,2,3)");
I was not able to find the IF function in the calcite docs. I think it is MySQL specific.
Instead you have CASE
I think the correct way to use it here would be
CASE WHEN
"id">=1 THEN 100
ELSE 200
END
Check out the docs here: https://calcite.apache.org/docs/reference.html
EDIT: good spot by OP, IF exists, but can only be used with the right databases and connection strings.

Redshift Exceptions on Insert

Using redshift jdbc driver: 1.2.16.1027
I routinely get exceptions such as:
java.sql.SQLNonTransientException: [Amazon][JDBC](10900) Not all parameters have been populated.
at com.amazon.exceptions.ExceptionConverter.toSQLException(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.common.SPreparedStatement.executeAnyUpdate(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.common.SPreparedStatement.executeUpdate(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.abc.dao.impl.AbcDaoImpl.storeAbc(AbcDaoImpl.java:80) ~[dao-1.0-SNAPSHOT.jar:?]
Also
com.amazon.support.exceptions.ErrorException: [Amazon](500310) Invalid operation: column "b" is of type double precision but expression is of type timestamp without time zone;
at com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.client.PGMessagingContext.handleErrorResponse(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.client.PGMessagingContext.handleMessage(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.communications.InboundMessagesPipeline.getNextMessageOfClass(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.client.PGMessagingContext.doMoveToNextClass(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.client.PGMessagingContext.getParameterDescription(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.client.PGClient.prepareStatement(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.dataengine.PGQueryExecutor.<init>(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.dataengine.PGDataEngine.prepare(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.common.SPreparedStatement.<init>(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.jdbc41.S41PreparedStatement.<init>(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.jdbc42.S42PreparedStatement.<init>(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.core.jdbc42.PGJDBC42PreparedStatement.<init>(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.redshift.core.jdbc42.PGJDBC42ObjectFactory.createPreparedStatement(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.common.SConnection.prepareStatement(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
at com.amazon.jdbc.common.SConnection.prepareStatement(Unknown Source) ~[RedshiftJDBC42-no-awssdk-1.2.16.1027.jar:?]
This insert works for thousands of records hundreds of batches sequentially, yet frequently, these exceptions get thrown.
The insert looks as follows:
private static final String INSERT_MANUAL =
"INSERT INTO abc(a, b, c, d, e, f, g, h, i) VALUES ";
private static final String INSERT_PART = "(?,?,?,?,?,?,?,?,?)";
--- Method:
StringBuilder query = new StringBuilder();
query.append(INSERT_MANUAL);
query.append(INSERT_PART);
for(int i = 1; i < abc.size();i++){
query.append(",").append(INSERT_PART);
}
try(
Connection connection = dataSource.getConnection();
PreparedStatement ps = connection.prepareStatement(query.toString())
){
int i = 1;
for(Abc q : abc){
ps.setTimestamp(i++,new Timestamp(q.a()));
ps.setInt(i++,q.b());
ps.setString(i++,q.c());
ps.setString(i++,q.d());
ps.setDouble(i++,q.e());
ps.setDouble(i++,q.f());
ps.setDouble(i++,q.g());
ps.setDouble(i++,q.h());
ps.setInt(i++,q.i());
}
ps.executeUpdate();
connection.commit();
} catch (SQLException e) {
throw new RuntimeException(e);
}
It isn't really possible for fields to get misaligned, or fields to not be set.
Writing a bulk insert like this proved more efficient by over 100x than doing jdbc batching.
Table:
create table abc
(
a timestamp,
b integer not null,
c varchar(45),
d varchar(12),
e double precision,
f double precision,
g double precision,
h double precision,
i integer
)

TESTNG ConcurrentModificationException while running with #Test invocation count greater than 1

I am experiencing the following issue while running my tests. However, it only happens when I am using #DataProvider with a #Test that has invocation count greater than 1. I am not sure if the following link below is the same issue, because if I read correctly this issue has already been fixed in higher versions of the Eclipse TESTNG plugin 6.9.13.X (using this 6.9.14.201610261028).
TestNG Reporter org.testng.reporters.JUnitReportReporter#7a79be86 failed java.util.ConcurrentModificationException
Here is a simple code that is encountering the same problem:
#Test(dataProvider = "data", timeOut = 6000, threadPoolSize = 10, invocationCount = 5)
public final void x(String a,
String b, String c, String d,
String e, String f, String g){
System.out.println("D"
+ " - "
+ java.lang.Thread.currentThread().getId()
+ " - "
+ new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS")
.format(new Date()));
}
Here is a sample output:
Thread - 1274 - 2016-10-28 08:47:23.889 PASSED: x("00336293615",
"02/24/2016", "03/10/2016", "3", "0", "3", "11.57")
Thread - 1275 - 2016-10-28 08:47:23.897
Thread - 1273 - 2016-10-28 08:47:23.898
java.util.ConcurrentModificationException at
java.util.ArrayList$Itr.checkForComodification(Unknown Source) at
java.util.ArrayList$Itr.next(Unknown Source) at
org.testng.reporters.TextReporter.logResults(TextReporter.java:80) at
org.testng.reporters.TextReporter.onFinish(TextReporter.java:32) at
org.testng.TestRunner.fireEvent(TestRunner.java:1239) at
org.testng.TestRunner.afterRun(TestRunner.java:1030) at
org.testng.TestRunner.run(TestRunner.java:636) at
org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at
org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) at
org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) at
org.testng.SuiteRunner.run(SuiteRunner.java:268) at
org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at
org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at
org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) at
org.testng.TestNG.runSuitesLocally(TestNG.java:1169) at
org.testng.TestNG.run(TestNG.java:1064) at
org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:236)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:81)
Sometimes it will also throw the following exceprions:
java.lang.NullPointerException at java.io.Writer.(Unknown
Source) at java.io.OutputStreamWriter.(Unknown Source) at
org.testng.remote.strprotocol.JsonMessageSender.sendMessage(JsonMessageSender.java:37)
at
org.testng.remote.strprotocol.MessageHub.sendMessage(MessageHub.java:43)
at
org.testng.remote.strprotocol.RemoteTestListener1.onTestSuccess(RemoteTestListener1.java:88)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1695)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1675)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:700) at
org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) at
org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:75)
at
org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
at java.util.concurrent.FutureTask.run(Unknown Source) at
java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source) at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at
java.lang.Thread.run(Unknown Source) java.lang.IllegalStateException:
Trying to send a message on a shutdown sender at
org.testng.remote.strprotocol.JsonMessageSender.sendMessage(JsonMessageSender.java:31)
at
org.testng.remote.strprotocol.MessageHub.sendMessage(MessageHub.java:43)
at
org.testng.remote.strprotocol.RemoteTestListener1.onTestStart(RemoteTestListener1.java:49)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1700)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1675)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:619) at
org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) at
org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:75)
at
org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
at java.util.concurrent.FutureTask.run(Unknown Source) at
java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source) at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at
java.lang.Thread.run(Unknown Source)
Please tell me if you guys need more information.
Thanks!!!

Implement dynamic location in PigStorage - right way

I'm trying to implement custom pig storer that based on org.apache.pig.builtin.PigStorage . In my storer I want to calculate the real hdfs location from the string that sent at 'INTO' word and other job properties.
-- pig
STORE data INTO 'MyObject' using ...
I extend PigStorage and override setStoreLocation function, like this:
public void setStoreLocation(String location, Job job) throws IOException {
// location is 'MyObject' here
// my manipulations on location
String newLocation = basePath + '/' + someVar + '/' + location;
super.setStoreLocation(newLocation, job);
}
It works and write the file into the newLocation (/basePath/someVar/MyObject) , but I'm getting following message in the log, how I can to avoid it?
java.io.FileNotFoundException: File hdfs://myMachine:8020/user/hdfs/MyObject does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.FileBasedOutputSizeReader.getOutputSize(FileBasedOutputSizeReader.java:65)
at org.apache.pig.tools.pigstats.JobStats.getOutputSize(JobStats.java:543)
at org.apache.pig.tools.pigstats.JobStats.addOneOutputStats(JobStats.java:567)
at org.apache.pig.tools.pigstats.JobStats.addOutputStatistics(JobStats.java:516)
at org.apache.pig.tools.pigstats.PigStatsUtil.addSuccessJobStats(PigStatsUtil.java:360)
at org.apache.pig.tools.pigstats.PigStatsUtil.accumulateStats(PigStatsUtil.java:257)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:286)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:226)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Looks like overriding of org.apache.pig.StoreFuncInterface#relToAbsPathForStoreLocation solves the problem:
#Override
public String relToAbsPathForStoreLocation(String location, Path curDir)
throws IOException {
return location;
}

Gradle test fails with error

I see next in my output:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> superClassName is empty!
I appreciate if someone knows how to fix it.
I've already took a look to the http://issues.gradle.org/browse/GRADLE-1682. But any solution didn't help
Created Gist with more information:
https://gist.github.com/emartynov/c88194c702b77f42a7c0
Here is stacktrace:
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':test'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.
java:72)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:49
)
at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskEx
ecuter.java:34)
at org.gradle.api.internal.changedetection.CacheLockHandlingTaskExecuter$1.run(CacheLockHandlingTaskExecuter.jav
a:34)
at org.gradle.internal.Factories$1.create(Factories.java:22)
at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:179)
at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:232)
at org.gradle.cache.internal.DefaultPersistentDirectoryStore.longRunningOperation(DefaultPersistentDirectoryStor
e.java:142)
at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateCacheAccess.longRunningOperation(DefaultTaskA
rtifactStateCacheAccess.java:83)
at org.gradle.api.internal.changedetection.CacheLockHandlingTaskExecuter.execute(CacheLockHandlingTaskExecuter.j
ava:32)
at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExec
uter.java:41)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.j
ava:51)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:52)
at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.j
ava:42)
at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:275)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.executeTask(DefaultTaskPlanExecutor.java:52)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.processTask(DefaultTaskPlanExecutor.java:38)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:30)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:84)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter$1.run(TaskCacheLockHandlingBuildEx
ecuter.java:31)
at org.gradle.internal.Factories$1.create(Factories.java:22)
at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:124)
at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:112)
at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:134)
at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateCacheAccess.useCache(DefaultTaskArtifactState
CacheAccess.java:79)
at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter.execute(TaskCacheLockHandlingBuild
Executer.java:29)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:165)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:113)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:81)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:38)
at org.gradle.launcher.exec.InProcessGradleLauncherActionExecuter.execute(InProcessGradleLauncherActionExecuter.
java:39)
at org.gradle.launcher.exec.InProcessGradleLauncherActionExecuter.execute(InProcessGradleLauncherActionExecuter.
java:25)
at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
at org.gradle.api.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:20
1)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:17
4)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:170)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:139)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
at org.gradle.launcher.Main.doAction(Main.java:48)
at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
at org.gradle.launcher.Main.main(Main.java:39)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:50)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:32)
at org.gradle.launcher.GradleMain.main(GradleMain.java:26)
Caused by: java.lang.IllegalArgumentException: superClassName is empty!
at org.gradle.api.internal.tasks.testing.detection.AbstractTestFrameworkDetector.getSuperTestClassFile(AbstractT
estFrameworkDetector.java:63)
at org.gradle.api.internal.tasks.testing.junit.JUnitDetector.processTestClass(JUnitDetector.java:51)
at org.gradle.api.internal.tasks.testing.detection.AbstractTestFrameworkDetector.processSuperClass(AbstractTestF
rameworkDetector.java:142)
at org.gradle.api.internal.tasks.testing.junit.JUnitDetector.processTestClass(JUnitDetector.java:54)
at org.gradle.api.internal.tasks.testing.detection.AbstractTestFrameworkDetector.processSuperClass(AbstractTestF
rameworkDetector.java:142)
at org.gradle.api.internal.tasks.testing.junit.JUnitDetector.processTestClass(JUnitDetector.java:54)
at org.gradle.api.internal.tasks.testing.detection.AbstractTestFrameworkDetector.processTestClass(AbstractTestFr
ameworkDetector.java:131)
at org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner$1.visitClassFile(DefaultTestClassScan
ner.java:58)
at org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner$ClassFileVisitor.visitFile(DefaultTes
tClassScanner.java:78)
at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:156)
at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:171)
at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:171)
at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:171)
at org.gradle.api.internal.file.collections.DirectoryFileTree.visitFrom(DirectoryFileTree.java:124)
at org.gradle.api.internal.file.collections.DirectoryFileTree.visit(DirectoryFileTree.java:114)
at org.gradle.api.internal.file.collections.FileTreeAdapter.visit(FileTreeAdapter.java:96)
at org.gradle.api.internal.file.CompositeFileTree.visit(CompositeFileTree.java:54)
at org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner.detectionScan(DefaultTestClassScanner
.java:56)
at org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner.run(DefaultTestClassScanner.java:50)
at org.gradle.api.internal.tasks.testing.processors.TestMainAction.run(TestMainAction.java:43)
at org.gradle.api.internal.tasks.testing.detection.DefaultTestExecuter.execute(DefaultTestExecuter.java:77)
at org.gradle.api.tasks.testing.Test.executeTests(Test.java:448)
at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:216)
at org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:122)
at org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147)
at org.gradle.api.tasks.testing.Test_Decorated.invokeMethod(Unknown Source)
at org.gradle.util.ReflectionUtil.invoke(ReflectionUtil.groovy:23)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTas
kFactory.java:161)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTas
kFactory.java:156)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:510)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:499)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.
java:64)
... 57 more
Sounds like GRADLE-1682. Turning off class file scanning should solve the problem:
tasks.withType(Test) {
scanForTestClasses = false
include "**/*Test.class" // whatever Ant pattern matches your test class files
}