Cannot create directory while running `sbt IntegrationTest/test` with HBaseTestingUtility.startMiniDFSCluster - unit-testing

When creating a mini HDFS cluster in integration tests with the help of HBaseTestingUtility.startMiniDFSCluster, tests are running fine in IntelliJ IDEA but fail when running via sbt IntegrationTest/test. An error looks like this:
22:00:38.430 [pool-5-thread-4] WARN o.a.h.hdfs.server.namenode.NameNode - Encountered exception during format:
java.io.IOException: Cannot create directory /Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/name-0-1/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:361)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:571)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:592)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:185)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1211)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:406)
at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1071)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:987)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:884)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:798)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:667)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:640)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1129)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1104)
...
...
22:00:38.435 [pool-5-thread-4] ERROR o.apache.hadoop.hdfs.MiniDFSCluster - IOE creating namenodes. Permissions dump:
path '/Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/data':
absolute:/Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/data
permissions: ----
...
...
[info] FooIntegrationTest:
[info] bar.foo.FooIntegrationTest *** ABORTED ***
[info] java.io.IOException: Cannot create directory /Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/name-0-1/current
...

Parallel execution of the test suites was causing the issue.
Set execution to serial in build.sbt:
IntegrationTest / parallelExecution := false, // Embedded HBase is having troubles when parallelled

Related

Bazel test does not work with mingw-gcc . How should I debug this?

I'm following this really basic example here to setup gtest with bazel:
http://google.github.io/googletest/quickstart-bazel.html
it works fine (and tests pass) but I would like to use mingw gcc compiler with this (not the default mysys). Upon running this I get a single test failed, with no output displayed.
I can easily build or run files using this compiler. My output is below ( if it helps at all ):
C:\Users\<user>\Desktop\testProj>bazel test --test_output=all //:hello_test --verbose_failures
DEBUG: Rule 'com_google_googletest' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "5cf189eb6847b4f8fc603a3ffff3b0771c08eec7dd4bd961bfd45477dd13eb73"
DEBUG: Repository com_google_googletest instantiated at:
C:/users/<user>/desktop/testproj/WORKSPACE:3:13: in <toplevel>
Repository rule http_archive defined at:
C:/users/<user>/_bazel_<user>/x2p76flj/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31: in <toplevel>
INFO: Analyzed target //:hello_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //:hello_test (see C:/users/<user>/_bazel_<user>/x2p76flj/execroot/__main__/bazel-out/x64_windows-fastbuild/testlogs/hello_test/test.log)
INFO: From Testing //:hello_test:
==================== Test output for //:hello_test:
================================================================================
Target //:hello_test up-to-date:
bazel-bin/hello_test.exe
INFO: Elapsed time: 0.269s, Critical Path: 0.08s
INFO: 2 processes: 2 local.
INFO: Build completed, 1 test FAILED, 2 total actions
//:hello_test FAILED in 0.0s
C:/users/<user>/_bazel_<user>/x2p76flj/execroot/__main__/bazel-out/x64_windows-fastbuild/testlogs/hello_test/test.log
INFO: Build completed, 1 test FAILED, 2 total actions
The log file is also empty, and shows what the stdout here shows
For anyone who runs into this issue in future, you simply need to add linkopts=["-static"], to your cc_test rule. Not sure why it wasn't correctly dynamically linking

Getting error on running test validator for solana

On running command solana-test-validator on windows system, getting an error
[2022-01-06T06:54:41.602352800Z INFO solana_test_validator] solana-validator 1.9.0 (src:7782d34b; feat:378846963)
[2022-01-06T06:54:41.602479300Z INFO solana_test_validator] Starting validator with: ArgsOs {
inner: [
"solana-test-validator",
],
}
[2022-01-06T06:54:41.602617400Z WARN solana_perf] CUDA is disabled
[2022-01-06T06:54:41.602720300Z INFO solana_perf] AVX detected
[2022-01-06T06:54:41.602823600Z INFO solana_perf] AVX2 detected
[2022-01-06T06:54:41.606436300Z INFO solana_faucet::faucet] Faucet started. Listening on: 0.0.0.0:9900
[2022-01-06T06:54:41.606635600Z INFO solana_faucet::faucet] Faucet account address: 2P4mpwfirxqrL3naJD7C5UHYynPgVugaRu9sBT6m73EZ
Ledger location: test-ledger
Log: test-ledger\validator.log
⠁
⠉ Initializing...
[2022-01-06T06:54:44.975790700Z INFO solana_ledger::blockstore] "test-ledger\\rocksdb" open took 3.3s
[2022-01-06T06:54:44.981740300Z INFO solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2022-01-06T06:54:44.982273700Z INFO solana_metrics::metrics] datapoint: shred_insert_is_full total_time_ms=0i slot=0i ⠒ Initializing...
[2022-01-06T06:54:45.595853800Z ERROR solana_ledger::blockstore] tar stdout:
[2022-01-06T06:54:45.596099200Z ERROR solana_ledger::blockstore] tar stderr: tar: Can't launch external program: bzip2
Error: failed to start validator: Failed to create ledger at test-ledger: blockstore error```
This is a current annoyance with natively using Windows with solana-test-validator. It shells out to tar with bzip2, which isn't available in the default Windows shell.
As a workaround, try installing Git BASH and then running solana-test-validator from a Git BASH shell.
Source code for the issue can be found at: https://github.com/solana-labs/solana/blob/f1e2598baa80a0ad4e8450c8b5e3c5ab164f501c/ledger/src/blockstore.rs#L3789-L3814 -- the j flag indicates to use bzip2

Quarkus native image build fails with unknown arguments

I am building a quarkus native executable image but it is failing because of unknown argument .
I have used quarkus.native.additional-build-args variable in property file but it is not working.
I am using java 11. Can
SLF4J: Found binding in [jar:file:/home/quarkus/.m2/repository/org/jboss/slf4j/slf4j-jboss-logging/1.2.0.Final/slf4j-jboss-logging-1.2.0.Final.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/quarkus/.m2/repository/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.jboss.slf4j.JBossLoggerFactory]
[INFO] [org.jboss.threads] JBoss Threads version 3.1.1.Final
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: /usr/src/app/target/quarkus-test-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-test-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /usr/src/app/target/quarkus-test-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-test-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 20.1.0 (Java Version 11.0.7)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] /opt/graalvm/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-run-time=com.wealdtech.hawk.HawkClient com.wealdtech.hawk.HawkCredential --allow-incomplete-classpath --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar quarkus-test-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services -H:NativeLinkerOption=-no-pie --no-server -H:-UseServiceLoaderFeature -H:+StackTrace quarkus-test-1.0.0-SNAPSHOT-runner
Error: Unknown argument: quarkus-test-1.0.0-SNAPSHOT-runner
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.640 s
[INFO]
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.9.0.Final:build (default) on project quarkus-test: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:307)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Do any one has any Idea??
Thank you!!!
I had the same issue, when using quarkus.native.additional-build-args in the applications.properties file. The issues I had was, that I defined multiple packages in the --initialize-at-build-time parameter, separated with a comma as: --initialize-at-build-time=javax.net.ssl,java.security
This needed to be masked as following:
--initialize-at-build-time=javax.net.ssl\\,java.security
Just in case anybody else is losing hair on this and ends up here. My build was failing in a similar way:
building quarkus jar
Error: Unknown argument: <yada, yada...>
looking at my application.properties I had some build args:
quarkus.native.additional-build-args=\
-H:+PrintClassInitialization,\
--report-unsupported-elements-at-runtime,\
--allow-incomplete-classpath,\
--initialize-at-run-time=a.b.c.d\\,\
a.b.c.e\\,\
a.b.c.f
Nothing obvious there then. Looking closer though:
--allow-incomplete-classpath,\
Had 2 white space characters at the end of the line after the ,\ removing the white spaces fixed it. OUCH!

How to use google provided template [pubsub to Datastore]?

I want to use this google provided template which stream data from pubsub to datastore.
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubsubToDatastore.java
I follow the step wrote this document.
https://github.com/GoogleCloudPlatform/DataflowTemplates
I pass this step.
mvn clean && mvn compile
But next step, the error occured.
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) # google-cloud-teleport-java ---
2018-08-17 13:36:19 INFO DataflowRunner:266 - PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 117 files. Enable logging at DEBUG level to see which files wi
ll be staged.
[WARNING]
java.lang.IllegalStateException: Missing required properties: errorTag
at com.google.cloud.teleport.templates.common.AutoValue_DatastoreConverters_WriteJsonEntities$Builder.build(AutoValue_DatastoreConverters_WriteJsonEntities.java:89)
at com.google.cloud.teleport.templates.PubsubToDatastore.main(PubsubToDatastore.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.348 s
[INFO] Finished at: 2018-08-17T13:36:20+09:00
[INFO] Final Memory: 59M/146M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project google-cloud-teleport-java: An exception occured while executing the Java class. Missing required propert
ies: errorTag -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Then, I tried DatastoreToPubsub template and GSCTextToDatastore template, and these were successful.
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/DatastoreToPubsub.java
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/TextToDatastore.java
So, I can't understand what is problem.
Where is wrong?
Please give me some advices...
Regards.
Looks like you found a bug in that particular DataflowTemplate whereby the pipeline is not configuring an error path even though a path is required when writing JSON entities. The fix is relatively simple and should be pushed to master shortly. In the meantime, you can get the pipeline working with two changes to the PubsubToDatastore pipeline code.
First modify the code so the PubsubToDatastoreOptions extends the ErrorWriteOptions interface. Your new options declaration should look something similar to the following:
interface PubsubToDatastoreOptions
extends
PipelineOptions,
PubsubReadOptions,
JavascriptTextTransformerOptions,
DatastoreWriteOptions,
ErrorWriteOptions {}
Then modify the code within the main method so that the pipeline configures an error TupleTag and routes any error messages to the LogErrors transform. This will ensure any data which fails to be output to Datastore is captured and stored on GCS. Your new main method should look something similar to the following:
TupleTag<String> errorTag = new TupleTag<String>(){};
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply(PubsubIO.readStrings()
.fromTopic(options.getPubsubReadTopic()))
.apply(TransformTextViaJavascript.newBuilder()
.setFileSystemPath(options.getJavascriptTextTransformGcsPath())
.setFunctionName(options.getJavascriptTextTransformFunctionName())
.build())
.apply(WriteJsonEntities.newBuilder()
.setProjectId(options.getDatastoreWriteProjectId())
.setErrorTag(errorTag)
.build())
.apply(LogErrors.newBuilder()
.setErrorWritePath(options.getErrorWritePath())
.setErrorTag(errorTag)
.build());

Java binding for LMDB, need help to refresh LMDB JNI

we are using LMDB within java application.
The Java bindings that are available are 1 year old.
I would like to refresh the LMDBJNI
https://github.com/chirino/lmdbjni
However, the project owner, did not provide any instructions on how to build his project.
So I cannot just clone his git repository, and drop the new version of LMDB ( https://git.gitorious.org/mdb/mdb.git ) c and h files, and get it rebuilt
It seems that underneath LMDB JNI Is using hawkjni, but that's as far as I had gotten.
these are the steps I tried
a) git clone https://github.com/chirino/lmdbjni.git
b) cd lmdbjni; mvn install
It finishes successfully, however the resulting JAR does not have the actual lmdb library compiled.
So my test program fails with
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no lmdbjni64-99-vspmaster-SNAPSHOT in java.library.path, no lmdbjni-99-vspmaster-SNAPSHOT in java.library.path, no lmdbjni in java.library.path]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
at org.fusesource.lmdbjni.JNI.<clinit>(JNI.java:41)
at org.fusesource.lmdbjni.Env.create(Env.java:42)
at org.fusesource.lmdbjni.Env.<init>(Env.java:36)
at com.db.locrefdcache.App.main(App.java:27)
... 6 more
c) then I figured, I may not just be able to run mvn install for lmdbjni, and instead I need to explicitly build it's 64 bit Linux subsystem
so I did
cd lmdbjni/lmdbjni-linux64
mvn install
There I can I see that its trying to run configure script (generated by autotools), but I get the
...
[INFO] checking lmdb.h usability... no
[INFO] checking lmdb.h presence... no
[INFO] checking for lmdb.h... no
[INFO] configure: error: cannot find headers for lmdb
[INFO] rc: 1
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
So what I do not quite understand is whether lmdb files (lmdb.h , mdb.c, midl.h ) need to be explicitly dropped somewhere, or if hawkjni actually needs to be ran prior on them and create some sort of 'intermediate' c and h files that later on get dropped into this build environment.
Update with Compile Error I am getting, when using the LMDBJNI deephacks fork
Reason for recompile: DeepHacks's LMDBJNI project had produced a maven archive for LMDBJNI with latest LMDB, however it was compiled with Java 8 (which we do not yet use). So I need to recompile it with Java 7.
I modified pom.xml and changed Sources 1.8 to 1.7
Then, mvn install -P linux64 produces an error
...
[INFO] [hawtjni:build {execution: default}]
[INFO] Extracting /home/dev01/.m2/repository/org/deephacks/lmdbjni/lmdbjni/0.1.3-SNAPSHOT/lmdbjni-0.1.3-SNAPSHOT-native-src.zip to /home/dev01/devel/3dp/lmdbjni/lmdbjni-linux64/target/native-build-extracted
[INFO] executing: /bin/sh -c make install
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] build failed: org.apache.maven.plugin.MojoExecutionException: Make based build did not generate: /home/dev01/devel/3dp/lmdbjni/lmdbjni-linux64/target/native-build/target/lib/liblmdbjni.so
You might have better luck with this fork, which is actively being maintained
https://github.com/deephacks/lmdbjni
They've also provided LMDB itself in Maven, you can see how that was setup here
https://github.com/deephacks/lmdb
It builds fine on my machine with Java 7. Did you provide the correct profile when building the packages? For linux you must use: mvn install -P linux64