I am trying to write test case of spark scala application code. So I am planning to use SharedSparkSession for this purpose.
I've seen other framework such as com.holdenkarau but I am looking for any other alternative especially using SharedSparkSeesion.
So I tried finding sample examples using this SharedSparkSession from the web, but I am unable to do so.
If you have any example, please post.
Spark own test framework can be used in Scala, SparkSession present there. Some dependencies have to be included, for Maven below, can be converted to Sbt.
ScalaTest example: https://apache.googlesource.com/spark/+/master/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.suffix}</artifactId>
<version>${spark.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.suffix}</artifactId>
<version>${spark.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.suffix}</artifactId>
<version>${spark.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
Related
I am writing unit tests for a project which already has a few TestNG tests. Since I am using JUnit Jupiter (aka JUnit 5) to write new tests, I observed the problem that only TestNG tests are run when i do "mvn clean verify"
Below are my dependencies in pom.xml
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.5.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.2.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-junit-jupiter</artifactId>
<version>2.23.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<version>5.6.0</version>
<scope>test</scope>
</dependency>
And since surefire plugin from 2.22.0 onwards provides JUnit 5 support below is my plugin section for surefire
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
</plugin>
Is there any way I can keep both in the project and be able to run all tests through maven ? This is necesary so that code coverage report shows all tests together.
Also I noted that all tests(JUnit + TestNG) run together through maven if I use JUNIT 4 tests with the help of vintage-engine dependency. I am confused why this does not happen with JUnit 5.
It's supported in a "preview feature" style.
(1) Have the TestNGine in your test runtime dependencies.
(2) Help (force) Surefire to use the JUnit Platform provider
See this simple integration test pom.xml for a possible setup:
<dependencies>
<dependency> <!-- (1) -->
<groupId>com.github.testng-team</groupId>
<artifactId>testng-junit5</artifactId>
<version>0.0.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.6.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<dependencies>
<dependency> <!-- (2) -->
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit-platform</artifactId>
<version>2.22.2</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
I try to test something in a Kotlin project using JUnit 5 and Spring Boot, but I'm unable to inject a bean in my test class.
I tried many different annotations, but the injection nerver worked...
Here's my test class:
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
#SpringBootTest
#ExtendWith(SpringExtension::class)
class FooTest {
#Autowired
lateinit var repo: BarRepository
#BeforeAll
fun setup() {
}
#Test
fun testToto() {
}
}
With this annotations combination, the code raises the following exception:
java.lang.NoClassDefFoundError:org/springframework/boot/context/properties/source/ConfigurationPropertySource.
And I'm actually unable to find where does this exception come from... I tried to do some research about this exception, but I didn't find anything satisfying...
I finally found how to fix my problem. My Spring Boot version was initially "1.5.3", so I changed it my pom.xml to the "2.0.2" version. Now my tests run fine and my bean is correctly injected as expected. Here's the modified part of my pom.xml:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.2.RELEASE</version>
<relativePath/>
</parent>
Everything is fine after modifying the version.
Here are the usefull dependencies to test with Junit:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>junit</artifactId>
<groupId>junit</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.3.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
I guess you have an error in your dependencies. If you generate a new Spring Boot Kotlin project from https://start.spring.io/#!language=kotlin then customize your dependencies as following, it will work as expected:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>junit</artifactId>
<groupId>junit</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
Please also notice that you don't need to specify #ExtendWith(SpringExtension::class) since #SpringBootTest is already meta annotated with this annotation as of Spring Boot 2.1.
I want to implement a maven project, that helps me unit test a Hadoop MapReduce job. My biggest problem is defining the Maven dependencies to be able to make use of the test classes: MiniDFSCluster & MiniMRCluster.
I am using Hadoop 2.4.1. Any ideas?
In case someone else is still searchinf for an answer:
MiniMRCluster is now deprecated.
You can get MiniDFSCluster and MiniMRCluster in the dependency (shown for Gradle)
compile group: 'org.apache.hadoop', name: 'hadoop-minicluster', version: '2.7.2'
The dependency is basically just a pom file that lists out the dependencies in this package. For those who want to look this up, MiniDFSCluster is in the artifact hadoop-hdfs:tests
You don't have to use the dependencies from the Cloudera repository
Guess I figured it out. In your Maven pom file, first add a new repository:
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
Then add the following to your project dependencies
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-auth</artifactId>
<version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<version>2.0.0-mr1-cdh4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.0.0-cdh4.3.0</version>
<classifier>tests</classifier>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.0.0-cdh4.3.0</version>
<classifier>tests</classifier>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.0.0-mr1-cdh4.3.0</version>
</dependency>
In case someone is interested to get the whole project (unit test for the famous WordCount MapReduce job, I am willing to share it)
I have the following test method
#RunWith(MyCustomJUnit4Runner.class)
public class MyIntegrationTestWithACustomIntegrationRunner{
#Test
public void ensureStuffIsWiredCorrectly(Dependency myDependency){
}
}
This method is ran with a custom runner that injects the wired dependency.
JMockit also injects mocks in this scenario.
Normally I would want the jmockit behaviors, however this case I need the platform injected item rather than the mocked version.
Any advice would be appreciated.
Please feel free to comment on how I can clarify the obviously ambiguous question.
pom entries:
<dependencies>
<!-- CUSTOM INTEGRATION RUNNER -->
<dependency>
<groupId>com.clinkworks</groupId>
<artifactId>neptical</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.googlecode.jmockit</groupId>
<artifactId>jmockit</artifactId>
<version>1.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>[4.0,5.0)</version>
<scope>test</scope>
</dependency>
</dependencies>
You will have to upgrade JMockit to version 1.7 or newer:
<dependency>
<groupId>org.jmockit</groupId>
<artifactId>jmockit</artifactId>
<version>1.8</version>
<scope>test</scope>
</dependency
Using Solrj to connect to solr indexes. Used jar solr-solrj-3.6.1.jar which I got by adding below maven dependency
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>3.6.1</version>
</dependency>
I see that CommonsHttpSolrServer is deprecated and hence using HttpSolrServer . During run time i get the below error,
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient
Just adding solr-solrj-3.6.1.jar is not sufficient? Should I have to add more dependencies? I also tried adding httpclient 4.1, It started asking for org/apache/http/HttpRequestInterceptor.
Add this package to your maven dependencies:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.2.1</version>
</dependency>
Added below dependencies to get this working
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.2.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.5.8</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.8</version>
</dependency>
My simple project of solrj uses the following dependencies:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>4.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
<version>4.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.3</version>
</dependency>
</dependencies>
Between, Maven can automatically download them (for example, I started the Java Application of Maven in NetBeans, and then just added the dependencies). Additionally, I needed just to download the library of commons-logging 1.1.3 (http://commons.apache.org/proper/commons-logging/download_logging.cgi). You can read more about libraries and dependencies here: (http://wiki.apache.org/solr/Solrj).