I have written a Map only hadoop job in which i have used MultipleOutputs concept. The problem here is, i want to test this code with MRUnit. I don't see any working example for MultipleOutputs testing.
My mapper code will be like,
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String inputString = value.toString();
String outputString = null;
Text resultValue = null;
String finalResult = null;
String exceptionMessage = null;
try {
outputString = processInput(dataSet, inputString);
} catch (MalformedURLException me) {
System.out.println("MalformedURLException Occurred in Mapper:"
+ me.getMessage());
exceptionMessage = me.getMessage();
} catch (SolrServerException se) {
System.out.println("SolrServerException Occurred in Mapper:"
+ se.getMessage());
exceptionMessage = se.getMessage();
}
if (outputString == null || outputString.isEmpty()
&& exceptionMessage != null) {
exceptionMessage = exceptionMessage.replaceAll("\n", ", ");
finalResult = inputString + "\t[Error] =" + exceptionMessage;
resultValue = new Text(finalResult);
multipleOutputs.write(SearchConstants.FAILURE_FILE,NullWritable.get(), resultValue);
} else {
finalResult = inputString + outputString;
resultValue = new Text(finalResult);
multipleOutputs.write(SearchConstants.SUCCESS_FILE,NullWritable.get(), resultValue);
}
}
Can anyone of you guys give me a working example of MRUnit test with MultipleOutputs?
Here's an example with a slightly simplified version of your class
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
import java.io.IOException;
public class SomeMapper extends Mapper<LongWritable, Text, NullWritable, Text> {
public static final String SUCCESS_FILE = "successFile";
private static MultipleOutputs<NullWritable, Text> multipleOutputs;
private static Text result = new Text();
#Override
public void setup(Context context) throws IOException, InterruptedException {
multipleOutputs = new MultipleOutputs<>(context);
super.setup(context);
}
#Override
public void map(LongWritable key, Text value, Mapper.Context context) throws IOException, InterruptedException {
String outputString = "some result"; // logic here
result.set(outputString);
multipleOutputs.write(SUCCESS_FILE, NullWritable.get(), result);
}
}
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
#RunWith(PowerMockRunner.class)
#PrepareForTest({MultipleOutputs.class, SomeMapper.class})
public class SomeMapperTest {
#Test
public void someTest() throws Exception {
MapDriver<LongWritable, Text, NullWritable, Text> mapDriver = MapDriver.newMapDriver(new SomeMapper());
mapDriver.withInput(new LongWritable(0), new Text("some input"))
.withMultiOutput(SomeMapper.SUCCESS_FILE, NullWritable.get(), new Text("some result"))
.runTest();
}
}
and the build.gradle
apply plugin: "java"
sourceCompatibility = 1.7
targetCompatibility = 1.7
repositories {
mavenCentral()
}
dependencies {
compile "org.apache.hadoop:hadoop-client:2.4.0"
testCompile "junit:junit:4.12"
testCompile("org.apache.mrunit:mrunit:1.1.0:hadoop2") {
exclude(group: "org.mockito")
}
testCompile "org.powermock:powermock-module-junit4:1.6.2"
testCompile "org.powermock:powermock-api-mockito:1.6.2"
}
Note the Mockito exclusion. Without it, I got the exception java.lang.NoSuchMethodError: org.mockito.mock.MockCreationSettings.getSerializableMode()Lorg/mockito/mock/SerializableMode; because that Hadoop dependency pulled in org.mockito:mockito-core:1.9.5, which conflicted with the Mockito version Powermock wanted to use.
You can find additional examples in MRUnit's org.apache.hadoop.mrunit.mapreduce.TestMultipleOutput unit tests.
Related
Google Cloud and to PubSub Service.
I have implemented a simple CustomAttributeReceiver in the lines of below sample code
https://github.com/googleapis/java-pubsub/blob/master/samples/snippets/src/main/java/pubsub/PublishWithCustomAttributesExample.java
How ever i am getting the below exception
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $ at com.google.gson.Gson.fromJson(Gson.java:944) at
com.google.gson.Gson.fromJson(Gson.java:1003) at
com.google.cloud.functions.invoker.NewBackgroundFunctionExecutor$TypedFunctionExecutor.serviceLegacyEvent(NewBackgroundFunctionExecutor.java:257) at
com.google.cloud.functions.invoker.NewBackgroundFunctionExecutor.serviceLegacyEvent(NewBackgroundFunctionExecutor.java:343) at
Can someone throw light on what i am missing here?
Publisher side
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
import com.google.api.core.ApiFuture;
import com.google.cloud.pubsub.v1.Publisher;
import com.google.protobuf.ByteString;
import com.google.pubsub.v1.PubsubMessage;
import com.google.pubsub.v1.TopicName;
import PubSubMessage;
TopicName topicName = TopicName.of(projectId, topicId);
System.out.println("informListenersAboutSucessfulRegisteration=" + topicName);
Publisher publisher = null;
try {
publisher = Publisher.newBuilder(topicName).build();
PubSubMessage newUserRegisterMsg = new PubSubMessage();
Map<String, String> attributes = new HashMap<String, String>();
attributes.put(PubSubMessage.newUserLanguage, newUserLanguage);
newUserRegisterMsg.setAttributes(attributes);
ByteString data = ByteString.copyFromUtf8("NewUserRegisteration");
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).putAllAttributes(attributes).build();
ApiFuture<String> messageIdFuture = publisher.publish(pubsubMessage);
String messageId = messageIdFuture.get();
System.out.println("Published message ID: " + messageId);
} catch (Exception e) {
Logger.getLogger(PubSubMessenger.name).log(Level.SEVERE, "Exception while publishing message", e);
} finally {
if (publisher != null) {
publisher.shutdown();
publisher.awaitTermination(1, TimeUnit.MINUTES);
}
}
Receiver side
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import PubSubMessage;
public class SendEmailFromSendGrid implements BackgroundFunction<PubSubMessage> {
public SendEmailFromSendGrid() {
}
public void accept(PubSubMessage message, Context context) throws Exception {
System.out.println("invoked accept");
String name = "World";
if (message != null && message.getData() != null) {
name = new String(Base64.getDecoder().decode(message.getData().getBytes(StandardCharsets.UTF_8)),
StandardCharsets.UTF_8);
}
System.out.println(String.format("Hello %s!", name));
return;
}
}
PubSubMessage Definition
import java.util.Map;
public class PubSubMessage {
public static final String newUserLanguage = "userLanguage";
private String data;
private Map<String, String> attributes;
private String messageId;
private String publishTime;
public String getData() {
return data;
}
public void setData(String data) {
this.data = data;
}
public Map<String, String> getAttributes() {
return attributes;
}
public void setAttributes(Map<String, String> attributes) {
this.attributes = attributes;
}
public String getMessageId() {
return messageId;
}
public void setMessageId(String messageId) {
this.messageId = messageId;
}
public String getPublishTime() {
return publishTime;
}
public void setPublishTime(String publishTime) {
this.publishTime = publishTime;
}
}
THANK YOU
This answer was provided by #user1241724 in the comment section:
Redid the whole exercise and it is working now. Only difference is
added default constructor in PubSubMessage.
I was trying to mock a private method's output which is being called inside another private method, I have no choice but to test the later private method, so I have added sample test code which I can represent here,
This Sample Class
package com.testableClass;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class TestableClass {
private int initialMockMethod(Object obj)
{
System.out.println(" ++++ Came Here ++++ ");
String str = getRestString("");
System.out.println("str ="+str);
return str.length();
}
private String getRestString(String abc)
{
String output="";
try {
URL url = new URL("https://gorest.co.in/public-api/users");//your url i.e fetch data from .
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP Error code : "
+ conn.getResponseCode());
}
InputStreamReader in = new InputStreamReader(conn.getInputStream());
BufferedReader br = new BufferedReader(in);
while ((output = br.readLine()) != null) {
System.out.println(output);
}
conn.disconnect();
return output;
} catch (Exception e) {
System.out.println("Exception in NetClientGet:- " + e);
}
return abc;
}
}
Now This PowerMock Class
package com.testableClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mockito;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import org.powermock.reflect.Whitebox;
import junit.framework.Assert;
#RunWith(PowerMockRunner.class)
#PrepareForTest(fullyQualifiedNames = "com.testableClass.TestableClass")
public class PowerMockTest {
#Test
public void testPrivateMethod() throws Exception
{
String message = "Hello PowerMockito";
String expectation = "Expectation";
TestableClass mock = PowerMockito.spy(new TestableClass());
// PowerMockito.doReturn(expectation).when(mock, "getRestString", message);
PowerMockito.when(mock, "getRestString", message).thenReturn(expectation);
int count = Whitebox.invokeMethod(mock, "initialMockMethod", new Object());
System.out.println(" +++ Count : "+count+" ++++ ");
Assert.assertTrue(true);
}
}
my issue is when I am running my test case then
PowerMockito.when(mock, "getRestString", message).thenReturn(expectation);
executes original method and returns original output while my requirement is that,
when my test case is actually calling private method initialMockMethod it should not call getRestString instead of that it should return my mocked expected output which is "Expectation"
Instead of using reflection and PowerMock, I'd say that do not try to mock a private method. It is a hidden detail of the class.
What I suggest is that, if your method makes an HTTP request, then let it do that. But you can use a mock server to mock the response. But to do that, you need to make your endpoint external, so that it can be injected.
I changed your class a little bit; still the same purpose tho.
public class TestableClass {
private final String resourceUrl;
public TestableClass(String resourceUrl) {
this.resourceUrl = resourceUrl;
}
public int publicMethod() {
return initialMockMethod(null);
}
private int initialMockMethod(Object obj) {
var str = getRestString("");
return str.length();
}
private String getRestString(String abc) {
try {
var url = new URL(resourceUrl);
var conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP Error code : "
+ conn.getResponseCode());
}
var in = new InputStreamReader(conn.getInputStream());
var br = new BufferedReader(in);
var result = br.lines().collect(Collectors.joining("\n"));
conn.disconnect();
return result;
} catch (Exception e) {
System.out.println("Exception in NetClientGet:- " + e);
}
return abc;
}
}
and here's the test
public class TestingTestableClass {
#Test
#SneakyThrows
public void test() {
final var server = new MockWebServer();
server.start(9999);
var instance = new TestableClass("http://127.0.0.1:9999");
server.enqueue(
new MockResponse()
.setResponseCode(200)
.setBody("this is the response")
);
final int result = instance.publicMethod();
Assertions.assertEquals(
"this is the response".length(),
result
);
final var record = server.takeRequest();
final var method = record.getMethod();
Assertions.assertEquals("GET", method);
server.shutdown();
}
}
Check out MockWebServer here
I am using easymock framework to write UT's. I have one class which uses java.util.Properties in public function. Below is sample code:
public URI generateSASURI() throws CloudStorageException, DataProcessingException {
final String token = "token";
try {
return new URI(URIUtils.getAssetURI(azureStorageProperties.getProperty(StorageConstant.STORAGE_ACCOUNT_NAME)
, cloudBlob).toString() + "?" + token);
} catch (URISyntaxException e) {
throw new DataProcessingException("Error while creating SAS URL " + e.getMessage(), e);
}
}
where azureStorageProperties is instance of java.util.Properties, which is created as bean and injected in class.
Now while writing unit test case , i am trying to mock Properties azureStorageProperties, i am getting below error:
Unable to evaluate the expression Method threw 'java.lang.IllegalStateException' exception.
Below is ss:
Below is class to be tested:
package com.company.ops.azure.storage;
import com.company.ops.azure.storage.constants.StorageConstant;
import com.company.ops.azure.storage.dto.CloudBlob;
import com.company.ops.cloudopsfacade.dto.SignedUrlDTO;
import com.company.ops.cloudopsfacade.exception.storage.CloudStorageException;
import com.company.ops.cloudopsfacade.exception.storage.DataProcessingException;
import com.company.ops.azure.storage.util.URIUtils;
import com.company.ops.cloudopsfacade.storage.ICloudStorageClient;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import com.microsoft.azure.storage.blob.SharedAccessBlobHeaders;
import com.microsoft.azure.storage.blob.SharedAccessBlobPermissions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.inject.Inject;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.EnumSet;
import java.util.Optional;
import java.util.Properties;
import java.util.Set;
public class StorageClient implements ICloudStorageClient {
private static final Logger LOG = LoggerFactory.getLogger(StorageClient.class);
#Inject
private CloudBlobClient cloudBlobClient;
#Inject
Properties azureStorageProperties;
#Inject
private SASTokenGenerator sasTokenGenerator;
private long sasUrlDuration;
private Optional<String> sasUrlDurationKey;
public StorageClient() {
sasUrlDurationKey = Optional.ofNullable(System.getenv(StorageConstant.SAS_URL_DURATION_KEY));
if (sasUrlDurationKey.isPresent()) {
try {
sasUrlDuration = Integer.parseInt(sasUrlDurationKey.get());
} catch(NumberFormatException ex) {
LOG.debug("sasURLDurationKey invalid" + ex.getMessage());
sasUrlDuration = StorageConstant.DEFAULT_SAS_URL_DURATION;
}
} else {
sasUrlDuration = StorageConstant.DEFAULT_SAS_URL_DURATION;
}
}
//NOTE: This constructor is just created for test case. As #Mock of final class is NOT supported in easymock
public StorageClient(long sasUrlDuration, CloudBlobClient cloudBlobClient) {
this.sasUrlDuration = sasUrlDuration;
this.cloudBlobClient = cloudBlobClient;
}
/**
*
* #param containerName
* #param blob
* #param expiryTime
* #return {#link URI}
* #throws CloudStorageException
* #throws DataProcessingException
*/
private URI generateSASURI(String containerName,
String blob, Long expiryTime) throws CloudStorageException, DataProcessingException {
CloudBlob cloudBlob = new CloudBlob(Optional.ofNullable(containerName)
.orElseThrow(() -> new CloudStorageException("container name is null")),
Optional.ofNullable(blob).orElseThrow(() -> new CloudStorageException("blob name is null")));
//#TODO: Need to check permissions: Currently create and write assigned
final Set<SharedAccessBlobPermissions> permissions = EnumSet.of(SharedAccessBlobPermissions.WRITE, SharedAccessBlobPermissions.CREATE);
final SharedAccessBlobHeaders contentHeaders = new SharedAccessBlobHeaders();
//in case if duration need to set manually via api
if (Optional.ofNullable(expiryTime).isPresent()) {
sasUrlDuration = expiryTime;
}
sasTokenGenerator.sasTokenGeneratorInitializer(
cloudBlobClient,
cloudBlob,
permissions,
sasUrlDuration,
contentHeaders);
final String token = sasTokenGenerator.getToken();
try {
return new URI(URIUtils.getAssetURI(azureStorageProperties.getProperty(StorageConstant.STORAGE_ACCOUNT_NAME)
, cloudBlob).toString() + "?" + token);
} catch (URISyntaxException e) {
throw new DataProcessingException("Error while creating SAS URL " + e.getMessage(), e);
}
}
/**
*
* #param containerName
* #param blob
* #param expiryTime
* #return {#link URI}
* #throws CloudStorageException
* #throws DataProcessingException
*/
#Override
public SignedUrlDTO generateSignedUrl(String containerName, String blob, Long expiryTime)
throws CloudStorageException, DataProcessingException, MalformedURLException {
try {
URI uri = generateSASURI(containerName, blob, expiryTime);
SignedUrlDTO signedUrlDTO = new SignedUrlDTO();
signedUrlDTO.setSignedURL(uri.toURL());
return signedUrlDTO;
} catch (DataProcessingException ex) {
LOG.error(ex.getMessage());
throw ex;
} catch (CloudStorageException ex) {
LOG.error(ex.getMessage());
throw ex;
} catch (MalformedURLException e) {
LOG.error("Unable to get URL");
throw e;
}
}
}
Below is complete code for UT
package com.company.ops.azure.storage;
import com.company.ops.azure.constants.AzureConstants;
import com.company.ops.azure.storage.constants.StorageConstant;
import com.company.ops.azure.storage.util.URIUtils;
import com.company.ops.cloudopsfacade.dto.SignedUrlDTO;
import com.company.ops.cloudopsfacade.exception.storage.CloudStorageException;
import com.company.ops.cloudopsfacade.exception.storage.DataProcessingException;
import com.microsoft.azure.storage.StorageCredentials;
import com.microsoft.azure.storage.StorageCredentialsAccountAndKey;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import org.easymock.*;
import static org.easymock.EasyMock.*;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import javax.inject.Inject;
import java.net.MalformedURLException;
import java.net.URISyntaxException;
import java.util.Properties;
#RunWith(EasyMockRunner.class)
public class StorageClientTest extends EasyMockSupport {
private CloudBlobClient cloudBlobClient;
#Mock
private SASTokenGenerator sasTokenGenerator;
#Mock
private Properties azureStorageProperties;
#Mock
private StorageClient storageClient; /*= /*new StorageClient(StorageConstant.DEFAULT_SAS_URL_DURATION,
cloudBlobClient, azureStorageProperties);*/
#Before
public void setup() throws URISyntaxException {
resetAll();
}
#After
public void tearDown() {
verifyAll();
}
#Test
public void testGenerateSASURI() throws MalformedURLException, CloudStorageException, DataProcessingException {
String containerName = "myprojectlocal";
String blob = "testfile";
Long expiryTime = 1000L;
/*Properties azureStorageProperties = EasyMock.createMockBuilder(Properties.class)
.addMockedMethod("getProperty", String.class).createMock();*/
//azureStorageProperties = new Properties();
// azureStorageProperties.setProperty(StorageConstant.STORAGE_ACCOUNT_NAME, "storage-account-name");
expect(azureStorageProperties.getProperty(StorageConstant.STORAGE_ACCOUNT_NAME)).andReturn("storage-account-name");
expect(sasTokenGenerator.getToken()).andReturn("token");
sasTokenGenerator.sasTokenGeneratorInitializer(anyObject(CloudBlobClient.class),
anyObject(),
anyObject(),
anyLong(),
anyObject());
expectLastCall();
//azureStorageProperties.getProperty(anyString()); //.andReturn("storage-account-name");
//expectLastCall();
//expect(azureStorageProperties.getProperty(StorageConstant.STORAGE_ACCOUNT_NAME)).andReturn("storage-account-name");
replayAll();
SignedUrlDTO signedUrlDTO = storageClient.generateSignedUrl(containerName, blob, expiryTime);
Assert.assertNotNull(signedUrlDTO);
}
private CloudBlobClient getCloudBlobClient() {
final StorageCredentials credentials = new StorageCredentialsAccountAndKey(TestConstants.STORAGE_ACCOUNT_NAME,
TestConstants.STORAGE_ACCOUNT_KEY);
try {
return new CloudBlobClient(URIUtils.getStorageAccountURI(TestConstants.STORAGE_ACCOUNT_NAME), credentials);
} catch (URISyntaxException e) {}
return null;
}
}
I am a straight up newbie. After 2 weeks of research, I've arrived at this point, still very lost and desperate.
My goal: Retrieve data from EnergyStar Portfolio Manager Restful web services / API and put that data into a SQL database or an excel worksheet.
Progress so far:
1) I found an example code that seems like it would fit what I need well, except for the authentication part. I am more successful at getting WSDL services to work than RESTful services. Particularly for EnergySTAR, requiring some kind of authentication, which I can't seem to get around to.
2) The retrieve GET protocol: http://portfoliomanager.energystar.gov/webservices/home/test/api/reporting/designMetrics/get
My source code:
package com;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import com.User;
#Path("/wstest/property/")
public class UserManagementModule
{
#GET
#Path("/{propertyId}/design/metrics?measurementSystem=METRIC")
#Produces("application/xml")
public Response getUserById(#PathParam("propertyId") Integer id)
{
User user = new User();
user.setId(id);
return Response.status(200).entity(user).build();
}
}
package com;
import java.io.Serializable;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
#XmlAccessorType(XmlAccessType.NONE)
#XmlRootElement(name = "propertyMetrics")
public class User implements Serializable {
private static final long serialVersionUID = 1L;
#XmlAttribute(name = "propertyId")
private int propertyId;
#XmlElement(name="metric")
private double metric;
#XmlAttribute(name = "designEnergyCost")
private double designEnergyCost;
#XmlAttribute(name = "designScore")
private double designScore;
#XmlAttribute(name = "designSiteTotal")
private double designSiteTotal;
#XmlAttribute(name = "designSiteIntensity")
private double designSiteIntensity;
#XmlAttribute(name = "designTargetEnergyCost")
private double designTargetEnergyCost;
#XmlAttribute(name = "designTargetTotalGHGEmissions")
private double designTargetTotalGHGEmissions;
#XmlAttribute(name = "designTargetSiteTotal")
private double designTargetSiteTotal;
#XmlAttribute(name = "designTargetSiteIntensity")
private double designTargetSiteIntensity;
#XmlAttribute(name = "designTargetSourceTotal")
private double designTargetSourceTotal;
#XmlAttribute(name = "designTargetSourceIntensity")
private double designTargetSourceIntensity;
#XmlAttribute(name = "medianEnergyCost")
private double medianEnergyCost;
#XmlAttribute(name = "medianTotalGHGEmissions")
private double medianTotalGHGEmissions;
#XmlAttribute(name = "medianScore")
private double medianScore;
public int getId() {
return propertyId;
}
public void setId(int propertyId) {
this.propertyId = propertyId;
}
public double getmendianScore() {
return medianScore;
}
//
// public void setFirstName(String firstName) {
// this.firstName = firstName;
// }
// public String getLastName() {
// return lastName;
// }
// public void setLastName(String lastName) {
// this.lastName = lastName;
// }
}
package tests;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;
import com.User;
public class RestClientXML {
public static void main(String[] args)
{
try
{
URL url = new URL(" https://portfoliomanager.energystar.gov/wstest/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/xml");
if (conn.getResponseCode() != 200)
{
throw new RuntimeException("Failed : HTTP error code : " + conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
String apiOutput = br.readLine();
System.out.println(apiOutput);
conn.disconnect();
JAXBContext jaxbContext = JAXBContext.newInstance(User.class);
Unmarshaller jaxbUnmarshaller = jaxbContext.createUnmarshaller();
User user = (User) jaxbUnmarshaller.unmarshal(new StringReader(apiOutput));
System.out.println(user.getId());
/* System.out.println(user.getFirstName());
System.out.println(user.getLastName());
*/
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (JAXBException e) {
e.printStackTrace();
}
}
}
I'm know there is some authentication that needs username and pw,so I also tried this piece instead of the RestClient class, but neither works.
public static void main(String[] args) {
CloseableHttpClient httpClient = null;
HttpPost httpPost = null;
CloseableHttpResponse response = null;
try {
httpClient = HttpClients.createDefault();
httpPost = new HttpPost("https://portfoliomanager.energystar.gov/wstest/");
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("content-type", "application/xml"));
StringEntity input = new StringEntity("{\"username\": \"yungv1\",\"password\": \"dummypassword\"}");
input.setContentType("application/xml");
httpPost.setEntity(input);
for (NameValuePair h : nvps)
{
httpPost.addHeader(h.getName(), h.getValue());
}
response = httpClient.execute(httpPost);
if (response.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(response.getEntity().getContent())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try{
response.close();
httpClient.close();
}catch(Exception ex) {
ex.printStackTrace();
}
}
I know it's very sloppy. Can you please point the direction on how I can achieve my goal?
I also tried to run this, and keep getting the 401 errors, which I imagined the authentication fails. I don't know how else to trouble shoot this part and whether there's more thing I need to look into.
conn.setRequestProperty("Authorization", "Basic " + Base64.encodeBase64String((getWebServicesUsername() + ":" + getWebServicesPassword()).getBytes()));
Where conn is HttpURLConnection.
Im doing restfull webservices with soap using websphere and RAD. After generating my javaclient classes when I run the test class I get the following error. Ive been searching the web but not finding the correct solution. PLEASE HELP!
ERROR:
Check method call
org.w3c.dom.DOMException: HIERARCHY_REQUEST_ERR: An attempt was made to insert a node where it is not permitted.
at org.apache.xerces.dom.CoreDocumentImpl.insertBefore(Unknown Source)
at org.apache.xerces.dom.NodeImpl.appendChild(Unknown Source)
at com.ibm.ws.webservices.engine.xmlsoap.SOAPPart.appendChild(SOAPPart.java:282)
at com.sun.xml.internal.bind.marshaller.SAX2DOMEx.startElement(SAX2DOMEx.java:177)
at com.sun.xml.internal.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:159)
at com.sun.xml.internal.ws.message.AbstractMessageImpl.readAsSOAPMessage(AbstractMessageImpl.java:194)
at com.sun.xml.internal.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:80)
at test.CustomSoapHandler.handleMessage(CustomSoapHandler.java:37)
at test.CustomSoapHandler.handleMessage(CustomSoapHandler.java:1)
at com.sun.xml.internal.ws.handler.HandlerProcessor.callHandleMessage(HandlerProcessor.java:293)
at com.sun.xml.internal.ws.handler.HandlerProcessor.callHandlersRequest(HandlerProcessor.java:134)
at com.sun.xml.internal.ws.handler.ClientSOAPHandlerTube.callHandlersOnRequest(ClientSOAPHandlerTube.java:139)
at com.sun.xml.internal.ws.handler.HandlerTube.processRequest(HandlerTube.java:117)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:599)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:558)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:543)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:440)
at com.sun.xml.internal.ws.client.Stub.process(Stub.java:223)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:136)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:110)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:90)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:119)
at $Proxy33.getHistory(Unknown Source)
at test.ServiceTest.historyTest(ServiceTest.java:64)
at test.ServiceTest.main(ServiceTest.java:100)
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:227)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:119)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:90)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:119)
at $Proxy33.getHistory(Unknown Source)
at test.ServiceTest.historyTest(ServiceTest.java:64)
at test.ServiceTest.main(ServiceTest.java:100)
Caused by: java.lang.ClassCastException: com.ibm.xml.xlxp2.jaxb.JAXBContextImpl incompatible with com.sun.xml.internal.bind.api.JAXBRIContext
at java.lang.ClassCastException.<init>(ClassCastException.java:58)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.<clinit>(SOAPFaultBuilder.java:545)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:205)
... 6 more
Test Class:
package test;
import be.ipc.css.ws.GcssWebServiceService;
import be.ipc.css.ws.IGcssWebService;
import be.ipc.css.ws.InvalidItemIdStructureFault_Exception;
import be.ipc.css.ws.ProductNotAllowedFault_Exception;
import be.ipc.css.ws.common.Product;
import be.ipc.css.ws.history.GetHistoryInput;
import be.ipc.css.ws.history_output.GetHistoryOutput;
import be.ipc.css.ws.history_output.HistoryItem;
import be.ipc.css.ws.history_output.Type;
import javax.xml.namespace.QName;
import javax.xml.ws.BindingProvider;
import javax.xml.ws.Service;
import javax.xml.ws.handler.Handler;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
/**
* <strong>Project: TODO: project name</strong><br/>
* <b>Url:</b> TODO: url<br/>
* <b>Date:</b> 15.05.14<br/>
* <b>Time:</b> 23:42 <br/>
* Copyright(C) 2014 IT Service Plus <br/>
* <b>Description:</b><br/>
* TODO: description
*/
public class ServiceTest {
private static Service webService;
private static IGcssWebService servicePort;
public void initService() throws MalformedURLException {
URL url = new URL("http://cs-demo.ipc.be/CSS_UA2/services/gcssWebService/1.0");
QName qname = new QName("http://ws.css.ipc.be/", "GcssWebServiceService");
/*java.util.Properties props = System.getProperties();
props.setProperty("http.proxyHost", "proxy.test.com");
props.setProperty("http.proxyPort", "8080");*/
webService = GcssWebServiceService.create(url, qname);
servicePort = webService.getPort(IGcssWebService.class);
try {
CustomSoapHandler sh = new CustomSoapHandler("user_us", "*******");
List<Handler> new_handlerChain = new ArrayList<Handler>();
new_handlerChain.add(sh);
((BindingProvider)servicePort).getBinding().setHandlerChain(new_handlerChain);
} catch (Throwable e) {
e.printStackTrace();
}
//((BindingProvider)servicePort).getRequestContext().put(BindingProvider.USERNAME_PROPERTY, "user_us");
//((BindingProvider)servicePort).getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, "*******");
}
public void historyTest() throws ProductNotAllowedFault_Exception, InvalidItemIdStructureFault_Exception {
GetHistoryInput input = new GetHistoryInput();
input.setItemId("CC027607063NL");
input.setProduct(Product.EPG);
System.out.println("Check method call");
GetHistoryOutput output = servicePort.getHistory(input);
System.out.println("Check result");
//Assert.assertNotNull(output);
//Assert.assertNotNull(output.getHistory());
//Assert.assertNotNull(output.getHistory().getHistoryItem());
System.out.println("Received: " + output.getHistory().getHistoryItem().size() + "elements:");
System.out.println("-----------------------");
for(int i=0;i<output.getHistory().getHistoryItem().size();i++) {
HistoryItem it = output.getHistory().getHistoryItem().get(i);
System.out.println("#" + i + ": id=" + it.getId() + ", type=" + it.getType().value());
}
System.out.println("Check 3 history items");
//Assert.assertEquals(3, output.getHistory().getHistoryItem().size());
HistoryItem it1 = output.getHistory().getHistoryItem().get(0);
HistoryItem it2 = output.getHistory().getHistoryItem().get(1);
HistoryItem it3 = output.getHistory().getHistoryItem().get(2);
/*Assert.assertEquals(Type.QUMQ, it1.getType());
Assert.assertEquals(161L, it1.getId());
Assert.assertEquals(Type.SUM, it2.getType());
Assert.assertEquals(652L, it2.getId());
Assert.assertEquals(Type.L_1_Q, it3.getType());
Assert.assertEquals(13742L, it3.getId());*/
}
public static void main(String[] args) {
ServiceTest test = new ServiceTest();
try {
test.initService();
test.historyTest();
} catch(Exception e ){
e.printStackTrace();
}
}
}
Soap Handler:
package test;
import javax.xml.namespace.QName;
import javax.xml.soap.*;
import javax.xml.ws.handler.MessageContext;
import javax.xml.ws.handler.soap.SOAPHandler;
import javax.xml.ws.handler.soap.SOAPMessageContext;
import java.util.Set;
/**
* <strong>Project: TODO: project name</strong><br/>
* <b>Url:</b> TODO: url<br/>
* <b>Date:</b> 16.05.14<br/>
* <b>Time:</b> 0:48 <br/>
* Copyright(C) 2014 IT Service Plus <br/>
* <b>Description:</b><br/>
* TODO: description
*/
public class CustomSoapHandler implements SOAPHandler<SOAPMessageContext> {
private static final String AUTH_PREFIX = "wsse";
private static final String AUTH_NS =
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";
private String username;
private String password;
public CustomSoapHandler(String username, String password) {
this.username = username;
this.password = password;
}
public boolean handleMessage(SOAPMessageContext context) {
try {
SOAPEnvelope envelope =
context.getMessage().getSOAPPart().getEnvelope();
SOAPFactory soapFactory = SOAPFactory.newInstance();
SOAPElement wsSecHeaderElm =
soapFactory.createElement("Security", AUTH_PREFIX, AUTH_NS);
Name wsSecHdrMustUnderstandAttr =
soapFactory.createName("mustUnderstand", "S",
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd");
wsSecHeaderElm.addAttribute(wsSecHdrMustUnderstandAttr, "1");
SOAPElement userNameTokenElm =
soapFactory.createElement("UsernameToken", AUTH_PREFIX,
AUTH_NS);
Name userNameTokenIdName =
soapFactory.createName("id", "wsu", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
userNameTokenElm.addAttribute(userNameTokenIdName,
"UsernameToken-ORbTEPzNsEMDfzrI9sscVA22");
SOAPElement userNameElm =
soapFactory.createElement("Username", AUTH_PREFIX, AUTH_NS);
userNameElm.addTextNode(username);
SOAPElement passwdElm =
soapFactory.createElement("Password", AUTH_PREFIX, AUTH_NS);
Name passwdTypeAttr = soapFactory.createName("Type");
passwdElm.addAttribute(passwdTypeAttr,
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText");
passwdElm.addTextNode(password);
userNameTokenElm.addChildElement(userNameElm);
userNameTokenElm.addChildElement(passwdElm);
wsSecHeaderElm.addChildElement(userNameTokenElm);
if (envelope.getHeader() == null) {
SOAPHeader sh = envelope.addHeader();
sh.addChildElement(wsSecHeaderElm);
} else {
SOAPHeader sh = envelope.getHeader();
sh.addChildElement(wsSecHeaderElm);
}
} catch (Throwable e) {
e.printStackTrace();
}
return true;
}
#Override
public boolean handleFault(SOAPMessageContext context) {
return false; //To change body of implemented methods use File | Settings | File Templates.
}
#Override
public void close(MessageContext context) {
//To change body of implemented methods use File | Settings | File Templates.
}
#Override
public Set<QName> getHeaders() {
return null;
}
}
The main reason for your error is incompatible JAXB implementation classes:
aused by: java.lang.ClassCastException: com.ibm.xml.xlxp2.jaxb.JAXBContextImpl incompatible with com.sun.xml.internal.bind.api.JAXBRIContext
Probably the best way to fix this is packaging your own version of JAXB within your application(inside the lib folder) and than change the class loader from WebSphere to be Parent Last.
After that, restart and try it again. If still doesn't work you can try adding your JAXB implementation libraries directly to the Application Server classloader. You can do that creating a directory under $WEBSPHERE_HOME/AppServer/classes and placing your JAXB implementation classes there. Be aware that this approach adds the dropped jars to all WebSphere instances running using this binary codebase.
You can learn more about WebSphere classloaders.
Hope this helps.