I have to write a Map Reduce job in Java which use an external web service. my questions is whether it is allowed to invoke an HTTP request from the Map function of a Map Reduce job, i.e:
public class GeoLocator {
private static String genderCheck = "female";
public static class Map extends MapReduceBase implements Mapper {
/* CALL EXTERNAL WEB SERVICE HERE */
}
..
}
If so, how can I invoke the web service?
I think you are using the old API, since, I see this line in your class,
public static class Map extends MapReduceBase implements Mapper {}
So, with respect to that, first, you need to set the webservice URL in your JobConf object, later, from the JobConf instance retrieve the same url in the mapper.
Furthermore, make sure your webservice url is accessible from all the Hadoop nodes
For e.g-
In main() function, you can do something like,
JobConf job = (JobConf) getConf();
job.set("webservice.url", "http://your_address_whatsoever");
And in the Mapper class -
String url = null;
public void configure(JobConf job) {
url = (String) job.get("webservice.url");
}
But I suggest you use the new API to kind of make things easy -
in main() you can simply go ,
Configuration conf = getConf();
set("webservice.url", "http://your_address_whatsoever");
And the mapper class's map() function simply do
String url = context.getConfiguration().get("webservice.url");
Related
Currently we have an elaborate POJO object structure for handling a webservice request, called a 'processor'.
Remote and Local EJB's and PersistenceContext called during serving this request are initialized in the statless bean and handed to this 'processors' constructor which is re-created during each webservice request.
If I do not want to revert to JNDI lookups deep down in my 'processor' I keep on dragging around all these EJB's through my code.
Enter CDI. I would like to be able to inject these EJB's whenever I need them in this 'processor'.
However, I also noticed this means that the current 'processor' has to become a CDI bean itselve: so, using the #Inject in the Stateless Session Bean that implements the webservice.
When I do this the entiry lifecycle of the processor becomes bound to the bean and not to the request its serving.
Suddenly I have to take into consideration that I should not retain state (other than the injected objects) in the processor, since this state will be shared between multiple webservice invocations. As a programmer, this is not making my life more easy.
So: how should I go about doing this? I've read about the scoping but I'm not sure how / if this would be helping me.
Example, stateless bean:
#Stateless
#WebService
public class ExampleBean {
#Inject
Processor requestScopedInstance;
int beanDependentScopeCounter;
public String sayHello() {
System.out.println( "bean object id: " + this.toString() );
return requestScopedInstance.sayHello(beanDependentScopeCounter++);
}
}
interface:
public interface Processor {
String sayHello(int beanScopedCounter);
}
Implementation:
public class ProcessorImpl implements Processor {
private int requestScopedCounter = 0;
#Override
public String sayHello(int beanScopedCounter) {
return "test, requestScoped: " + requestScopedCounter++ + ", beansScoped: " + beanScopedCounter;
}
}
When I do this the entiry lifecycle of the processor becomes bound to the bean and not to the request its serving that is not correct. That is only the case if you don't use #ApplicationScoped, #SessionScoped, #RequestScoped.
So:
Annotate your processor with #RequestScoped.
You don't need to hand over the EJBs, you can just inject them, where needed.
Use #PostConstruct annotated methods for constructor-code which uses injected objects.
stateless POJOs can be annotated #ApplicationScoped, not stateless POJOs can stay dependent-scoped which is default.
That is made possible because proxies are injected, not actual beans. Using these proxies CDI makes sure that the correct scope is used for your particular calls.
Trying to write some proper AEM integration tests using the aem-mocks framework. The goal is to try and test a servlet by calling its path,
E.g. an AEM servlet
#SlingServlet(
paths = {"/bin/utils/emailSignUp"},
methods = {"POST"},
selectors = {"form"}
)
public class EmailSignUpFormServlet extends SlingAllMethodsServlet {
#Reference
SubmissionAgent submissionAgent;
#Reference
XSSFilter xssFilter;
public EmailSignUpFormServlet(){
}
public EmailSignUpFormServlet(SubmissionAgent submissionAgent, XSSFilter xssFilter) {
this.submissionAgent = submissionAgent;
this.xssFilter = xssFilter;
}
#Override
public void doPost(SlingHttpServletRequest request, SlingHttpServletResponse response) throws IOException {
String email = request.getParameter("email");
submissionAgent.saveForm(xssFilter.filter(email));
}
}
Here is the corresponding test to try and do the integration testing. Notice how I've called the servlet's 'doPost' method, instead of 'POST'ing via some API.
public class EmailSignUpFormServletTest {
#Rule
public final AemContext context = new AemContext();
#Mock
SubmissionAgent submissionAgent;
#Mock
XSSFilter xssFilter;
private EmailSignUpFormServlet emailSignUpFormServlet;
#Before
public void setup(){
MockitoAnnotations.initMocks(this);
Map<String,String> report = new HashMap<>();
report.put("statusCode","302");
when(submissionAgent.saveForm(any(String.class)).thenReturn(report);
}
#Test
public void emailSignUpFormDoesNotRequireRecaptchaChallenge() throws IOException {
// Setup test email value
context.request().setQueryString("email=test.only#mail.com");
//===================================================================
/*
* WHAT I END UP DOING:
*/
// instantiate a new class of the servlet
emailSignUpFormServlet = new EmailSignUpFormServlet(submissionAgent, xssFilter);
// call the post method (Simulate the POST call)
emailSignUpFormServlet.doPost(context.request(),context.response());
/*
* WHAT I WOULD LIKE TO DO:
*/
// send request using some API that allows me to do post to the framework
// Example:
// context.request().POST("/bin/utils/emailSignUp") <--- doesn't exist!
//===================================================================
// assert response is internally redirected, hence expected status is a 302
assertEquals(302,context.response().getStatus());
}
}
I've done a lot of research on how this could be done (here) and (here), and these links show a lot about how you can set various parameters for context.request() object. However, they just don't show how to finally execute the 'post' call.
What you are trying to do is mix a UT with IT so this won't be easy at least with the aem-mocks framework. Let me explain why.
Assuming that you are able to call your required code
/*
* WHAT I WOULD LIKE TO DO:
*/
// send request using some API that allows me to do post to the framework
// Example:
// context.request().POST("/bin/utils/emailSignUp") <--- doesn't exist!
//===================================================================
Your test will end up executing all the logic in SlingAllMethodsServlet class and its parent classes. I am assuming that this is not what you want to test as these classes are not part of your logic and they already have other UT/IT (under respective Apache projects) to cater for testing requirements.
Also, looking at your code, bulk of your core logic resides in following snipper
String email = request.getParameter("email");
submissionAgent.saveForm(xssFilter.filter(email));
Your UT criteria is already met by the following line of your code:
emailSignUpFormServlet.doPost(context.request(),context.response());
as it covers most of that logic.
Now, if you are looking for proper IT for posting the parameters and parsing them all the way down to doPost method then aem-mocks is not the framework for that because it does not provide it in a simple way.
You can, in theory, mock all the layers from resource resolver, resource provider and sling servlet executors to pass the parameters all the way to your core logic. This can work but it won't benefit your cause because:
Most of the code is already tested via other UT
Too many internal mocking dependencies might make the tests flaky or version dependant.
If you really want to do pure IT, then it will be easier to host the servlet in an instance and access it via HttpClient. This will ensure that all the layers are hit. A lot of tests are done this way but it feels a bit heavy handed for the functionality you want to test and there are better ways of doing it.
Also the reason why context.request().POST doesn't exist is because context.request() for is a mocked state for the sake of testing. You want to actually bind and mock Http.Post operations which needs some way to resolve to your servlet and that is not supported by the framework.
Hope this helps.
I have inherited some code that isn't tested and which loads a resource using a method like :
SomeClass.class.getClassLoader().getResource("somefile");
I've written the following test but there are 0 interactions with the Mock class loader I've created. Can anyone comment on whether this type of test is possible.
public enum SomeClass {
INSTANCE;
public boolean someMethod() {
URL pathToLicense = SomeClass.class.getClassLoader().getResource("somefile");
return false;
}
}
#Test
public void testLicenseWorkflow(){
ClassLoader cl = PowerMockito.mock(ClassLoader.class);
File f = new File("someFile");
assertTrue(f.exists());
logger.info(f.getCanonicalPath() );
when(cl.getResource("somefile")).thenReturn(f.toURL());
verify(cl).getResource("somefile");
assertTrue(SomeClass.INSTANCE.someMethod());
}
Update - Adding a resources via Classloader
I've also tried the following but the someMethod this doens't seem to work either
new URLClassLoader(((URLClassLoader) SomeClass.INSTANCE.getClass().getClassLoader()).getURLs()) {
#Override
public void addURL(URL url) {
super.addURL(url);
logger.info("Calling add URL");
}
}.addURL(f.toURI().toURL());
You are not passing cl to anything. You prepare a mock for a classloader but then proceed to load the resource with another classloader, the one that loaded SomeClass. That is why you have 0 interactions in your mock.
And about your first question, it is possible if somehow you pass your mocked classloader to the method that actually loads the resource. For example, something like
public boolean someMethod(Classloader loader) {
URL pathToLicense = loader.getResource("somefile");
return false;
}
But I have to say that IMO, this test is not very useful, you should be mocking your own components, not java classes. If your goal mocking the classloader is to inject a different file when testing, a better approach is to change your code to receive a Stream and inject a stream connected to the file in production and in testing inject a stream connected to an element in memory.
In other words, resources make for bad testing when they need to be changed at test time
I'm developing an application to perform a series of tests on various web services. These web services consume and produce json, and for each of them we have a class to model the json request and response. For example:
If the json request for serviceX is something like this:
{
"name":"Alex",
"id":"123"
}
We have a class serviceXrequest like this:
public class serviceXrequest {
String name;
String id;
//Constructor, getters/setters, etc
...
}
With an object of that class as the starting point, we can perform a series of test on the web service. The idea is to make those test as generic as possible so they can be used with any web service by just writing a class that models it's request and a class to model the response.
For that reason, all of the test methods developed so far work with plain java objects. This is an example of what I want to have:
public class WebServiceTest {
String serviceURL;
String requestJson;
String requestClass;
String responseClass;
public WebServiceTest() {}
#Test
public static void Test1() { ... }
#Test
public static void Test2() { ... }
....
#Test
public static void TestN() { ... }
}
And then, from another class, invoke those tests with doing something like this:
public class LoginTest { //To test the login web service, for example
public static void main(String[] args) {
WebServiceTest loginTest = New WebServiceTest();
loginTest.setServiceURL("172.0.0.1/services/login");
loginTest.setRequestJson("{"user":"ale","pass":"1234"}");
...
loginTest.runTests();
}
}
I know it's not that simple, but any ideas on how to get there?
Thanks in advance!!
You might also look into REST-assured
One of the best tools for testing your webservices is SOAP UI, but this is more for functional testing
As well I integrated very well FitNesse tests
JMeter goes hand in hand with LoadUI ..kind of same things in terms of stress and load tests for webservices.
Junit...i never used directly applied to the webservice itself.
Most of the times I had a Spring service called by the implemetation of the WebService interface (Port) and I unit tested that one.
You should consider using http-matchers (https://github.com/valid4j/http-matchers) which let's you write JUnit-tests, using regular hamcrest-matchers (bundled with JUnit) to test your web-service via standard JAX-RS interface.
Im reading "The art of unit testing" atm and im having some issues with using properties to pass in an interface. The book states the following: "If you want parameters to be optional, use property getters/setters, which is a better way of defining optional parameters than adding different constructors to the class for each dependency."
The code for the property example is as follows:
public class LogAnalyzer
{
private IExtensionManager manager;
public LogAnalyzer ()
{
manager = new FileExtensionManager();
}
public IExtensionManager ExtensionManager
{
get { return manager; }
set { manager = value; }
}
public bool IsValidLogFileName(string fileName)
{
return manager.IsValid(fileName);
}
}
[Test]
Public void
IsValidFileName_NameShorterThan6CharsButSupportedExtension_ReturnsFalse()
{
//set up the stub to use, make sure it returns true
...
//create analyzer and inject stub
LogAnalyzer log = new LogAnalyzer ();
log.ExtensionManager=someFakeManagerCreatedEarlier;
//Assert logic assuming extension is supported
...
}
When/how would i use this feature?? The only scenario i can think of (This is probably wrong!) is if i had two methods in one class,
Method1() retrieves the database connection string from the config file and contains some form of check on the retrieved string.
Method2() then connect to the database and returns some data. The check here could be that that returned data is not null?
In this case, to test Method1() i could declare a stub that implements the IExtensionManager Interface, where the stub has a string which should pass any error checks i have in method1().
For Method2(), i declare a stub which implements the interface, and declare a datatable which contains some data, in the stub class. id then use the properties to assign this to the private manager variable and then call Method2?
The above may be complete BS, so if it is, id appreciate it if someone would let me know and ill remove it.
Thanks
Property injection used to change object's behavior after it was created.
BTW your code is tight coupled to FileExtensionManager, which is concrete implementation of IExtensionManager. How you are going to test LogAnalyzer with default manager? Use constructor injection to provide dependencies to your objects - this will make them testable:
public LogAnalyzer (IExtensionManager manager)
{
this.manager = manager();
}