Dataflow job is failing with below exception when I pass parameters staging,temp & output GCS bucket locations.
Java code:
final String[] used = Arrays.copyOf(args, args.length + 1);
used[used.length - 1] = "--project=OVERWRITTEN"; final T options =
PipelineOptionsFactory.fromArgs(used).withValidation().as(clazz);
options.setProject(PROJECT_ID);
options.setStagingLocation("gs://abc/staging/");
options.setTempLocation("gs://abc/temp");
options.setRunner(DataflowRunner.class);
options.setGcpTempLocation("gs://abc");
The error:
INFO: Staging pipeline description to gs://ups-heat-dev- tmp/mniazstaging_ingest_validation/staging/
May 10, 2018 11:56:35 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <42088 bytes, hash E7urYrjAOjwy6_5H-UoUxA> to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/staging/pipeline-E7urYrjAOjwy6_5H-UoUxA.pb
Dataflow SDK version: 2.4.0
May 10, 2018 11:56:38 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Printed job specification to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/templates/DataValidationPipeline
May 10, 2018 11:56:40 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Template successfully created.
Exception in thread "main" java.lang.NullPointerException
at org.apache.beam.runners.dataflow.DataflowPipelineJob.getJobWithRetries(DataflowPipelineJob.java:501)
at org.apache.beam.runners.dataflow.DataflowPipelineJob.getStateWithRetries(DataflowPipelineJob.java:477)
at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:312)
at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:248)
at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:202)
at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:195)
at com.example.DataValidationPipeline.main(DataValidationPipeline.java:66)
I was also facing the same issue, the error was throwing at p.run().waitForFinish();. Then I have tried following code
PipelineResult result = p.run();
System.out.println(result.getState().hasReplacementJob());
result.waitUntilFinish();
This was throwing the following exception
java.lang.UnsupportedOperationException: The result of template creation should not be used.
at org.apache.beam.runners.dataflow.util.DataflowTemplateJob.getState (DataflowTemplateJob.java:67)
Then to fix the issue I used the following code
PipelineResult result = pipeline.run();
try {
result.getState();
result.waitUntilFinish();
} catch (UnsupportedOperationException e) {
// do nothing
} catch (Exception e) {
e.printStackTrace();
}
I was running into the problem with java.lang.UnsupportedOperationException: The result of template creation should not be used. today aswell and I tried to fixed it by checking if the job was of type DataflowTemplateJob first:
val (sc, args) = ContextAndArgs(cmdlineArgs)
// ...
val result = sc.run()
if (!result.isInstanceOf[DataflowTemplateJob]) result.waitUntilFinish()
I think this should work for bare java jobs, but if you use Scio, then the result will be some anonymous type, so in the end I had to do the try catch version aswell.
try {
val result = sc.run().waitUntilFinish()
} catch {
case _: UnsupportedOperationException => // this happens during template creation
}
as displayed in the official Flex Template sample
There is a comment saying:
// For a Dataflow Flex Template, do NOT waitUntilFinish().
The same applies if you call any of those methods of the Runner if you pass the --templateRunner argument
if you change the pipeline to pipeline.run(); it is not going to fail.
The Issue is still flagged as opened by apache beam
https://github.com/apache/beam/issues/20106
Related
I am using Grails 4.0.3 and want to assert that an exception is thrown in a given situation. Below, my PidService.
def validatePidIssuingConclusionDocument(JSONObject pidIssuingConclusionDocument){
String message = "Document issuing number is mandatory"
if(!pidIssuingConclusionDocument.has("docIssuingNumber")){throw new Exception(message)}
}
Below, my test case:
void "wrong document"() throws Exception {
JSONObject documentWithoutIssuingNumber = new JSONObject()
documentWithoutIssuingNumber.accumulate("docIssuingNumber","123")
pidService.buildPidIssuingOrder(documentWithoutIssuingNumber)
// How can I assert that the exception is thrown and verify it message.
}
I tried to use try/catch in the test case without success. Could anyone help me? I need to assert that the exception is thrown and the message is Document issuing number is mandatory in the given case.
I need to assert that the exception is thrown and the message is
Document issuing number is mandatory in the given case.
I can't provide an executable example around the code you provided because some things are missing (like buildPidIssuingOrder). See the project at https://github.com/jeffbrown/dnunesexception which contains an example that shows a way to deal with the exception in a test though.
grails-app/services/dnunesexception/PidService.groovy
package dnunesexception
import org.grails.web.json.JSONObject
class PidService {
def buildPidIssuingOrder(JSONObject pidIssuingConclusionDocument) {
throw new Exception('Document issuing number is mandatory')
}
}
src/test/groovy/dnunesexception/PidServiceSpec.groovy
package dnunesexception
import grails.testing.services.ServiceUnitTest
import spock.lang.Specification
import org.grails.web.json.JSONObject
class PidServiceSpec extends Specification implements ServiceUnitTest<PidService>{
void "test something"() {
when:
JSONObject documentWithoutIssuingNumber = new JSONObject()
documentWithoutIssuingNumber.accumulate("docIssuingNumber","123")
service.buildPidIssuingOrder(documentWithoutIssuingNumber)
then:
def e = thrown(Exception)
and:
e.message == 'Document issuing number is mandatory'
}
}
Though it was working previously, for some reason my code has now stopped working. Though it fetches the required json data, the build doesn't render. The error on my app page which is supposed to display a page view was;
type String is not a subtype of 'Map<String,dynamic>'
But after some tweaks, now the error is;
invalid arguments(s)
I think I may have narrowed it down to the Future;
FutureBuilder<List<SpeakContent>> futureStage() {
return new FutureBuilder<List<SpeakContent>>(
future: downloadJSON(),
builder: (context, snapshot) {
if (snapshot.hasData) {
print("Snapshot has data.");
List<SpeakContent> speakcrafts = snapshot.data;
return new CustomPageView(speakcrafts);
} else if (snapshot.hasError) {
return Text('${snapshot.error}');
}
return new CircularProgressIndicator();
},
);
}
Future<List<SpeakContent>> downloadJSON() async {
final jsonEndpoint =
"http://example.com/getContent.php?";
final response = await get(jsonEndpoint);
if (response.statusCode == 200) {
List speakcrafts = json.decode(response.body);
debugPrint(speakcrafts.toString());
return speakcrafts
.map((speakcraft) => new SpeakContent.fromJson(speakcraft))
.toList();
} else
throw Exception('We were not able to successfully download the json data.');
}
Although it doesn't throw an error, I've noticed that it doesn't print my test statement after the "if (snapshot.hasData)" line.
Shouldn't I see "Snapshot has data." appear in my Android Studio console?
Based on what you provided, this
type String is not a subtype of 'Map<String,dynamic>'
must be this:
return Text('${snapshot.error}');
which means that your downloadJSON() threw an exception. In that case, print('Snapshot has data.'); never executes and the next case that I quoted above is executed.
Please put a breakpoint in the body of downloadJSON(), run it line by line and see what's thrown where.
ALSO, you are making an irrelevant but big mistake here. Do not call downloadJSON() like this. This function is executed at every re-render, which can be many times. You are initiating a JSON download at every redraw of your widget. This may stack up your backend bills... I explain it in this talk in more detail: https://youtu.be/vPHxckiSmKY?t=4771
After performing the troubleshooting tips as suggested by Gazihan Alankus and scrimau, I've found the culprit which was a single null entry in the MYSQL Database. Scary... gotta find out how to prevent that problem in the future.
I have written the code using camel-sql which is working fine. Now I have to write test cases for the same. I have used in-memory database H2. I have initialized the database and assigned the datasource to sqlComponent.
// Setup code
#Override
protected JndiRegistry createRegistry() throws Exception {
JndiRegistry jndi = super.createRegistry();
// this is the database we create with some initial data for our unit test
database = new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2).addScript("createTableAndInsert.sql").build();
jndi.bind("myDataSource", database);
return jndi;
}
// Testcase code
#SuppressWarnings("unchecked")
#Test
public void testRoute() throws Exception {
Exchange receivedExchange = template.send("direct:myRoute", ExchangePattern.InOut ,exchange -> {
exchange.getIn().setHeader("ID", new Integer(1));
});
camelContext.start();
MyClass updatedEntity = (MyClass)jdbcTemplate.queryForObject("select * from MY_TABLE where id=?", new Long[] { 1l } ,
new RouteTest.CustomerRowMapper() );
// Here I can get the updatedEntity from jdbcTemplate
assertNotNull(receivedExchange);
assertNotNull(updatedEntity);
}
// Main code
from("direct:myRoute")
.routeId("pollDbRoute")
.transacted()
.to("sql:select * from MY_TABLE msg where msg.id = :#"+ID+"?dataSource=#myDataSource&outputType=SelectOne&outputClass=com.entity.MyClass")
.log(LoggingLevel.INFO,"Polled message from DB");
The problem is, as soon as the test case starts, it is saying
No bean could be found in the registry for: myDataSource of type: javax.sql.DataSource
I looked into camel-SQL component test cases and doing the same thing but the code is not able to find dataSource. Please help. Thanks in advance.
After spending a lot of time on this issue, I identified that H2 database was using JDBCUtils to fetch records and It was throwing ClassNotFoundException. I was getting it nowhere in Camel exception hierarchy because this exception was being suppressed and all I was getting a generic exception message. Here is the exception:
ClassNotFoundException: com.vividsolutions.jts.geom.Geometry
After searching for the issue I found out that It requires one more dependency. So I added it and it resolved the issue.
Issue URL: https://github.com/elastic/elasticsearch/issues/9891
Dependency: https://mvnrepository.com/artifact/com.vividsolutions/jts-core/1.14.0
I'm trying to unit test a Workflow Service by using Microsoft.Activities.UnitTesting
The goal is to mock the service's extensions in order to ensure that all steps are executed.
The mock objects don't seem to get called even though the extensions are registered in the Host. As expected, if the extensions are not registered an exception is thrown.
WorkflowServiceTestHost host = null;
try
{
Mock<ISubscriber> publisher = new Mock<ISubscriber>();
Mock<IWebWorker> webWorker = new Mock<IWebWorker>();
var voucher = new Voucher();
using (host = new WorkflowServiceTestHost(workflowServiceFile, serviceAddress))
{
host.WorkflowExtensions.Add<ISubscriber>(() => publisher.Object);
host.WorkflowExtensions.Add<IWebWorker>(() => webWorker.Object);
host.Open();
using (var factory = new ChannelFactory<IServiceInterface>(clientBinding, serviceAddress))
{
var proxy = factory.CreateChannel() as IServiceInterface;
proxy.Process(voucher);
}
}
**//These validations fail...**
publisher.Verify(m => m.Push(It.IsAny<Voucher>()), Times.Once(), "ISubscriber.Push was not called.");
webWorker.Verify(m => m.Done(It.IsAny<Voucher>()), Times.Once(), "IWebWorker.Done was not called.");
// The host must be closed before asserting tracking
// Explicitly call host.Close or exit the using block to do this.
}
finally
{
if (host != null)
{
host.Tracking.Trace(TrackingOptions.All);
}
}
The workflow runs as expected in IIS.
Thanks!
Edit: This error is being written in the Workflow Host output:
WorkflowInstance "Sequential Service" Unhandled Exception Source "Receive Process Message"
Exception <System.NotSupportedException: Expression Activity type 'CSharpReference`1' requires compilation in order to run.
Please ensure that the workflow has been compiled.
at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)
at Microsoft.CSharp.Activities.CSharpReference`1.Execute(CodeActivityContext context)
at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)
at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)
at System.Activities.OutArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance targetActivityInstance, ActivityExecutor executor)
at System.Activities.RuntimeArgument.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance targetActivityInstance, ActivityExecutor executor, Object argumentValueOverride, Location resultLocation, Boolean skipFastPath)
at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)
at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)>
I've just realized WorkflowServiceTestHost is a Microsoft.Activities.UnitTesting class and not yours.
So, let's see if this is possible. As I saw on its source code you can pass to the constructor the WorkflowService's object itself instead of the XAMLX file. Something like this:
// Load WorkflowService from .xamlx
// Actually this is the method WorkflowserviceTestHost uses when you pass a
// .xamlx so we're taking a step back to be able to compile the body
var wfService = XamlServices.Load("c:\\workflowservice.xamlx") as WorkflowService;
// Compile workflow body
CompileExpressions(wfService.Body);
// Now you can use WorkflowServiceTestHost
using (host = new WorkflowServiceTestHost(wfService, serviceAddress))
{
// ... do your thing
}
CompileExpressions is taken from the link that I gave you earlier.
That being said, it seems odd consider testing a WCF service as unit-testing. Unit tests should be focused on small activities of your service, those are truly unit-testable. Integration tests (or functional tests) is where you test services with all its dependencies (IIS\WAS, network, DBs, etc).
I create a simple silverlight 4.0 application used to read the excel file data in the share point 2010 server. I try to use the "Excel Web Services" but I get an error here when calling the GetRangeA1 method:
An unhandled exception of type 'System.ServiceModel.Dispatcher.NetDispatcherFaultException' occurred in mscorlib.dll
Additional information: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://schemas.microsoft.com/office/excel/server/webservices:GetRangeA1Response. The InnerException message was 'Error in line 1 position 361. Element 'http://schemas.microsoft.com/office/excel/server/webservices:anyType' contains data from a type that maps to the name 'http://schemas.microsoft.com/office/excel/server/webservices:ArrayOfAnyType'. The deserializer has no knowledge of any type that maps to this name. Consider using a DataContractResolver or add the type corresponding to 'ArrayOfAnyType' to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding it to the list of known types passed to DataContractSerializer.'. Please see InnerException for more details.
the source code is like:
namespace SampleApplication
{
class Program
{
static void Main(string[] args)
{
ExcelServiceSoapClient xlservice = new ExcelServiceSoapClient();
xlservice.ClientCredentials.Windows.AllowedImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Impersonation;
Status[] outStatus;
string targetWorkbookPath = "http://phc/Shared%20Documents/sample.xlsx";
try
{
// Call open workbook, and point to the trusted location of the workbook to open.
string sessionId = xlservice.OpenWorkbook(targetWorkbookPath, "en-US", "en-US", out outStatus);
Console.WriteLine("sessionID : {0}", sessionId);
//1. works fines.
object res = xlservice.GetCellA1(sessionId, "CER by Feature", "B1", true, out outStatus);
//2. exception
xlservice.GetRangeA1(sessionId, "CER by Feature", "H19:H21", true, out outStatus);
// Close workbook. This also closes session.
xlservice.CloseWorkbook(sessionId);
}
catch (SoapException e)
{
Console.WriteLine("SOAP Exception Message: {0}", e.Message);
}
}
}
}
I am totally new to the silverlight and sharepoint developping, I search around but didn't get any luck, just found another post here, any one could help me?
This appears to be an oustanding issue, but two workarounds I found so far:
1) Requiring a change in App.config.
http://social.technet.microsoft.com/Forums/en-US/sharepoint2010programming/thread/ab2a08d5-2e91-4dc1-bd80-6fc29b5f14eb
2) Indicating to rebuild service reference with svcutil instead of using Add Service Reference:
http://social.msdn.microsoft.com/Forums/en-GB/sharepointexcel/thread/2fd36e6b-5fa7-47a4-9d79-b11493d18107