Firing up a cluster using whirr - amazon-web-services

I'm new to whirr and AWS so apologies in advance if I'm asking something silly.
I'm following the directions here to set up whirr and
bin/whirr launch-cluster --config hadoop.properties
fails with the following:
[~/src/cloudera/whirr-0.1.0+23]$ bin/whirr version rvm:ruby-1.8.7-p299
Apache Whirr 0.1.0+23
[~/src/cloudera/whirr-0.1.0+23]$ bin/whirr launch-cluster --config hadoop.properties rvm:ruby-1.8.7-p299
Launching myhadoopcluster cluster
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:
1) No implementation for java.lang.String annotated with #com.google.inject.name.Named(value=jclouds.credential) was bound.
while locating java.lang.String annotated with #com.google.inject.name.Named(value=jclouds.credential)
for parameter 2 at org.jclouds.aws.filters.FormSigner.<init>(FormSigner.java:91)
at org.jclouds.aws.config.AWSFormSigningRestClientModule.provideRequestSigner(AWSFormSigningRestClientModule.java:66)
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:410)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:166)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:118)
at com.google.inject.InjectorBuilder.build(InjectorBuilder.java:100)
at com.google.inject.Guice.createInjector(Guice.java:95)
at com.google.inject.Guice.createInjector(Guice.java:72)
at org.jclouds.rest.RestContextBuilder.buildInjector(RestContextBuilder.java:141)
at org.jclouds.compute.ComputeServiceContextBuilder.buildInjector(ComputeServiceContextBuilder.java:53)
at org.jclouds.aws.ec2.EC2ContextBuilder.buildInjector(EC2ContextBuilder.java:101)
at org.jclouds.compute.ComputeServiceContextBuilder.buildComputeServiceContext(ComputeServiceContextBuilder.java:66)
at org.jclouds.compute.ComputeServiceContextFactory.buildContextUnwrappingExceptions(ComputeServiceContextFactory.java:72)
at org.jclouds.compute.ComputeServiceContextFactory.createContext(ComputeServiceContextFactory.java:114)
at org.apache.whirr.service.ComputeServiceContextBuilder.build(ComputeServiceContextBuilder.java:41)
at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:84)
at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:61)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:65)
at org.apache.whirr.cli.Main.main(Main.java:91)
My hadoop.properties file has an AWS Access Key and Secret Access Key.
Any pointers on what I might have done wrong and what I need to do to fix this?
Thanks!

Okay so this appears to be a problem with the syntax in my hadoop.properties file. In the process of copying my keys across from the AWS management console, "Whirr.credential" got truncated to "Whirr.cred."
A classic face palm moment!
Anyway, leaving this up so that anyone googling for this error message knows to go triple check their hadoop.properties file!

Related

MassTransit.RabbitMqTransport.RabbitMqAddressException: 'The invalid scheme was specified: amqps'

Trying to configure a .Net Core service to talk to an AmazonMQ - RabbitMQ instance. This is the config I'm using:
https://masstransit-project.com/usage/transports/rabbitmq.html#amazonmq-rabbitmq
cfg.Host(new Uri("amqps://b-mdefg-ff37-4e33-855c-577a0c749659.mq.us-westeast-2.amazonaws.com:5432/"), h =>
{
h.Username("MyUsername");
h.Password("XXXXXXXXX");
});
Nuget packages are:
MassTransit.AspNetCore v5.5.6
MassTransit.Autofac v5.3.2
MassTransit.RabbitMQ v5.3.2
MassTransit.SerilogIntegration v5.1.5
The exception being thrown is:
MassTransit.RabbitMqTransport.RabbitMqAddressException: 'The invalid scheme was specified: amqps'
Upgrading to the latest MassTransit caused a lot of unrelated issues. Any clues as to a workaround?
OK, solution was fairly straightforward, but google gave me nothing so perhaps this will save someone some time. I simply upgraded MassTransit.RabbitMQ to v5.5.6 and it all works now.

Resource 'data.template_file.user_data' does not have attribute 'rendered'

Terraform Version : v0.11.8
Use case
Try to terminate the resources using terraform, got error while running output command.
Code:
output "frontend_rendered" {
value = "${data.template_file.user_data.rendered}"
}
Debug Output
module.test.output.test_rendered: Resource
'data.template_file.user_data' does not have attribute 'rendered' for
variable 'data.template_file.user_data.rendered'
Expected Behavior
Termination without any error.
Additional Context
This issue came after i upgraded terraform from v0.11.4 to v0.11.8 and i have also updated the aws provider to latest 1.33.0
Any help?
Thanks!
Finally i was able to find the solution.
After Terraform v0.11.4, we shouldn't evaluate unused outputs during a full destroy operation.

wso2am-2.0.1-SNAPSHOT modify subscription tier save error

I run wso2 apim 2.0.1 snapshot on windows, and when i modify subscription tier and save, it report below exception, and although the bill plan changed , but the API still display FREE label.
[2016-08-12 15:30:02,504] ERROR - EventProcessorAdminService Error while deleting the execution plan file
org.wso2.carbon.event.processor.core.exception.ExecutionPlanConfigurationException: Error while deleting the execution plan file
at org.wso2.carbon.event.processor.core.internal.util.EventProcessorConfigurationFilesystemInvoker.delete(EventProcessorConfigurationFilesystemInvoker.java:124)
......
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 2: /D:/emman/PROJECT/AA/apimgmt/wso2am-2.0.1-SNAPSHOT/repository/deployment/server/\executionplans
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at org.wso2.carbon.event.processor.core.internal.util.EventProcessorUtil.validateFilePath(EventProcessorUtil.java:387)
at org.wso2.carbon.event.processor.core.internal.util.EventProcessorConfigurationFilesystemInvoker.delete(EventProcessorConfigurationFilesystemInvoker.j
ava:109)
... 65 more
[2016-08-12 15:30:02,539] ERROR - ThrottlePolicyDeploymentManager Error while deploying policy to global policy server.Error while deleting the execution plan file
[2016-08-12 15:30:02,541] INFO - subscription-policy-edit:jag SubscriptionPolicy [policyName=Gold, description=Allows 5000 requests per minute, defaultQuotaPolicy=QuotaPolicy [type=requestCount, limit=RequestCountLimit [requestCount=5000,
toString()=Limit [timeUnit=min, unitTime=1]]]rateLimitCount=-1, tenantId=-1234,ratelimitTimeUnit=NA]
As per your logs, error happens due to invalid file path below.
/D:/emman/PROJECT/AA/apimgmt/wso2am-2.0.1-SNAPSHOT/repository/deployment/server/\executionplans
I had a look at code. It reads the first part of this path from <RepositoryLocation> tag of carbon.xml file. By default, it should look like this.
<RepositoryLocation>${carbon.home}/repository/deployment/server</RepositoryLocation>
Please verify if you have the same in carbon.xml. If you are getting this error with the same config, please change it to the absolute path like below and try again.
D:\emman\PROJECT\AA\apimgmt\wso2am-2.0.1-SNAPSHOT\repository\deployment\server
To make your path more linux-like I used this trick.
Share your carbon home folder. Change carbon.xml setting RepositoryLocation in //machinenaam/share.

Jetty - java.lang.IllegalStateException: zip file closed

I'm getting some error with some connection to our web server.
I saw that a bug causing this was solved in Jetty 7.6. Yes we get this error on our application running under Jetty 7.5.4 but we also get this with another apps running on a newer version 9.
Do you have any idea what this can be?
We are getting this error randomly:
java.lang.IllegalStateException: zip file closed
at java.util.zip.ZipFile.ensureOpen(ZipFile.java:632)
at java.util.zip.ZipFile.access$200(ZipFile.java:56)
at java.util.zip.ZipFile$1.hasMoreElements(ZipFile.java:485)
at java.util.jar.JarFile$1.hasMoreElements(JarFile.java:239)
at org.eclipse.jetty.util.resource.JarFileResource.exists(JarFileResource.java:163)
at org.eclipse.jetty.webapp.WebAppContext$Context.getResource(WebAppContext.java:1223)
at org.eclipse.jetty.servlet.DefaultServlet.getResource(DefaultServlet.java:366)
at org.eclipse.jetty.server.ResourceCache.lookup(ResourceCache.java:188)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:445)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:480)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:483)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:941)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:345)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:919)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:582)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:722)
There are 2 main causes for this.
A bad/corrupt JAR file in your classpath.
The JVM built-in URL caching getting in the way.
For a bad/corrupt JAR file, you'll have to isolate that on your own, figure out which one it is. Maybe by just unjaring all of them one by one till you find the problematic one.
As for the JVM URL caching, this seems to cause problems with dynamic classloaders like OSGi or hot-deploy scenarios the most.
For this scenario, you can tell jetty to set the URLConnection.setUseCaches(boolean) for each URLConnection attempt of its own.
To disable the JVM caches, add the following snippet of XML to your etc/jetty.xml
<Set class="org.eclipse.jetty.util.resource.Resource"
name="defaultUseCaches">false</Set>
You don't need to test all files to ensure the correctness of them. Just put a break point at java.util.zip.ZipFile.ensureOpen(ZipFile.java:632) and check the name field in ZipFile class: private final String name
I had the same problem after deploying jenkins war; all I had to do was to restart the server after the deploy.
Hope it helps.
This issue can also occur if you are doing any of your own reflection in that code path.
I was having the same issue because of the actual WAR being corrupt. Try rerunning mvn clean install and redeploy.
I had the same problem, after deleting all .jar files and build the path to all .jar file once again. Now it's working properly

Twig template manager error on AWS Cloud

I hope I find somebody who knows the Twig in details. I've developed several sites with Twig and I'm really satisfied. Now I'm trying to put the website into an AWS cloud. Generally it works fine, but sometimes after a new deploy I get the following error message:
Fatal error: Class 'Twig_TokenParser_If' not found in /var/www/html/lib/twig/lib /Twig/Extension/Core.php on line 99 Call Stack: 0.0001 633192 1. {main}() /var/www/html/index.php:0 0.1184 1516544 2. Twig_Environment->render() /var/www/html/index.php:13 0.1184 1516544 3. Twig_Environment->loadTemplate() /var/www/html/lib/twig/lib/Twig/Environment.php:283 0.1186 1518664 4. Twig_Environment->compileSource() /var/www/html/lib/twig/lib/Twig/Environment.php:314 0.1228 1692688 5. Twig_Environment->parse() /var/www/html/lib/twig/lib/Twig/Environment.php:523 0.1231 1713752 6. Twig_Parser->parse() /var/www/html/lib/twig/lib/Twig/Environment.php:473 0.1231 1715296 7. Twig_Environment->getTokenParsers() /var/www/html/lib/twig/lib/Twig/Parser.php:74 0.1233 1731256 8. Twig_Extension_Core->getTokenParsers() /var/www/html/lib/twig/lib/Twig/Environment.php:694
If I set up a new environment in the cloud and I deploy exactly the same application(all files are the same), it works. I don't want to set everything on a new release.
If somebody knows what could be the error, please help me.
Thanks