AWS SQS and ElasticBeanStalk environment set up - amazon-web-services

Following is my spring configuration for aws sqs.
<bean id="CredentialsProviderBean"
class="com.myapp.util.ClasspathPropertiesFileCredentialsProvider" />
<bean id="ConnectionFactoryBuilder"
class="com.amazon.sqs.javamessaging.SQSConnectionFactory$Builder">
<property name="regionName" value="us-east-1" />
<property name="numberOfMessagesToPrefetch" value="1" />
<property name="awsCredentialsProvider" ref="CredentialsProviderBean" />
</bean>
<bean id="ConnectionFactory" class="com.amazon.sqs.javamessaging.SQSConnectionFactory"
factory-bean="ConnectionFactoryBuilder" factory-method="build" />
<bean id="Connection" class="javax.jms.Connection" factory-bean="ConnectionFactory"
factory-method="createConnection" init-method="start" destroy-method="close" />
<bean id="QueueName" class="java.lang.String">
<constructor-arg value="myqueue" />
</bean>
<bean id="amazonMessageListener" class="com.myapp.daemon.AsyncMessageListener" />
<bean id="messageListener"
class="org.springframework.jms.listener.adapter.MessageListenerAdapter">
<property name="delegate" ref="amazonMessageListener" />
<property name="defaultListenerMethod" value="onMessage" />
<property name="messageConverter">
<null />
</property>
</bean>
<bean id="jmsContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="ConnectionFactory" />
<property name="destinationName" ref="QueueName" />
<property name="messageListener" ref="messageListener" />
</bean>
I am having trouble understanding how to set this up in the AWS.
ElasticBeanStalk provides 2 types of environments,
1. worker and 2. webserver.
Thought the worker type environment fits the bill, according to the documentation. AWS deploys a daemon in ec2 instances, and pulls off the message from SQS and envelopes that into a message body of the http post request. This post request can be used to post to a web server.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-environ
However the component I wrote does that. Pulls off the message from the queue, and does some background processing.
In this case which environment type should i opt?

If you have to use worker tier you do not need to use your component and you make sure the background processing part of your application has an HTTP Post interface. You can leave polling to the daemon provided by beanstalk. I would recommend this if you want to use elasticbeanstalk.
If you want to use your component then you might as well just use web tier although if you are just doing background processing and not serving Web traffic then you will have ports listening for Web traffic, ELB for no reason.

Related

Apache Ignite autoscaling in AWS (linux)

Currently we host Apache Ignite node in AWS using Apache Ignite Image with 16 RAM.
We want to dynamically add new nodes while load on cache increases.
For this purpose we need to somehow trigger that node will run out of memory soon and we need to add additional node. Is there any way to track that?
I've tried to load cache with random data, cache failed with OutOfMemoryException when java process took over 30-40% of RAM.
Here is default-config.xml from {IGNITE_HOME}\config:
<?xml version="1.0" encoding="UTF-8"?>
...
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
</bean>
</list>
</property>
<!-- Enabling Apache Ignite Persistent Store. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="false"/>
<property name="metricsEnabled" value="true"/>
<property name="maxSize" value="#{10L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
<property name="awsCredentialsProvider" ref="aws.creds"/>
<property name="bucketName" value="dev-apache-ignite"/>
</bean>
</property>
</bean>
</property>
</bean>
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
<constructor-arg value="false" />
</bean>
</beans>
Sorry if this question was alsready answered in documentation.
Is there any predefined guidelines to configure AWS autoscaling for ignite?
Amount of RAM taken by JVM is not relevant. You will see IgniteOutOfMemoryException when you run out of data region - in your case, 10G.
You may divide DataRegionMetrics.getOffheapUsedSize() by data region size to understand how much runway do you have left.
Then you can probably use GridGain K8S Operator to scale your cluster:
https://www.gridgain.com/docs/latest/installation-guide/operator/how-tos

Is it possible to set expiry time using Apache Ignite in C++?

I am using the C++ thin client API and I want to have the data deleted from the cache after 5 minutes. I am connecting to ignite through docker and using the persistence storage. In the documentation for the C++ libraries, I cannot find anything relating to "expiry" and I tried adding the expiry option into the config xml file that my docker container reads in, but that didn't seem to work either. I put data into the cache and checked for the data after 5 minutes (I also checked 10, 20, 30 minutes later) and the data was still there.
Here is my config xml file:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Enabling Apache Ignite Persistent Store. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="name" value="Default_Region" />
</bean>
</property>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47502</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="MINUTES"/>
<constructor-arg value="5"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</beans>
Yes, the c++ thin client doesn't support this feature at the moment.
I think you could either define a cache with expiration completely on the server-side or to define only a cache template https://apacheignite.readme.io/docs/cache-template with expiry policy and use it from a client.

issue when profiling JVM Thread Pool ( under Spring intégration + Weblogic )

I have to maintain an application under Spring Integration (SI) and Weblogic 11g (with no workmanager configuration).
When I'm testing the performance on this application with JProfiler, I see a strange behavior regarding the threads : http://postimg.org/image/60w21njp3/
a thread pool with (approximatively) 160 threads
threads which are always created and destroyed
Under Spring, we use a SimpleAsyncTaskExecutor named "guillaume" and injected into a ListenerContainer like this :
<bean id="listenersExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor">
<property name="threadNamePrefix" value="guillaume-" />
<property name="daemon" value="true" />
</bean>
With :
<bean id="myListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="..." />
<property name="destination" ref="..." />
<property name="transactionManager" ref="..." />
<property name="taskExecutor" ref="listenersExecutor" />
<property name="errorHandler" ref="..." />
<property name="concurrentConsumers" value="..." />
<property name="idleConsumerLimit" value="..." />
<property name="cacheLevelName" value="..." />
</bean>
My question
I quote Javadoc on SimpleAsyncTaskExecutor :
TaskExecutor implementation that fires up a new Thread for each task, executing it asynchronously.
Who created the threadpool into JVM named "guillaume" ? not Spring ? Weblogic ?
Even if I don't have any activities with the SI, are there threads which are created and destroyed ?
Let me know if you need any additional informations.
Thanks.

Testing Spring-JPA

I am developping a web application using Spring (3.1.x), JSF 2, JPA 2 (Hibernate Provider) for tomcat 6.x.
I want to test my DAO classes.
In my DAO class: i do this:
#PersistenceContext
private EntityManager entityManager;
In Spring Configuration;
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalEntityManagerFactoryBean">
<property name="persistenceUnitName" value="OpenPU" />
</bean>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
In persistence.xml
<persistence-unit name="OpenPU" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<non-jta-data-source>java:comp/env/jdbc/mysql_open</non-jta-data-source>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQLInnoDBDialect" />
<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.transaction.flush_before_completion" value="true"/>
<property name="hibernate.cache.provider_class" value="org.hibernate.cache.HashtableCacheProvider"/>
<property name="hibernate.connection.zeroDateTimeBehavior" value="convertToNull"/>
</properties>
</persistence-unit>
It the first time I make test, and when I test I don't want to use the same persistence unit. I heard about dbunit for using xml data, but i don't understand how to change the persistence unit during the test.
Can you help me or give me some example, tutorial.
Thanks you.
Maybe this tutorial will help.
BTW, there is one interesting Spring feature to fit your needs - embedded database support. So, I usually just use following construction to create in-memory H2 db, create schema with schema.sql and fill it with some data from test-data.sql:
<jdbc:embedded-database id="dataSource" type="H2">
<jdbc:script location="classpath:schema.sql"/>
<jdbc:script location="classpath:test-data.sql"/>
</jdbc:embedded-database>
Then you could use this bean as datasource for you EntityManagerFactory bean:
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"
p:dataSource-ref="dataSource"
p:persistence-xml-location="classpath:META-INF/persistence.xml">
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="databasePlatform" value="org.hibernate.dialect.MySQLInnoDBDialect"/>
<property name="showSql" value="true" />
<!-- other properties -->
</bean>
</property>
<property name="persistenceUnitName" value="OpenPU" />
</bean>
This is very convenient and concise way to create in-memory database for tests with Spring.
(don't forget to add H2 in your classpath)
See documentation for details, chapter "13.8 Embedded database support".

Apache CXF + resource handler with embedded jetty in osgi with spring dm

I'm trying to run an apache cxf endpoint in an equinox osgi environment with jetty 7. I need the endpoint to be on address http://x.x.x.x:8080/ws/endpoint1 and have static resources on the root path http://x.x.x.x:8080/*.
I have a dedicated bundle for this purpose containing the cxf libraries. Spring dynamic modules are part of my target platform.
After some research I tried to start the jetty webserver in my spring application context.
<bean id="Server" class="org.eclipse.jetty.server.Server"
init-method="start" destroy-method="stop">
<property name="connectors">
<list>
<bean id="Connector" class="org.eclipse.jetty.server.nio.SelectChannelConnector">
<property name="port" value="8080" />
</bean>
</list>
</property>
<property name="handler">
<bean id="handlers" class="org.eclipse.jetty.server.handler.HandlerList">
<property name="handlers">
<list>
<bean class="org.eclipse.jetty.server.handler.ResourceHandler">
<property name="directoriesListed" value="true" />
<property name="welcomeFiles">
<list>
<value>index.html</value>
</list>
</property>
<property name="resourceBase" value="./someDir" />
</bean>
<ref bean="web-service-cxf" />
<bean class="org.eclipse.jetty.server.handler.DefaultHandler" />
</list>
</property>
</bean>
</property>
</bean>
<bean name="web-service-cxf" class="org.eclipse.jetty.servlet.ServletContextHandler">
<property name="contextPath" value="/ws" />
<property name="handler">
<bean class="org.eclipse.jetty.servlet.ServletHandler">
<property name="servlets">
<list>
<bean class="org.eclipse.jetty.servlet.ServletHolder">
<property name="name" value="cxf-servlet-holder" />
<property name="servlet">
<bean class="org.apache.cxf.transport.servlet.CXFServlet">
</bean>
</property>
</bean>
</list>
</property>
<property name="servletMappings">
<list>
<bean class="org.eclipse.jetty.servlet.ServletMapping">
<property name="servletName" value="cxf-servlet-holder" />
<property name="pathSpec" value="/*" />
</bean>
</list>
</property>
</bean>
</property>
</bean>
My WebService Endpoint is declared with:
<import resource="classpath:META-INF/cxf/cxf.xml" />
<import resource="classpath:META-INF/cxf/cxf-servlet.xml" />
<bean id="someService" class="abc.xyz.SomeClass" />
<jaxws:endpoint id="endpointId" implementor="#someBean"
address="/endpoint1">
</jaxws:endpoint>
Unfortunatly this is not working. I can reach the static resources, but not the webservice.
The log says, the WebService is published under address /endpoint1. No warnings, no exceptions.
When I change the address of the webservice to its full url
<bean id="someService" class="abc.xyz.SomeClass" />
<jaxws:endpoint id="endpointId" implementor="#someBean"
address="http://x.x.x.x:8080/ws/endpoint1">
</jaxws:endpoint>
the webservice works fine, but the static ressources are not available any more.
Is it possible with a configuration like this to publish an endpoint to a running jetty with relative address? Or am I totally wrong? Most examples I found on the web use a web.xml, but I'm not working with an application server like tomcat and need the application to be a standalone eclipse app.
Spend the whole last two nights on this, any help is highly appreciated.
Kind regards,
Onno
There are sooo many samples here. U shud be able to find what u r looking for
http://svn.apache.org/repos/asf/cxf/branches/2.4.x-fixes/distribution/src/main/release/samples