Recommended Heap Configuration - heap

I can not choose optimal configuration for wildfly as. I have a droplet on DigitalOcean and it has 2GB Ram and 1vCPU. I have a social media application with mongodb(login and see your followers). This app triggers the Firebase Cloud Message service every 15 min. after FCM send notification to clients and clients send a request to server. After server make some db read/write operations. But the problem is server can not response approximately every 2-3 hours so I need to restart it. I track memory usage graph. after reboot memory usage graph increase slowly but always increase. Is this about wildfly conf or you can say anything about this? There is Nginx front of the Wildfly.
Wildfly conf:
bin/standalone.conf
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms768m -Xmx1024m -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true -Duser.timezone=GMT+3"
else
echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi
domain/configuration/domain.xml
<server-group name="main-server-group" profile="full">
<jvm name="default">
<heap size="1024m" max-size="1536m"/>
</jvm>
<socket-binding-group ref="full-sockets"/>
</server-group>
<server-group name="other-server-group" profile="full-ha">
<jvm name="default">
<heap size="1024m" max-size="1536m"/>
</jvm>
<socket-binding-group ref="full-ha-sockets"/>
</server-group>
domain/configuration/host.xml
<jvms>
<jvm name="default">
<heap size="1024m" max-size="1536m"/>
<jvm-options>
<option value="-server"/>
<option value="-XX:MetaspaceSize=96m"/>
<option value="-XX:MaxMetaspaceSize=256m"/>
<option value="--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"/>
</jvm-options>
</jvm>
</jvms>
Thank you.

Have you tried setting the Garbage collection in 'standalone.conf' located in libexec/bin?
I switched to Oracle's G1 garbage collection and it sorted out all my "out of memory" problems on WildFly 10/11. Now using it with 12.
http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
# G1 Garbage Collector
JAVA_OPTS="-server -Xms2g -Xmx8g"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=200"
JAVA_OPTS="$JAVA_OPTS -XX:InitiatingHeapOccupancyPercent=45"
JAVA_OPTS="$JAVA_OPTS -XX:G1ReservePercent=25"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/Users/NOTiFY/IdeaProjects/MigrationTool/garbage-collection.log"

Related

Bypassing Cloud Run 32mb error via HTTP2 end to end solution

I have an api query that runs during a post request on one of my views to populate my dashboard page. I know the response size is ~35mb (greater than the 32mb limits set by cloud run). I was wondering how I could by pass this.
My configuration is set via a hypercorn server and serving my django web app as an asgi app. I have 2 minimum instances, 1gb ram, 2 cpus per instance. I have run this docker container locally and can't bypass the amount of data required and also do not want to store the data due to costs. This seems to be the cheapest route. Any pointers or ideas would be helpful. I understand that I can bypass this via http2 end to end solution but I am unable to do so currently. I haven't created any additional hypecorn configurations. Any help appreciated!
The Cloud Run HTTP response limit is 32 MB and cannot be increased.
One suggestion is to compress the response data. Django has compression libraries for Python or just use zlib.
import gzip
data = b"Lots of content to compress"
cdata = gzip.compress(s_in)
# return compressed data in response
Cloud Run supports HTTP/1.1 server side streaming, which has unlimited response size. All you need to do is use chunked transfer encoding.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Simple HelloWorld app on cloudrun (or knative) seems too slow

I deployed a sample HelloWorld app on Google Cloud Run, which is basically k-native, and every call to the API takes 1.4 seconds at best, in an end-to-end manner. Is it supposed to be so?
The sample app is at https://cloud.google.com/run/docs/quickstarts/build-and-deploy
I deployed the very same app on my localhost as a docker container and it takes about 22ms, end-to-end.
The same app on my GKE cluster takes about 150 ms, end-to-end.
import os
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
target = os.environ.get('TARGET', 'World')
return 'Hello {}!\n'.format(target)
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
I am a little experience in FaaS and I expect API calls would get faster as I invoked them in a row. (as in cold start vs. warm start)
But no matter how many times I execute the command it doesn't go below 1.4 seconds.
I think the network distance isn't the dominant factor here. The round-trip time via ping to the API endpoint is only 50ms away, more or less
So my questions are as follows:
Is it potentially an unintended bug? Is it a technical difficulty which will be resolved eventually? Or maybe nothing's wrong, it's just the SLA of k-native?
If nothing's wrong with Google Cloud Run and/or k-native, what is the dominant time-consuming factor here for my API call? I'd love to learn the mechanism.
Additional Details:
Where I am located at: Seoul/Asia
The region for my Cloud Run app: us-central1
type of Internet connection I am testing under: Business, Wired
app's container image size: 343.3MB
the bucket location that Container Registry is using: gcr.io
WebPageTest from Seoul/Asia (warmup time):
Content Type: text/html
Request Start: 0.44 s
DNS Lookup: 249 ms
Initial Connection: 59 ms
SSL Negotiation: 106 ms
Time to First Byte: 961 ms
Content Download: 2 ms
WebPageTest from Chicago/US (warmup time):
Content Type: text/html
Request Start: 0.171 s
DNS Lookup: 41 ms
Initial Connection: 29 ms
SSL Negotiation: 57 ms
Time to First Byte: 61 ms
Content Download: 3 ms
ANSWER by Steren, the Cloud Run product manager
We have detected high latency when calling Cloud Run services from
some particular regions in the world. Sadly, Seoul seems to be one of
them.
[Update: This person has a networking problem in his area. I tested his endpoint from Seattle with no problems. Details in the comments below.]
I have worked with Cloud Run constantly for the past several months. I have deployed several production applications and dozens of test services. I am in Seattle, Cloud Run is in us-central1. I have never noticed a delay. Actually, I am impressed with how fast a container starts up.
For one of my services, I am seeing cold start time to first byte of 485ms. Next invocation 266ms, 360ms. My container is checking SSL certificates (2) on the Internet. The response time is very good.
For another service which is a PHP website, time to first byte on cold start is 312ms, then 94ms, 112ms.
What could be factors that are different for you?
How large is your container image? Check Container Registry for the size. My containers are under 100 MB. The larger the container the longer the cold start time.
Where is the bucket located that Container Registry is using? You want the bucket to be in us-central1 or at least US. This will change soon with when new Cloud Run regions are announced.
What type of Internet connection are you testing under? Home based or Business. Wireless or Ethernet connection? Where in the world are you testing from? Launch a temporary Compute Engine instance, repeat your tests to Cloud Run and compare. This will remove your ISP from the equation.
Increase the memory allocated to the container. Does this affect performance? Python/Flask does not require much memory, my containers are typically 128 MB and 256 MB. Container images are loaded into memory, so if you have a bloated container, you might now have enough memory left reducing performance.
What does Stackdriver logs show you? You can see container starts, requests, and container terminations.
(Cloud Run product manager here)
We have detected high latency when calling Cloud Run services from some particular regions in the world. Sadly, Seoul seems to be one of them.
We will explicitly capture this as a Known issue and we are working on fixing this before General Availability. Feel free to open a new issue in our public issue tracker

System Center Virtual Machine Manager Service Unable to start

I have install Virtual Machine Manager on windows server 2012 R2 . it installed sucessfuly but the SCVMM service can't be started please check the following log for and the error
when i try to start SCVMM service its endup with the following error
" The System Center Virtual Machine Manager Service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs.
"
Log Name: Application
Source: SCVMMService
Date: 11/4/2014 7:58:55 AM
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: HC**********
Description:
Service cannot be started. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at Microsoft.VirtualManager.Remoting.IndigoSerializableObject.BuildKnownAssemblyTypes(Assembly assembly)
at Microsoft.VirtualManager.Remoting.IndigoSerializableObject.InitializeKnownTypesCache(List`1 assembliesToExamine)
at Microsoft.VirtualManager.Engine.Remoting.IndigoServiceHost.InitializeKnownTypesCache()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TimeStartupMethod(String description, TimedStartupMethod methodToTime)
at Microsoft.VirtualManager.Engine.VirtualManagerService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="SCVMMService" />
<EventID Qualifiers="0">0</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-11-04T07:58:55.000000000Z" />
<EventRecordID>10656</EventRecordID>
<Channel>Application</Channel>
<Computer>HC-S*********** </Computer>
<Security />
</System>
<EventData>
<Data>Service cannot be started. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at Microsoft.VirtualManager.Remoting.IndigoSerializableObject.BuildKnownAssemblyTypes(Assembly assembly)
at Microsoft.VirtualManager.Remoting.IndigoSerializableObject.InitializeKnownTypesCache(List`1 assembliesToExamine)
at Microsoft.VirtualManager.Engine.Remoting.IndigoServiceHost.InitializeKnownTypesCache()
at Microsoft.VirtualManager.Engine.VirtualManagerService.TimeStartupMethod(String description, TimedStartupMethod methodToTime)
at Microsoft.VirtualManager.Engine.VirtualManagerService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)</Data>
</EventData>
</Event>
Thanks for posting here.
To resolve this issue, review the AD group membership for any circular references and remove them. If the circular reference is not obvious, the script below can list the nested members of a particular group.
NOTE Use Active Directory Users and Computers to determine the top most group of the group causing issues.
dsquery group -name "Group_VMMSSU" | dsget group -members -expand
I hope you have followed these steps before installing SCVMM
High Level Deployment Process:
In AD, create the following accounts and groups, according to your naming convention:
•DOMAIN\scvmmsvc SCVMM Service Account account
•DOMAIN\scvmmadmin SCVMM RunAs account for managing hosts
•DOMAIN\sqlsvc SQL service account
•DOMAIN\SCVMMAdmins SCVMM Administrators security group
Add the “scvmmsvc” and “scvmmadmin” account to the “SCVMMAdmins” global group.
Add the domain user accounts for yourself and your team to the SCVMMAdmins group.
I suggest you to check this link for more details.
http://blogs.technet.com/b/kevinholman/archive/2013/10/18/scvmm-2012-r2-quickstart-deployment-guide.aspx
Girish Prajwal
I had this error and couldn't find an updated answer to the problem.
In my case, I have SCVMM (server AND console) installed on the same machine (i guess this is pretty normal for a first time installation).
I had seen the system working, then I installed all updates offered from Windows Update. After reboot, my service "System Center Virtual Machine Manager" could not start.
Solution was to look in control panel: "Programs and Features". Clicking "installed updates" there was an entry for System Center Console.
This was a rollup 10.
I uninstalled this, now the service is able to start, and I can log on.
Hope this helps someone.
This KB article from Microsoft addresses the issue- https://support.microsoft.com/en-us/kb/3045931
To resolve this issue, complete the update rollup installation by installing the VMM server update package in addition to the console update. Both the VMM server and VMM Admin Console updates must be at compatible version levels. This will typically be the same update rollup level. For example, a VMM Admin Console should be at the VMM UR5 level if the server is also at UR5.  
In some instances, there may not be a comparable update for a component. In that case, the component should be updated to the most recent applicable update. For example, if the VMM server is updated to a post-UR5 hotfix, but there is no corresponding hotfix for the VMM Admin Console, the VMM Admin Console should be updated to UR5.
In my case update rollup 11 was the cause for the break. Going in to Programs and Features -> Installed Updates, I noticed that Update rollup 11 for the admin console had installed successfully but Update rollup 11 for the VMM service had failed.
I manually installed the failed update (in my case http://download.microsoft.com/download/7/5/1/7515E3DB-1B43-4E7F-8774-A877CCEAA96E/kb3184831_vmmserver_amd64.msp) and now VMM service is able to start and stay started.

Load testing results on blitz.io and new relic for the same config show different queuing times

i am testing a rails 4.0 app on blitz.io and new relic..
Blitz: 200 users, 4000 Timeout, in 60sec.
Heroku: 20 dynos (2x standard dynos).
Database: Postgresql 200 connections plan
Using Unicorn at 7 processes(new relic shows that my app uses about 140mb,
so for a 2x standard dyno, you get 1gb, and so 1024/140 is about 7. link ref for this in below line)
ref: http://blog.bignerdranch.com/2188-how-to-improve-rails-application-performance-on-heroku/
For the same config and multiple tests i see response queuing in new relic ranging from 240ms to 443ms. I would like to know why its varying that much. what else should i be monitoring. how can i reduce it to atleast 50ms? also what is the ideal response queuing time?
thanks

How to determine if application/application container has started on AWS elastic beanstalk?

I am writing a generic application deployment tool. It takes an application from the user and deploys it to Elastic Beanstalk. That part is working. The issue is that the users want to compose the use of the deployment tool with other operations, and right now my tool reports success when it has told the Beanstalk APIs to start the application.
Unfortunately, it is thus returning before the application itself has started. So the user is forced to write polling logic themselves to await the starting of their application.
Looking at the AWS Elastic Beanstalk API and I cannot see any methods that return any indication of such a state reporting. The closest I can find is DescribeEvents... which looks hopeful, however it seems from the examples that the granularity of the application / application container starting within the environment is not part of that API:
<DescribeEventsResponse xmlns="https://elasticbeanstalk.amazonaws.com/docs/2010-12-01/">
<DescribeEventsResult>
<Events>
<member>
<Message>Successfully completed createEnvironment activity.</Message>
<EventDate>2010-11-17T20:25:35.191Z</EventDate>
<VersionLabel>New Version</VersionLabel>
<RequestId>bb01fa74-f287-11df-8a78-9f77047e0d0c</RequestId>
<ApplicationName>SampleApp</ApplicationName>
<EnvironmentName>SampleAppVersion</EnvironmentName>
<Severity>INFO</Severity>
</member>
Note: the INFO level event is that the environment was created, nothing at the lower level of the application container starting within the environment appears to be reported...
I could mandate that the applications deployed with this tool expose a status REST endpoint, but that puts restrictions on the application.
Is there some API that I am missing that will report when the application container (e.g. Tomcat, Node, etc) is started... or better yet when the application deployed within the container is started... but I can live with the application container
Every application is supposed to expose a health URL (Beanstalk/ELB will have problems any case otherwise - it will think the instances are not responding, and might replace). This is typically a HEAD request expecting a 200 OK.
Since this is anyway expected to be there in all apps, you can probably hit this URL and check the deployment is OK. I guess Beanstalk console itself is using this method.
You can also poll using DescribeEnvironments API call which will give you the Environment CNAME (the URL to check), Health of the environment (RED, GREEN), Status (Launching | Updating | Ready | Terminating | Terminated). This API takes Environment Name as an argument. So you can just get the description of one environment.
API Documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/APIReference/API_DescribeEnvironments.html
Explanation of Environment Description in response: http://docs.aws.amazon.com/elasticbeanstalk/latest/APIReference/API_EnvironmentDescription.html
Sample Response Below:
<DescribeEnvironmentsResponse xmlns="https://elasticbeanstalk.amazonaws.com/docs/2010-12-01/">
<DescribeEnvironmentsResult>
<Environments>
<member>
<VersionLabel>Version1</VersionLabel>
<Status>Available</Status>
<ApplicationName>SampleApp</ApplicationName>
<EndpointURL>elasticbeanstalk-SampleApp-1394386994.us-east-1.elb.amazonaws.com</EndpointURL>
<CNAME>SampleApp-jxb293wg7n.elasticbeanstalk.amazonaws.com</CNAME>
<Health>Green</Health>
<EnvironmentId>e-icsgecu3wf</EnvironmentId>
<DateUpdated>2010-11-17T04:01:40.668Z</DateUpdated>
<SolutionStackName>32bit Amazon Linux running Tomcat 7</SolutionStackName>
<Description>EnvDescrip</Description>
<EnvironmentName>SampleApp</EnvironmentName>
<DateCreated>2010-11-17T03:59:33.520Z</DateCreated>
</member>
</Environments>
</DescribeEnvironmentsResult>
<ResponseMetadata>
<RequestId>44790c68-f260-11df-8a78-9f77047e0d0c</RequestId>
In your case you may want to read following documentation:
Monitoring Application Health
You can also configure Application Health Check URL for your environment. By default AWS Elastic Beanstalk uses TCP:80 check on your instances. But using the Application Health Check URL you can override this health check to use HTTP:80 by using the Application Health Check URL option as described here.
Using Status/Health from DescribeEnvironments you can check if the application has been deployed.