Since past two days, rsyslogd is using 100% CPU Utilization on all RHEL EC2 Instances in my environment. I Stopped and started rsyslog service but still issue persists.
This is first time we are facing this kind of behaviour in multiple servers.
There is sufficient disk space/memory exists in all servers.
I checked kernel logs (/var/log/kernel) and server messages( /var/log/messages),but not find any useful info.
Following is the OS and Kernal versions of all servers.
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
Kernel: Linux 3.10.0-1160.42.2.el7.x86_64
Can someone please suggest on this.
Issue is due to cert expiry for rsyslog. Observed a lot of connection retry errors in system messages for all clients. Post renewing certs in rsyslog, cpu went down.
A recent security scan of our systems has shown that our AWS Linux 2 instances are vulnerable to an issue with systemd
https://nvd.nist.gov/vuln/detail/CVE-2021-33910
I've had a look through the AWS Linux 2 security bulletin board (https://alas.aws.amazon.com/alas2.html) and at present they don't seem to have anything for this problem.
I tried to manually update systemd but we're already running the latest version.
Has anyone come across this, or found a resolution? At the moment all I can do is sit and wait for AWS to get around to patching the vulnerability.
As this vulnerability has been modified (14/07) since its last analysis, patching might take some time.!
In the mean time if possible you could use Red Hat Enterprise Linux as those are fixed and some older versions are unaffected like Red Hat Enterprise Linux 7.
I have Informatica 9.5 installed on a Windows 2008 server. In Task Manager, I've noticed java.exe *32 is consuming a lot of memory on the Informatica server. Can someone tell me what this process is used for? What could be the cause for the high memory? Is there a fix?
Thank you.
We are trying to use VisualVM to track down some memory leakage in CF8, however, cannot get the tool to work 100%. Basically, everything comes up, except the Memory sampling. Says that the "JVM is not supported".
However, all the other features work (we can do CPU sampling, just not memory). Found this kind of weird that we can do everything else but the memory stuff, so am wondering if maybe we need to specify another JVM argument to allow this?
Some other info:
We are connecting locally via 127.0.0.1 or localhost.
I installed the Visual GC plugin, and it cannot connect either.
VisualVM and JRUN/CF8 are both using the same Java version (1.6.0_31), however, they are not pulled from the same location (maybe this matters). VisualVM uses the installed JDK, whereas JURN/CF8 uses just the binaries that we copied locally to the CF8 installation folder.
Installed another plugin that shows JVM properties, and it says that the JVM is not "attachable". Don't know what that means, but am just wanting to mention it.
Any help with this would be greatly appreciated. If we can just get that memory sampling, I think we can get on top of our performance issues that have plagued us here recently. Thanks in advance!
EDIT:
Also, just checked, and JRUN is being started under "administrator", whereas I am launching VisualVM under a different user. Maybe this is relevant?
Yes, it is relevant that you are running VisualVM under different user. Memory Sampling uses Attach API, which only works if you are running monitored application and VisualVM as the same user. This is also reason that the JVM properties reports that your application is not attachable. If you run VisualVM as "administrator", it will automatically detect your Coldfusion 8 application and the Memory sampler will work.
I have noticed that our VMWare VMs often have the incorrect time on them. No matter how many times I reset the time they keep on desyncing.
Has anyone else noticed this? What do other people do to keep their VM time in sync?
Edit: These are CLI linux VMs btw..
If your host time is correct, you can set the following .vmx configuration file option to enable periodic synchronization:
tools.syncTime = true
By default, this synchronizes the time every minute. To change the periodic rate, set the following option to the desired synch time in seconds:
tools.syncTime.period = 60
For this to work you need to have VMWare tools installed in your guest OS.
See http://www.vmware.com/pdf/vmware_timekeeping.pdf for more information
according to VMware's knowledge base, the actual solution depends on the Linux distro and release, in RHEL 5.3 I usually edit /etc/grub.conf and append this parameters to the kernel entry: divider=10 clocksource=acpi_pm
Then enable NTP, disable VMware time synchronization from vmware-toolbox and finally reboot the VM
A complete table with guidelines for each Linux distro can be found here:
TIMEKEEPING BEST PRACTICES FOR LINUX GUESTS
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
I'll answer for Windows guests. If you have VMware Tools installed, then the taskbar's notification area (near the clock) has an icon for VMware Tools. Double-click that and set your options.
If you don't have VMware Tools installed, you can still set the clock's option for internet time to sync with some NTP server. If your physical machine serves the NTP protocol to your guest machines then you can get that done with host-only networking. Otherwise you'll have to let your guests sync with a genuine NTP server out on the internet, for example time.windows.com.
Something to note here. We had the same issue with Windows VM's running on an ESXi host. The time sync was turned on in VMWare Tools on the guest, but the guest clocks were consistently off (by about 30 seconds) from the host clock. The ESXi host was configured to get time updates from an internal time server.
It turns out we had the Internet Time setting turned on in the Windows VM's (Control Panel > Date and Time > Internet Time tab) so the guest was getting time updates from two places and the internet time was winning. We turned that off and now the guest clocks are good, getting their time exclusively from the ESXi host.
In my case we are running VMWare Server 2.02 on Windows Server 2003 R2 Standard. The Host is also Windows Server 2003 R2 Standard. I had the VMware Tools installed and set to sync the time. I did everything imaginable that I found on various internet sites. We still had horrendous drift, although it had shrunk from 15 minutes or more down to the 3 or 4 minute range.
Finally in the vmware.log I found this entry (resides in the folder as the .vmx file):
"Your host system does not guarantee synchronized TSCs across different CPUs, so please set the /usepmtimer option in your Windows Boot.ini file to ensure that timekeeping is reliable. See Microsoft KB http://support.microsoft.com/kb... for details and Microsoft KB http://support.microsoft.com/kb... for additional information."
Cause: This problem occurs when the computer has the AMD Cool'n'Quiet technology (AMD dual cores) enabled in the BIOS or some Intel multi core processors. Multi core or multiprocessor systems may encounter Time Stamp Counter (TSC) drift when the time between different cores is not synchronized. The operating systems which use TSC as a timekeeping resource may experience the issue. Newer operating systems typically do not use the TSC by default if other timers are available in the system which can be used as a timekeeping source. Other available timers include the PM_Timer and the High Precision Event Timer (HPET).
Resolution: To resolve this problem check with the hardware vendor to see if a new driver/firmware update is available to fix the issue.
Note The driver installation may add the /usepmtimer switch in the Boot.ini file.
Once this (/usepmtimer switch) was done the clock was dead on time.
This documentation solved this problem for me.
The CPU speed varies due to power saving. I originally noticed this because VMware gave me a helpful tip on my laptop, but this page mentions the same thing:
Quote from : VMWare tips and tricks
Power saving (SpeedStep, C-states, P-States,...)
Your power saving settings may interfere significantly with vmware's performance. There are several levels of power saving.
CPU frequency
This should not lead to performance degradation, outside of having the obvious lower performance when running the CPU at a lower frequency (either manually of via governors like "ondemand" or "conservative"). The only problem with varying the CPU speed while vmware is running is that the Windows clock will gain of lose time. To prevent this, specify your full CPU speed in kHz in /etc/vmware/config
host.cpukHz = 2167000
VMware experiences a lot of clock drift. This Google search for 'vmware clock drift' links to several articles.
The first hit may be the most useful for you: http://www.fjc.net/linux/linux-and-vmware-related-issues/linux-2-6-kernels-and-vmware-clock-drift-issues
When installing VMware Tools on a Windows Guest, “Time Synchronisation” is not enabled by default.
However – “best practise” is to enable time synch on Windows Guests.
There a several ways to do this from outside the VM, but I wanted to find a way to enable time sync from within the guest itself either on or after tools install.
Surprisingly, this wasn’t quite as straightforward as I expected.
(I assumed it would be posible to set this as a parameter / config option during tools install)
After a bit of searching I found a way to do this in a VMware article called “Using the VMware Tools Command-Line Interface“.
So, if time sync is disabled, you can enable it by running the following command line in the guest:
VMwareService.exe –cmd “vmx.set_option synctime 0 1″
Additional Notes
For some (IMHO stupid) reason, this utility requires you to specify the current as well as the new value
0 = disabled
1 = enabled
So – if you run this command on a machine which has this already set, you will get an error saying – “Invalid old value“.
Obviously you can “ignore” this error when run (so not a huge deal) but the current design seems a bit dumb.
IMHO it would be much more sensible if you could simply specify the value you want to set and not require the current value to be specified.
i.e.
VMwareService.exe –cmd “vmx.set_option synctime <0|1>”
In Active Directory environment, it's important to know:
All member machines synchronizes with any domain controller.
In a domain, all domain controllers synchronize from the PDC Emulator (PDCe) of that domain.
The PDC Emulator of a domain should synchronize with local or NTP.
It's important to consider this when setting the time in vmware or configuring the time sync.
Extracted from: http://www.sysadmit.com/2016/12/vmware-esxi-configurar-hora.html
I added the following job to crontab. It is hacky but i think should work.
*/5 * * * * service ntpd stop && ntpdate pool.ntp.org && service ntpd start
It stops ntpd service updates from service and starts ntpd again