I'm working with a system where a number of jobs, implemented as Java applications, can be started simultaneously. Each job runs in a separate JVM.
Some of these jobs require bigger permgen size than others. However, it is not feasible to allow all jobs to use the maximum value, as the OS memory is limited.
Therefore, I want to specify -XX:MaxPermSize for every job. Currently, the jobs are running without any -XX:MaxPermSize argument, so they must be using the default value. But how can I find out what the default value is?
I have seen Default values for Xmx, Xms, MaxPermSize on non-server-class machines where the accepted answer is to run java -XX:+PrintFlagsFinal, which should output the default values. However, the JVM version I'm running does not support that argument (Unrecognized VM option '+PrintFlagsFinal'). Updating to a newer JVM is not currently an option.
So what are my options for finding the default value?
System information:
> java -version
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
> cat /etc/issue
Welcome to SUSE Linux Enterprise Server 11 SP2 (x86_64) - Kernel \r (\l).
> uname -r
3.0.101-0.7.17-default
Default values for various regions would depend on:
Collector being used (which would depend on Java version in case you're specifying it explicitly using CLI args).
Sizes of heap etc you have specified using CLI args. GC would distribute the space according to some ratios.
Installed (or may be it was available) RAM on the machine.
How to find out:
From GC Log files (-Xloggc:gc.log), I would expect that at least in the Full GC logs, GC would report the Perm Gen sizes. See examples at bottom. You can take a representative gc log file and find the max perm gen size from it, and decide based on that.
Additional params like PrintFlagsFinal etc. (specific to Java version)
I'll look through the 1.6 options to see if I can find something and update the post, otherwise it's time to upgrade. :-)
Here are 3 examples from different GCs (Metaspace, CMS Perm & PSPermGen is what you're looking for):
2014-11-14T08:43:53.197-0500: 782.973: [Full GC (Ergonomics) [PSYoungGen: 54477K->0K(917504K)] [ParOldGen: 1042738K->367416K(1048576K)] 1097216K->367416K(1966080K), [Metaspace: 46416K->46389K(1091584K)], 0.4689827 secs] [Times: user=3.52 sys=0.07, real=0.47 secs]
2014-10-29T06:14:56.491-0400: 6.754: [Full GC2014-10-29T06:14:56.491-0400: 6.754: [CMS: 96098K->113997K(5242880K), 0.7076870 secs] 735545K->113997K(6186624K), [CMS Perm : 13505K->13500K(51200K)], 0.7078280 secs] [Times: user=0.69 sys=0.01, real=0.71 secs]
2014-10-29T21:13:33.140-0500: 2644.411: [Full GC [PSYoungGen: 2379K->0K(695296K)] [ParOldGen: 1397977K->665667K(1398272K)] 1400357K->665667K(2093568K) [PSPermGen: 106995K->106326K(262144K)], 1.2151010 secs] [Times: user=6.83 sys=0.09, real=1.22 secs]
Related
I am following the tutorial on
Center for High Throughput Computing and Introduction to Configuration in the HTCondor website to set up a Partitionable slot. Before any configuration I run
condor_status
and get the following output.
I update the file 00-minicondor in /etc/condor/config.d by adding the following lines at the end of the file.
NUM_SLOTS = 1
NUM_SLOTS_TYPE_1 = 1
SLOT_TYPE_1 = cpus=4
SLOT_TYPE_1_PARTITIONABLE = TRUE
and reconfigure
sudo condor_reconfig
Now with
condor_status
I get this output as expected. Now, I run the following command to check everything is fine
condor_status -af Name Slotype Cpus
and find slot1#ip-172-31-54-214.ec2.internal undefined 1 instead of slot1#ip-172-31-54-214.ec2.internal Partitionable 4 61295 that is what I would expect. Moreover, when I try to summit a job that asks for more than 1 cpu it does not allocate space for it (It stays waiting forever) as it should.
I don't know if I made some mistake during the installation process or what could be happening. I would really appreciate any help!
EXTRA INFO: If it can be of any help have have installed HTCondor with the command
curl -fsSL https://get.htcondor.org | sudo /bin/bash -s – –no-dry-run
on Ubuntu 18.04 running on an old p2.xlarge instance (it has 4 cores).
UPDATE: After rebooting the whole thing it seems to be working. I can now send jobs with different CPUs requests and it will start them properly.
The only issue I would say persists is that Memory allocation is not showing properly, for example:
But in reality it is allocating enough memory for the job (in this case around 12 GB).
If I run again
condor_status -af Name Slotype Cpus
I still get something I am not supposed to
But at least it is showing the correct number of CPUs (even if it just says undefined).
What is the output of condor_q -better when the job is idle?
I am creating a ubicom-32 core under MCI-Server configuration in T32 tree(t32 start).
When i am starting the Core, I get
PBI=MCIServer (Illegal command)
Config=C:\Temp\T321000023.t32
T32SYS = C:\T32\ (config file)
I have the below settings in my T321000023.t32 file which is autogenerated.
;Connection to Host
PBI=MCISERVER
NODE=localhost
INSTANCE=1
CORE=4
How to fix this? Any other config is required for MCIServer set up?
I am not getting the error when i start ubicom32 core via simulator/JTAG dongle.
Lauterbach supports MCI-Server configuration only for a limited number of target processor architectures and currently Ubicom32 is not in the list of supported architectures.
Support for MCI-Server configuration requires not only effort to implement this feature for an architecture, but it also requires customers that have interest in this feature and are willing to purchase it. Please contact Lauterbach support support#lauterbach.com to discuss feasibility and conditions.
So here it goes: I wanted to use TensorFlow with GPU on AWS - p2.xlarge plan. Unfortunately, something must have gone wrong and I continue to get:
InvalidArgumentError (see above for traceback): Cannot assign a device to node 'Variable_1': Could not satisfy explicit device specification '/device:GPU:0' because no devices matching that specification are registered in this process; available devices: /job:localhost/replica:0/task:0/cpu:0
I checked both CUDA and cuDNN:
nvcc -V
cat /usr/local/cuda/include/cudnn.h
and got 8.0 and 5.1, respectively.
I call gpu like this:
with tf.device('/gpu:0'):
a = tf.Variable(tf.truncated_normal([100, 100]))
b = tf.Variable(tf.truncated_normal([100, 1000]))
with tf.Session() as sess:
sess.run(tf.matmul(a,b))
happy to post more details if necessary - don't know what will be useful yet.
I suppose you're trying to set up an EC2 instance from scratch? That can be difficult.
Instead, I'd strongly recommend using the Deep Learning AMI (https://aws.amazon.com/machine-learning/amis/). It comes preinstalled with everything you need (drivers, popular DL libraries, etc.). It's also free to use, you just pay for the instance itself.
Though the terminal on ubuntu:
db#morris:~/lisbet/elki-master/elki/target$ elki-cli -algorithm outlier.lof.LOF -dbc.parser ArffParser -dbc.in /home/db/lisbet/AllData/literature/WBC/WBC_withoutdupl_norm_v10_no_ids.arff -lof.k 8 -evaluator outlier.OutlierROCCurve -rocauc.positive yes
giving
# ROCAUC: 0.6230046948356808
and in ELKI's GUI:
Running: -verbose -dbc.in /home/db/lisbet/AllData/literature/WBC/WBC_withoutdupl_norm_v10_no_ids.arff -dbc.parser ArffParser -algorithm outlier.lof.LOF -lof.k 8 -evaluator outlier.OutlierROCCurve -rocauc.positive yes
de.lmu.ifi.dbs.elki.datasource.FileBasedDatabaseConnection.parse: 18 ms
de.lmu.ifi.dbs.elki.datasource.FileBasedDatabaseConnection.filter: 0 ms
LOF #1/3: Materializing LOF neighborhoods.
de.lmu.ifi.dbs.elki.index.preprocessed.knn.MaterializeKNNPreprocessor.k: 9
Materializing k nearest neighbors (k=9): 223 [100%]
de.lmu.ifi.dbs.elki.index.preprocessed.knn.MaterializeKNNPreprocessor.precomputation-time: 10 ms
LOF #2/3: Computing LRDs.
LOF #3/3: Computing LOFs.
LOF: complete.
de.lmu.ifi.dbs.elki.algorithm.outlier.lof.LOF.runtime: 39 ms
ROCAUC: **0.6220657276995305**
I don't understand why the 2 ROCAUCcurves aren't the same.
My goal in testing this is to be comfortable with my result, that what I do is right, but it is hard when I don't get matching results. When I see that my settings are right I will move on to making my own experiments, that I can trust.
Pass cli as first command line parameter to launche the CLI, or minigui to launch the MiniGUI. The following are equivalent:
java -jar elki/target/elki-0.6.5-SNAPSHOT.jar cli
java -jar elki/target/elki-0.6.5-SNAPSHOT.jar KDDCLIApplication
java -jar elki/target/elki-0.6.5-SNAPSHOT.jar de.lmu.ifi.dbs.elki.application.KDDCLIApplication
This will work for any class extending the class AbstractApplication.
Your can also do:
java -cp elki/target/elki-0.6.5-SNAPSHOT.jar de.lmu.ifi.dbs.elki.application.KDDCLIApplication
(Which will load 1 class less, but this is usually not worth the effort.)
This will work for any class that has a standard public void main(String[]) method, as this is the standard Java invocation.
But notice that -h currently will still print 0.6.0 (2014, January), that value was not updated for the 0.6.5 interim versions. It will be bumped for 0.7.0. That version number is therefore not reliable.
As for the differences you observed: try varing k by 1. If I recall correctly, we changed the meaning of the k parameter to be more consistent across different algorithms. (They are not consistent in literature anyway.)
Trying to read the sizes of disks that were created in multiple sessions using GetDiskFreeSpaceEx() gives the size of the last session only. How do I read correctly the number and sizes of all sessions in C/C++?
Thanks.
You might want to look at the DeviceIoControl API function. See here for control codes. Here is a code example that retrieves the size of a CD disk. Substitute
CreateFile(TEXT("\\\\.\\PhysicalDrive0")
for e.g.
CreateFile(TEXT("\\\\.\\F:") /* Drive is F: */
if you wish.
Note: The page says that DeviceIoControl can be used to "retrieve information about a floppy disk drive, hard disk drive, tape drive, or CD-ROM drive", but I have also tested it on a DVD, and it seemed to work perfectly. I did not have access to any multisession DVDs to test, so you'll have to test if that works yourself. If it doesn't work, I'd try some of the other control codes, at least IOCTL_DISK_GET_DRIVE_GEOMETRY_EX, IOCTL_DISK_GET_DRIVE_LAYOUT_EX, IOCTL_DISK_GET_LENGTH_INFO and IOCTL_DISK_GET_PARTITION_INFO_EX.
If all fails with DeviceIoControl, you could possibly make use of the Windows Image Mastering API (IMAPI). You'll need v2 of the API (included with Vista & later, can be added to XP & 2003 too, see here: What's new in IMAPIv2) for DVD support. This API is primarily for CD burning, but does perhaps contain some functionality for retrieving disk size, I'd find it weird if it didn't. Particularly, this example seems to be interesting. I do not know if this one works for multisession disks either, but since it can create them, I guess it's likely.
Here are some resources for IMAPI:
MSDN - IMAPI
MSDN - IMAPI interfaces
MSDN - Creating multisession disks with IMAPI (note: example with VB, not C or C++)
Hey I got at least 2 solutions for you:
1) Download dvd+rw-mediainfo.exe from http://fy.chalmers.se/~appro/linux/DVD+RW/tools/win32/, it's a tool that reads info about your disc. Then just make a system call from your app and parse the results. Here's example output:
D:\Downloads>"dvd+rw-mediainfo.exe" f:
INQUIRY: [HL-DT-ST][DVDRAM GT30N ][1.01]
GET [CURRENT] CONFIGURATION:
Mounted Media: 10h, DVD-ROM
Current Write Speed: 1.0x1385=1385KB/s
Write Speed #0: 8.0x1385=11080KB/s
Write Speed #1: 4.0x1385=5540KB/s
Write Speed #2: 2.0x1385=2770KB/s
Write Speed #3: 1.0x1385=1385KB/s
Speed Descriptor#0: 00/2292991 R#8.0x1385=11080KB/s W#8.0x1385=11080KB/s
READ DVD STRUCTURE[#0h]:
Media Book Type: 01h, DVD-ROM book [revision 1]
Legacy lead-out at: 2292992*2KB=4696047616
READ DISC INFORMATION:
Disc status: complete
Number of Sessions: 1
State of Last Session: complete
Number of Tracks: 1
READ TRACK INFORMATION[#1]:
Track State: complete
Track Start Address: 0*2KB
Free Blocks: 0*2KB
Track Size: 2292992*2KB
Last Recorded Address: 2292991*2KB
FABRICATED TOC:
Track#1 : 17#0
Track#AA : 17#2292992
Multi-session Info: #1#0
READ CAPACITY: 2292992*2048=4696047616
2) Investigate mciSendString from [DllImport("winmm.dll", EntryPoint = "mciSendStringA", CharSet = CharSet.Ansi)], I suspect you can send some command and get the desired results.
PS: of course you may download dvd+rw-mediainfo.exe sources from here and investigate further, I am just giving you ideas to think of.
UPDATE
Link to source code updated, thanks #oystein
There are many way to do this since the DVD drives have several interfaces for this due to legacy and backward-compatibility issues.
You could send an IOCTL_SCSI_PASSTHROUGH_DIRECT command to the DVD-drive ( the physicaldevice handle for it). With it you issue a SCSI commands that will be answered by the drive. You can read session information, disk information disk capcity and more.
I believe that dvd+rw-mediainfo.exe issues these.
Unfortunatly, the interface is a bit tricky and obscure, since it is a command within a command. Th passthrough has a byte buffer you will have to fill in yourself with the command structure.
Or you can call IOCTL_CDROM_READ_TOC_EX:
http://www.osronline.com/ddkx/storage/k306_2cs2.htm
I also believe that the exact set of the IOCTL / commands that will work depends on on the drive and its firmaware.
Older drives will not support the newr interfaces and some of the newer drives will not support legacy interfaces.
Thus, some of the libraries & tools might use one or more of these interfaces.
Accseeing the older sessons is all quite messy, really, since most OS will not care about them, only the most recent ones.