I'm using the Symfony2 Framework with Doctrine using a mysql database, when running a Symfony command on the command line such as ...
/path/to/php app/console myCommand
This command is to update loads of records (about 35,000+) and I get a memory exhaustion at 1Gb, is there a better and probably more efficient way to run these commands so I dont get this issue?
I could increase the memory limit but ultimately this is going to fail again at some point.
Thanks in advance
Try using batch processing To handle the updates in chunks.
http://docs.doctrine-project.org/en/2.1/reference/batch-processing.html
Call clear on the entity manager
$em = $this->getContainer()->get('doctrine')->getEntityManager();
$em->flush();
$em->clear();
As you are only updating your data, and not requiring it to remain in memory.
Related
first of all, I assure you, I have googled for hours now. My main problem is that I'm trying to fix a corrupted database of the tool paperless-ngx that I'm using. I am an IT admin but I have no experience with SQL whatsoever.
I'm getting this error:
ERROR: missing chunk number 0 for toast value 52399 in pg_toast_2619
Now every guide on the entire internet (I'm gonna post this one for reference) on how to fix this tells me to REINDEX the table.
When I do this using
reindex (verbose) table django_q_task;
it keeps waiting indefinitely with this errormessage:
WARNING: concurrent insert in progress within table "django_q_task"
I am positive that there is no write happening from paperless side, all containers except for the database container have been stopped. I tried locking the table using
lock table django_q_task in exclusive mode nowait;
but the error persists. I'm at the end of my wits. I beg of you, can someone provide me with detailed instructions for someone with no postgresql experience at all?
I want to make an application run slower, it that possible? I have created application which read file created by another process but that process create file and delete it so fast, so it is possible to make that application be slow so I can read file faster?
I tried
SetPriorityClass(GetProcessHandleByName("dd.exe"), IDLE_PRIORITY_CLASS);
and set my process to
SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS);
but yet the process run faster it is possible to slow it down? thanks.
Modifying the working directory permissions to allow processes to read/write data to new files, but not modify/delete existing files might be a different approach that would work.
See https://superuser.com/questions/745923/ntfs-permissions-create-files-and-folder-but-prevent-deletion-and-modification
See the answer SO : Suspend/Resume a process. Which gives information on the three choices for suspending an application.
They are basically stop each thread. Use the undocumented SuspendProcess and Debug the process.
These are the methods of substantially delaying the process.
I'm developing a server application that preforms various (probably hundreds to thousands) of MySQL Queries a day (SELECTS, INSERTS, & UPDATES).
The querying works great, until....
For some reason after the Server has been up for roughly 1 to 2 days it generates a MySQL Error any time I try to preform any MySQL Query from the Server... The Server was developed using C++.
The error says that The MySQL Client has run out of memory.
I'm Using
MySQL Community Server 5.6.24
Is there some kind of hidden cache of data stored in memory that I don't know about that gets occupied anytime a MySQL Query gets executed....? That's the only thing I can think of.
This is probably related to the queries you're using.
Indexing can help as well as limiting the results returned from these queries.
I'm guessing you're loading a certain amount of data using SELECT commands and the returned values need to be stored somewhere, which is the memory.
http://dev.mysql.com/doc/refman/5.7/en/out-of-memory.html
This link offers some possible solutions.
To remedy the problem, first check whether your query is correct. Is it reasonable that it should return so many rows? If not, correct the query and try again. Otherwise, you can invoke mysql with the --quick option. This causes it to use the mysql_use_result() C API function to retrieve the result set, which places less of a load on the client (but more on the server).
Also if you have enough RAM you could try to increase the memory limit in the config file.
To be more specific this is what you want to adjust:
InnoDB Settings
innodb_buffer_pool_size - Defaults to 128M. This is the main setting you want to change, since it sets how much memory InnoDB will use for data+indexes loaded into memory. For a dedicated MySQL server, the recommended size is 50-80% of installed memory. So for example, a server with 64G of RAM should have around a 50G buffer pool.
The danger of setting this value too high is that there will be no memory left for the operating system and some MySQL-subsystems that rely on filesystem cache such as binary logs, and InnoDB's transaction logs.
Taken from:
http://www.tocker.ca/2013/09/17/what-to-tune-in-mysql-56-after-installation.html
Another possibility is that something in the c++ code is not allocating the memory and after it's done deallocating the memory properly.
Another thing that could be leaking is connections. Database connections are very expensive, and hang around. I googled "mysql connection pooling". If you're using Connector/J, you could look at http://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-j2ee-concepts-connection-pooling.html, and if you're using connector/net you could try http://dev.mysql.com/doc/connector-net/en/connector-net-programming-connection-pooling.html.
My application creates a lot of "configuration" models (ie- they only live in the app at runtime and they won't ever be persisted). I load these on demand so my app is constantly creating records and then throwing them away.
//create a record that will never be persisted
this.store.createRecord('foo', {name: 'wat'});
In the past I would just do a clear of the store but I realized this doesn't actually "remove" anything. I've decided to use the unloadAll instead
this.store.unloadAll('foo');
... but I run into this error as I have these "configuration" models
Error while loading route: Error: Attempted to handle event
unloadRecord on while in state
root.loaded.created.uncommitted.
at new Error (native)
How can I avoid this error (while still using the unloadAll as I need to truly remove these from the browser) ?
Actually this has now (should be) fixed with my PR which was merged 2 days ago:
see: https://github.com/emberjs/data/pull/1714
That PR loosens the constraint which disallowed unloading all dirty records, to disallowing only inFlight records. I believe with some time and proper thought, that constraint may also be lifted.
The rest of the PR, is specifically around proper cleanup when unloading a model, a record array, or destroying the store. I do believe this is a good first pass at proper cleanup.
I hope this (merged) PR solves your issue, if not please open a descriptive issue, and lets squash the bug.
I'm developing a small in house alternative to Tripwire, so I've coded a small script to hash files in a JBoss EAP server, and store the path and the hash in a MySQL database.
Every day the script compares the hashes in the filesystem with those saved in the DB, so any change is logged and finally reported using JasperServer.
The script runs at night using cron, to avoid a large number of scripts quering the DB at the same time it uses time.sleep(RANDOM_NUMBER_OF_SECONDS) before doing the fun stuff, but sometimes time.sleep seems to sleep forever and the script ends without any error, I check the mail cron sends and no error is logged. Any help would be appreciated. I'm Using jython-standalone-2.5.3, IBM's JDK and RHEL 5.6 running inside VMWare.
I just found http://bugs.jython.org/issue1974 and a code comment seems to point that OS signals can cause this behavior, but not sure if this is my case.
If you want to see the code checkout at http://code.google.com/p/pysnapshot/
Luis García Bustos.
I don't know why do you think time.sleep() can make less number of scripts querying the DB.
IMO ot is better to use cron to call that program periodically. After it is started it should check if in /tmp/ directory is "semaphore" file, for example /tmp/snapshot_working.txt. If there is no semaphore file, then create it and write to it something like: "snapshot started: 2012-12-05 22:00:00". After your program completes checking it should remove this file. If at start program will find semaphore file then it could just stop or check if date & time saved in this file looks "old". If it is "old", then remove it and start normally writing in log that "old" file was found (administrator can find such long working snaphots and terminate it).
The only reason do make time.sleep() in your case is if you want to use such script at normal working hours without making Denial Of Service attack to your DB. Example: after making 100 DB queries you can make little sleep and give DB time to serve other user queries. But I think the sooner program finishes the better.