Doctrine 2 memory hogging - doctrine-orm

I'm using Doctrine 2 with my ZF2 project, but i'm getting some weird problem with my server CPU and memory. And my server simply crashes.
I'm getting a lot of sleep state querys and they seem not to get cleaned.
mysql> show processlist;
+---------+--------------+-----------+------------------+----------------+------+--------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+--------------+-----------+------------------+----------------+------+--------------------+------------------------------------------------------------------------------------------------------+
| 2832346 | leechprotect | localhost | leechprotect | Sleep | 197 | | NULL |
| 2832629 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832643 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832646 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832664 | db_user | localhost | db_exemple | Sleep | 154 | | NULL |
| 2832666 | db_user | localhost | db_exemple | Sleep | 153 | | NULL |
| 2832669 | db_user | localhost | db_exemple | Sleep | 152 | | NULL |
| 2832674 | db_user | localhost | db_exemple | Sleep | 7 | | NULL |
| 2832681 | db_user | localhost | db_exemple | Sleep | 1 | | NULL |
| 2832683 | db_user | localhost | db_exemple | Sleep | 4 | | NULL |
| 2832690 | db_user | localhost | db_exemple | Sleep | 149 | | NULL |
(.......)
Also, it seems php GC is not cleaning all the objects from memory, or even kill processes.
Is there a way to disable the cache system? Would it improve the use of my resorces=
Most my querys are similar to:
$query = $this->createQueryBuilder('i');
$query->innerJoin('\Application\Relation', 'r', 'WITH', 'r.child = i.id');
$query->innerJoin('\Application\Taxonomy', 't', 'WITH', 't.id = r.taxonomy');
$query->where('t.type = :type')->setParameter('type', $relation);
$query->groupBy('i.id');
$items = $query->getQuery()->getResult(2);
Thanks in advance.

Firstly check the mysql's wait_timout variable. From the documentation:
Wait_timeout : The number of seconds the server waits for activity on
a noninteractive connection before closing it.
In normal flow (which not using persistent connections), php closes the connection automatically after script execution. To ensure there are no sleeping threads; at the end of your script simply close the connection:
$entityManager->getConnection()->close();
If these queries are running in a big while/for loop, you might want to read doctrine 2 batch processing documentation.

Related

Keep message ordering on MQTT topics

I have a software application which communicates with a Graphic User Interface using MQTT protocol.
The first software (let's call it app) publish messages and the second software (let's call it gui) subscribes to topics.
App software
The app software is a C++ application which embeds a MQTT client based on mosquittopp.
It sends messages on different topics in order to notify gui software.
Gui software
The gui software is a Qt application which embeds a MQTT client also based on mosquittopp.
When a message is received on a topic, a signal messageReceived is emitted.
The signal messageReceived is connected to a slot Message_received_from_topic.
All messages from all topics are processed into Message_received_from_topic slot.
Presenting the issue
A gui screen is composed of several fields. A field is updated when a message is received on the associated topic.
When gui stays on the same screen, receiving messages through MQTT allow to update fields without reloading the full screen.
However when changing to a screen which contains many fields, there are some glitches that appear because the screen is refreshed before all fields are generated.
In order to avoid those glitches, I want to lock screen refreshing before updating all fields at least once and then unlock refresh.
That's why not receiving MQTT messages in the right order is problematic.
Issue analysis
The issue I'm facing is that messages are not received in order by gui.
Here is a table that present how messages are sent and received.
+----------------------------------------------+----------------------------------------------+----------------------------------------------+
| Sent by **app** | On broker (using mosquitto_sub) | Received by **gui** |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
| Topic | Message | Topic | Message | Topic | Message |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
| widget/refresh | 0 | widget/refresh | 0 | widget/refresh | 0 |
| widget/input/value/9 | {"label":"2020"} | widget/input/value/9 | {"label":"2020"} | widget/refresh | 1 |
| widget/input/value/10 | {"label":"02"} | widget/input/value/10 | {"label":"02"} | widget/input/value/9 | {"label":"2020"} |
| widget/input/value/11 | {"label":"20"} | widget/input/value/11 | {"label":"20"} | widget/input/position/9 | 1 |
| widget/input/value/12 | {"label":"15"} | widget/input/value/12 | {"label":"15"} | widget/input/selection/9 | true |
| widget/input/value/13 | {"label":"06"} | widget/input/value/13 | {"label":"06"} | widget/input/value/10 | {"label":"02"} |
| widget/input/position/12 | 0 | widget/input/position/12 | 0 | widget/input/position/10 | 0 |
| widget/input/selection/12 | false | widget/input/selection/12 | false | widget/input/selection/10 | true |
| widget/input/position/13 | 0 | widget/input/position/13 | 0 | widget/input/value/11 | {"label":"20"} |
| widget/input/selection/13 | false | widget/input/selection/13 | false | widget/input/position/11 | 0 |
| widget/input/position/9 | 0 | widget/input/position/9 | 0 | widget/input/selection/11 | true |
| widget/input/selection/9 | true | widget/input/selection/9 | true | widget/input/value/12 | {"label":"15"} |
| widget/input/position/10 | 0 | widget/input/position/10 | 0 | widget/input/position/12 | 0 |
| widget/input/selection/10 | true | widget/input/selection/10 | true | widget/input/selection/12 | false |
| widget/input/position/11 | 0 | widget/input/position/11 | 0 | widget/input/value/13 | {"label":"06"} |
| widget/input/selection/11 | true | widget/input/selection/11 | true | widget/input/position/13 | 0 |
| widget/input/position/9 | 1 | widget/input/position/9 | 1 | widget/input/selection/13 | false |
| widget/refresh | 1 | widget/refresh | 1 | | |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
Here messages are received in the wrong order, I'm also missing one message on topic "widget/input/position/9"
All publications and subscriptions are using QoS 2.
Questions
Does anybody knows why messages are not received in the right order by gui when they are correctly received using mosquitto_sub ?
Is there any configuration to apply to mosquitto broker in order to keep message order ?

Cannot dump Django database using pg_dump due to permissions

I want to dump PostgreSQL database to an SQL file. Here is the pg_dump DB_NAME > dump.sql command result:
pg_dump: [archiver (db)] query failed: ERROR: permission denied for relation django_migrations
pg_dump: [archiver (db)] query was: LOCK TABLE public.django_migrations IN ACCESS SHARE MODE
When I want to list relations:
postgres=# \d
No relations found.
List of schemas:
List of schemas
Name | Owner | Access privileges | Description
--------+----------+----------------------+------------------------
public | postgres | postgres=UC/postgres+| standard public schema
| | =UC/postgres +|
| | root=U/postgres +|
| | secretusr=U/postgres |
(1 row)
Database list:
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+------------------------
db_secret | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres +
| | | | | secretusr=CTc/postgres+
| | | | | root=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Where is the problem?

Django model get unique foreign key rows based on other column max value

I'm new to Django, and I'm trying to create a simple application that keep track on different server configurations in a SQlite database. I've created 2 database models:
from django.db import models
class Server(models.Model):
name = models.CharField(max_length=250)
class Config(models.Model):
server = models.ForeignKey(Server, on_delete=models.CASCADE)
configuration = models.CharField(max_length=250)
config_version = models.IntegerField()
Here are the 2 models sample data:
Server:
| id | name |
| ------ | ------ |
| 1 | Server1 |
| 2 | Server2 |
| 3 | Server3 |
Config:
| id | configuration | config_version | server |
| ------ | ------------- | -------------- | ------ |
| 1 | srv1_cfg1 | 1 | 1 |
| 2 | srv2_cfg1 | 1 | 2 |
| 3 | srv2_cfg2 | 2 | 2 |
| 4 | srv2_cfg3 | 3 | 2 |
| 5 | srv3_cfg1 | 1 | 3 |
| 6 | srv1_cfg2 | 2 | 1 |
| 7 | srv1_cfg3 | 3 | 1 |
I would like to query the Config table, and get only rows with the maximum value of "config_version" field for each server id, like:
Desired result:
| id | configuration | config_version | serverid | servername |
| ------ | ------------- | -------------- | -------- | ---------- |
| 4 | srv2_cfg3 | 3 | 2 | Server2 |
| 5 | srv3_cfg1 | 1 | 3 | Server3 |
| 7 | srv1_cfg3 | 3 | 1 | Server1 |
I've tried many different options to construct the correct query, but so far I cannot get what I want. My best result is to query the Server table:
Server.objects.annotate(maxver=Max('config__config_version'))
But it seems I cannot get access to the Config table objects, so I guess I need to query the Config table with some filtering?
I can do this with a raw SQL query, but I would strongly prefer to do it the "Django" way. Any help will be much appreciated.
After some more struggle with this, I've came with a solution that works for me. I'm sure it is not optimal, but at least seems to works:
from django.db.models import Max, F
s1 = Config.objects.annotate(maxver=Max('server__config__config_version'))
config_list = s1.filter(config_version=F('maxver'))
If there is a better way to do this, I would love to know it.

max value in annotate with condition in django

I am new to django and SQL queries. I am trying some annotation with django. but unable to get results
+-----------------------+-----------+---------------------+
| email | event | event_date |
|-----------------------+-----------+---------------------|
| hector#example.com | open | 2017-01-03 13:26:13 |
| hector#example.com | delivered | 2017-01-03 13:26:28 |
| hector#example.com | open | 2017-01-03 13:26:33 |
| hector#example.com | open | 2017-01-03 13:26:33 |
| tornedo#example.com | open | 2017-01-03 13:34:53 |
| tornedo#example.com | 1 | 2017-01-03 13:35:22 |
| tornedo#example.com | open | 2016-09-05 00:00:00 |
| tornedo#example.com | open | 2016-09-17 00:00:00 |
| sparrow#example.com | open | 2017-01-03 16:05:36 |
| tornedo#example.com | open | 2017-01-03 20:12:15 |
| hector#example.com | open | 2017-01-03 22:06:47 |
| sparrow#example.com | open | 2017-01-09 19:46:26 |
| sparrow#example.com | open | 2017-01-09 19:47:59 |
| sparrow#example.com | open | 2017-01-09 19:48:28 |
| sparrow#example.com | delivered | 2017-01-09 19:52:24 |
+-----------------------+-----------+---------------------+
I have a table like this which contains email activity. I want to find who opened recently and also i want to count of each event happened. I want results exactly like
email | open | click | delivered | max_open_date
hector#example.com 4 <null> 1 2017-01-03 22:06:47
sparrow#example.com 3 <null> 1 2017-01-09 19:48:28
tornedo#example.com 4 1 <null> 2017-01-03 20:12:15
my model looks:
class EmailEvent(models.Model):
event = models.TextField(blank=True, null=True)
email = models.TextField(blank=True, null=True)
event_date = models.DateTimeField(blank=True, null=True)
i tried the following code. it giving correct count for open, click, delivered but giving wrong result for max_open_date. but i don't know why
EmailEvent.objects.values('email').annotate(
max_open_date=Case(When(event='open', then=Max('event_date')))),
open=Sum(Case(When(event='open',then=1),output_field=IntegerField())),
click=Sum(Case(When(event='click',then=1),output_field=IntegerField())),
open=Sum(Case(When(event='open',then=1),output_field=IntegerField())),
delivered=Sum(Case(When(event='delivered',then=1),output_field=IntegerField())),
)
Help me to get exact results i want. sorry for my bad english. Thanks!
I do not use django, but probably you need something like this:
max_open_date=Max(Case(When(event='open', then='event_date')))

unlock local Cloud Foundry user. uaac?

Howto unlock local Cloud Foundry user. Didn't found an option on uaac Gem?
Our users use CI/CD tools with wrong passwords and then their CF users get locked.
By default, UAA does not permanently lock accounts after failed attempts. It will cause an account to be unavailable for a period of time (5 mins default) and will continue to lock for 5 mins until a successful login occurs. If you have not changed the default configuration, just wait 5 mins and try with the correct credentials.
Use uaadb
select * from sec_audit;
+--------------------------------------+------------+----------------------------+----------------+---------------------+------------------+
| principal_id | event_type | origin | event_data | created | identity_zone_id |
+--------------------------------------+------------+----------------------------+----------------+---------------------+------------------+
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 10:51:47 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 11:46:34 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 11:46:47 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 11:46:56 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 11:47:01 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 12:51:47 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 13:46:34 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 13:46:47 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 13:46:56 | uaa |
| e38af5c2-ae71-406e-bfd5-2df4f8891cf7 | 1 | remoteAddress=192.168.35.27 | admin-opr | 2015-11-11 13:47:01 | uaa |
+--------------------------------------+------------+----------------------------+----------------+---------------------+------------------+
Truncate this table and all locks are away