I have a software application which communicates with a Graphic User Interface using MQTT protocol.
The first software (let's call it app) publish messages and the second software (let's call it gui) subscribes to topics.
App software
The app software is a C++ application which embeds a MQTT client based on mosquittopp.
It sends messages on different topics in order to notify gui software.
Gui software
The gui software is a Qt application which embeds a MQTT client also based on mosquittopp.
When a message is received on a topic, a signal messageReceived is emitted.
The signal messageReceived is connected to a slot Message_received_from_topic.
All messages from all topics are processed into Message_received_from_topic slot.
Presenting the issue
A gui screen is composed of several fields. A field is updated when a message is received on the associated topic.
When gui stays on the same screen, receiving messages through MQTT allow to update fields without reloading the full screen.
However when changing to a screen which contains many fields, there are some glitches that appear because the screen is refreshed before all fields are generated.
In order to avoid those glitches, I want to lock screen refreshing before updating all fields at least once and then unlock refresh.
That's why not receiving MQTT messages in the right order is problematic.
Issue analysis
The issue I'm facing is that messages are not received in order by gui.
Here is a table that present how messages are sent and received.
+----------------------------------------------+----------------------------------------------+----------------------------------------------+
| Sent by **app** | On broker (using mosquitto_sub) | Received by **gui** |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
| Topic | Message | Topic | Message | Topic | Message |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
| widget/refresh | 0 | widget/refresh | 0 | widget/refresh | 0 |
| widget/input/value/9 | {"label":"2020"} | widget/input/value/9 | {"label":"2020"} | widget/refresh | 1 |
| widget/input/value/10 | {"label":"02"} | widget/input/value/10 | {"label":"02"} | widget/input/value/9 | {"label":"2020"} |
| widget/input/value/11 | {"label":"20"} | widget/input/value/11 | {"label":"20"} | widget/input/position/9 | 1 |
| widget/input/value/12 | {"label":"15"} | widget/input/value/12 | {"label":"15"} | widget/input/selection/9 | true |
| widget/input/value/13 | {"label":"06"} | widget/input/value/13 | {"label":"06"} | widget/input/value/10 | {"label":"02"} |
| widget/input/position/12 | 0 | widget/input/position/12 | 0 | widget/input/position/10 | 0 |
| widget/input/selection/12 | false | widget/input/selection/12 | false | widget/input/selection/10 | true |
| widget/input/position/13 | 0 | widget/input/position/13 | 0 | widget/input/value/11 | {"label":"20"} |
| widget/input/selection/13 | false | widget/input/selection/13 | false | widget/input/position/11 | 0 |
| widget/input/position/9 | 0 | widget/input/position/9 | 0 | widget/input/selection/11 | true |
| widget/input/selection/9 | true | widget/input/selection/9 | true | widget/input/value/12 | {"label":"15"} |
| widget/input/position/10 | 0 | widget/input/position/10 | 0 | widget/input/position/12 | 0 |
| widget/input/selection/10 | true | widget/input/selection/10 | true | widget/input/selection/12 | false |
| widget/input/position/11 | 0 | widget/input/position/11 | 0 | widget/input/value/13 | {"label":"06"} |
| widget/input/selection/11 | true | widget/input/selection/11 | true | widget/input/position/13 | 0 |
| widget/input/position/9 | 1 | widget/input/position/9 | 1 | widget/input/selection/13 | false |
| widget/refresh | 1 | widget/refresh | 1 | | |
+---------------------------+------------------+---------------------------+------------------+---------------------------+------------------+
Here messages are received in the wrong order, I'm also missing one message on topic "widget/input/position/9"
All publications and subscriptions are using QoS 2.
Questions
Does anybody knows why messages are not received in the right order by gui when they are correctly received using mosquitto_sub ?
Is there any configuration to apply to mosquitto broker in order to keep message order ?
Related
I am trying to build a data collection pipe-line on top of AWS services. Overal architecture is given below;
In summary system should get events from API gateway (1) ( one request for each event ) and the data should be written to Kinesis (2).
I am expecting ~100k events per second. My question is related to KPL usage on Lambda functions. On step 2 I am planning to write a Lambda method with KPL to write events on Kinesis with high throughput. But I am not sure it is possible as API Gateway calls lambda function for each event separately.
Is it possible/reasonable to use KPL in such architecture or I should using Kinesis Put API instead?
1 2 3 4
+----------------+ +----------------+ +----------------+ +----------------+
| | | | | | | |
| | | | | | | |
| AWS API GW +-----------> | AWS Lambda +-----------> | AWS Kinesis +----------> | AWS Lambda |
| | | Function with | | Streams | | |
| | | KPL | | | | |
| | | | | | | |
+----------------+ +----------------+ +----------------+ +-----+-----+----+
| |
| |
| |
| |
| |
5 | | 6
+----------------+ | | +----------------+
| | | | | |
| | | | | |
| AWS S3 <-------+ +----> | AWS Redshift |
| | | |
| | | |
| | | |
+----------------+ +----------------+
I am also thinking about writing directly to S3 instead of calling lambda function from api-gw. If first architecture is not reasonable this may be a solution but in that case I will have a delay till writing data to kinesis
1 2 3 4 5
+----------------+ +----------------+ +----------------+ +----------------+ +----------------+
| | | | | | | | | |
| | | | | | | | | |
| AWS API GW +-----------> | AWS Lambda +------> | AWS Lambda +-----------> | AWS Kinesis +----------> | AWS Lambda |
| | | to write data | | Function with | | Streams | | |
| | | to S3 | | KPL | | | | |
| | | | | | | | | |
+----------------+ +----------------+ +----------------+ +----------------+ +-----+-----+----+
| |
| |
| |
| |
| |
6 | | 7
+----------------+ | | +----------------+
| | | | | |
| | | | | |
I do not think using KPL is the right choice here. The key concept of KPL is, that records get collected at the client and then send as a batch operation to Kinesis. Since Lambdas are stateless per invocation, it would be rather difficult to store the records for aggregation (before sending it to Kinesis).
I think you should have a look at the following AWS article which explain how you can directly connect API-Gateway to Kinesis. This way, you can avoid the extra Lambda which just forwards your request.
Create an API Gateway API as an Kinesis Proxy
Obviously, if your data coming through AWS API Gateway corresponds to one Kinesis Data Streams record it makes no sense to use the KPL as pointed out by Jens. In this case you can make direct call of Kinesis API without using Lambda. Eventually, you may use some additional processing in Lambda and send the data through PutRecord (not PutRecords used by KPL). Your code in JAVA will looks like this
AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard();
clientBuilder.setRegion(REGION);
clientBuilder.setCredentials(new DefaultAWSCredentialsProviderChain());
clientBuilder.setClientConfiguration(new ClientConfiguration());
AmazonKinesis kinesisClient = clientBuilder.build();
...
//then later on each record
PutRecordRequest putRecordRequest = new PutRecordRequest();
putRecordRequest.setStreamName(STREAM_NAME);
putRecordRequest.setData(data);
putRecordRequest.setPartitionKey(daasEvent.getAnonymizedId());
putRecordRequest.setExplicitHashKey(Utils.randomExplicitHashKey());
putRecordRequest.setSequenceNumberForOrdering(sequenceNumberOfPreviousRecord);
PutRecordResult putRecordResult = kinesisClient.putRecord(putRecordRequest);
sequenceNumberOfPreviousRecord = putRecordResult.getSequenceNumber();
However, there may be cases when using KPL from lambda makes sense. For example the data sent to AWS API Gateway contains multiple individual records which will be sent to one or multiple streams. In that cases the benefits (see https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html) of KPL are still valid, but you have to be aware of specifics given by using of Lambda concretely an "issue" pointed out here https://github.com/awslabs/amazon-kinesis-producer/issues/143 and use
kinesisProducer.flushSync()
at the end of insertions which worked also for me.
I am new to django and SQL queries. I am trying some annotation with django. but unable to get results
+-----------------------+-----------+---------------------+
| email | event | event_date |
|-----------------------+-----------+---------------------|
| hector#example.com | open | 2017-01-03 13:26:13 |
| hector#example.com | delivered | 2017-01-03 13:26:28 |
| hector#example.com | open | 2017-01-03 13:26:33 |
| hector#example.com | open | 2017-01-03 13:26:33 |
| tornedo#example.com | open | 2017-01-03 13:34:53 |
| tornedo#example.com | 1 | 2017-01-03 13:35:22 |
| tornedo#example.com | open | 2016-09-05 00:00:00 |
| tornedo#example.com | open | 2016-09-17 00:00:00 |
| sparrow#example.com | open | 2017-01-03 16:05:36 |
| tornedo#example.com | open | 2017-01-03 20:12:15 |
| hector#example.com | open | 2017-01-03 22:06:47 |
| sparrow#example.com | open | 2017-01-09 19:46:26 |
| sparrow#example.com | open | 2017-01-09 19:47:59 |
| sparrow#example.com | open | 2017-01-09 19:48:28 |
| sparrow#example.com | delivered | 2017-01-09 19:52:24 |
+-----------------------+-----------+---------------------+
I have a table like this which contains email activity. I want to find who opened recently and also i want to count of each event happened. I want results exactly like
email | open | click | delivered | max_open_date
hector#example.com 4 <null> 1 2017-01-03 22:06:47
sparrow#example.com 3 <null> 1 2017-01-09 19:48:28
tornedo#example.com 4 1 <null> 2017-01-03 20:12:15
my model looks:
class EmailEvent(models.Model):
event = models.TextField(blank=True, null=True)
email = models.TextField(blank=True, null=True)
event_date = models.DateTimeField(blank=True, null=True)
i tried the following code. it giving correct count for open, click, delivered but giving wrong result for max_open_date. but i don't know why
EmailEvent.objects.values('email').annotate(
max_open_date=Case(When(event='open', then=Max('event_date')))),
open=Sum(Case(When(event='open',then=1),output_field=IntegerField())),
click=Sum(Case(When(event='click',then=1),output_field=IntegerField())),
open=Sum(Case(When(event='open',then=1),output_field=IntegerField())),
delivered=Sum(Case(When(event='delivered',then=1),output_field=IntegerField())),
)
Help me to get exact results i want. sorry for my bad english. Thanks!
I do not use django, but probably you need something like this:
max_open_date=Max(Case(When(event='open', then='event_date')))
My Qt app's context menu is displayed in the incorrect position when using multiple monitors on Gnome 3.
It would seem that perhaps the culprit here is Gnome Shell, rather than Qt itself, as I can't replicate the issue described below with Ubuntu Unity, it only happens when running Ubuntu Gnome 14.04.
Symptoms:
I am using QWidget::mapToGlobal(QPoint) from a QWidget::customContextMenuRequested signal in order to find the correct position to display a context menu.
I am then using QMenu::exec(QPoint) to display the menu in that position
void Window::slotOnShowContextMenu(const QPoint& p)
{
_menu->exec(_tree->viewport()->mapToGlobal(p));
}
My problem is that I have the following screen layout:
When my window is on the right hand screen, or on the left hand screen but at a position below the top of the right hand screen, the context menu is shown correctly:
When my window is on the left hand screen, at a level above the top of the right hand screen, even though the Y value of the QPoint returned from mapToGlobal is correct, the context menu is not displayed at that point, but is rather constrained to be at the same level as the right hand screen.
I have confirmed that _tree->viewport()->mapToGlobal(p) returns the correct results (just by logging the resulting QPoint)
QPoint point = _tree->viewport()->mapToGlobal(p);
std::cout << point.x() << ":" << point.y() << '\n';
It would therefore seem that QMenu::exec(QPoint) is the culprit?
How can I correctly display my context menu?
Edit:
I tried running the same app on standard Ubuntu 14.04 (ie: using Unity instead of Gnome), and the incorrect behaviour doesn't present itself, so this would seem to be a Gnome 3 issue?
I have tried changing my primary monitor so that the portrait monitor on the left is primary, and the context menu displays correctly.
Note black launch bar is on the left screen
When using the following layout (primary screen is landscape on the right) the context menu position is constrained to be the top of the primary monitor.
Note black launch bar is on the right screen
So it would seem that the primary monitor's top position is the maximum height that Qt will display it's context menu?
While the technical details may be slightly different, I found with elementary os and two monitors (a laptop and a smart tv), with my built-in display positioned on the right of the external display, the menu is invisible if I have the Qt app (KeePassXC in my case) on my built-in display. If I move it to my external display, the menus are at the top of the display, instead of in the window. There is clearly a bug somewhere, whether in Qt or ubuntu or gnome shell, I can't say.
I can say that switching my displays back to the default positioning, with the built-in on the left of external, the issue is resolved: I can finally access the menus where they should be.
I have tried moving the window location around, and as long as I have my built-in display situated to the left of my external display, it doesn't matter where the window is, it just works like it should.
This solution may not be ideal if your monitors are not physically positioned that way, but in my case, I'm already having to lie about where my monitors are positioned. My built-in is below my external, but when I try to tell elementary that, it stops functioning correctly (all windows are moved to outside the bounds of the screen, all windows launch off screen; deleting .config/monitors.xml fixes it, but I have to be able to get to a command prompt on the screen to do it.)
I ran into this same issue. As I dug into it I realized that it's a conflict between the perceived/"logical" monitor location and the rendering of the screen. In my case, I can see through the output of xrandr and the configuration of ~/.config/monitors.xml that my right most monitor (the equivalent of your Hewlett Packard 27" monitor) that it has a position offset of +360:
$ xrandr
Screen 0: minimum 320 x 200, current 7040 x 1440, maximum 16384 x 16384
eDP-1 connected primary 1920x1080+5120+360 (normal left inverted right x axis y axis) 309mm x 173mm
...
These 360 pixels correspond to the top of the window's "location" as understood by QT.
In my case the menu bar itself is 25 pixels high. Keeping all this in mind(360 + height of title bar + height of menu) actual offset of where perceived drawing of the menu.
+---------+---------------------------------------------+ ^
| | | |
| +-------------------------------------^-------------+ | |
| | | | 25 pixels | | | 360 pixels
| +-------------------------------------v-------------+ | |
| | | | | v
| | | 385 pixels | +---+---------------------------+
| | | | | |
| | +----v+ | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | +-----+ | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| +---------------------------------------------------+ | |
| | |
| | |
| | |
| | |
+-------------------------------------------------------+-------------------------------+
When I re-aligned me screens so that the position offset was zero like in the following setup the problem went away:
$ xrandr
Screen 0: minimum 320 x 200, current 7040 x 1440, maximum 16384 x 16384
eDP-1 connected primary 1920x1080+5120+0 (normal left inverted right x axis y axis) 309mm x 173mm
...
In this case the 360 pixel position offset is now zero and QT renders the menu drop down at the correct location:
+-------------------------------------------------------+-------------------------------+
| | |
| +-------------------------------------^-------------+ | |
| | | 25 pixels | | |
| +--+-----+----------------------------v-------------+ | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | +-----+ | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | +-------------------------------+
| +---------------------------------------------------+ |
| |
| |
| |
| |
+--------------------------------------------------------
I was in the process of filing a bug with QT about this (as I have multiple applications which are affected by this bug), but in the process of collecting the relevant information for the bugs I discovered that it's not just QT/QT5 affected but also Blender. As Blender does not use a graphics framework (e.g. QT/QT5, GTK+, etc) this is almost certainly a bug in one of the components of GNOME3.
I'm running into this with qt applications in fvwm if:
I have a monitor configuration with deadspace
and if qt menu animations are enabled
The menus end up off-screen and not visible at all or in completely the wrong place (depending on the monitor offsets) -
Thus I'm not so sure this is just a gnome shell bug.
I'm using Doctrine 2 with my ZF2 project, but i'm getting some weird problem with my server CPU and memory. And my server simply crashes.
I'm getting a lot of sleep state querys and they seem not to get cleaned.
mysql> show processlist;
+---------+--------------+-----------+------------------+----------------+------+--------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+--------------+-----------+------------------+----------------+------+--------------------+------------------------------------------------------------------------------------------------------+
| 2832346 | leechprotect | localhost | leechprotect | Sleep | 197 | | NULL |
| 2832629 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832643 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832646 | db_user | localhost | db_exemple | Sleep | 3 | | NULL |
| 2832664 | db_user | localhost | db_exemple | Sleep | 154 | | NULL |
| 2832666 | db_user | localhost | db_exemple | Sleep | 153 | | NULL |
| 2832669 | db_user | localhost | db_exemple | Sleep | 152 | | NULL |
| 2832674 | db_user | localhost | db_exemple | Sleep | 7 | | NULL |
| 2832681 | db_user | localhost | db_exemple | Sleep | 1 | | NULL |
| 2832683 | db_user | localhost | db_exemple | Sleep | 4 | | NULL |
| 2832690 | db_user | localhost | db_exemple | Sleep | 149 | | NULL |
(.......)
Also, it seems php GC is not cleaning all the objects from memory, or even kill processes.
Is there a way to disable the cache system? Would it improve the use of my resorces=
Most my querys are similar to:
$query = $this->createQueryBuilder('i');
$query->innerJoin('\Application\Relation', 'r', 'WITH', 'r.child = i.id');
$query->innerJoin('\Application\Taxonomy', 't', 'WITH', 't.id = r.taxonomy');
$query->where('t.type = :type')->setParameter('type', $relation);
$query->groupBy('i.id');
$items = $query->getQuery()->getResult(2);
Thanks in advance.
Firstly check the mysql's wait_timout variable. From the documentation:
Wait_timeout : The number of seconds the server waits for activity on
a noninteractive connection before closing it.
In normal flow (which not using persistent connections), php closes the connection automatically after script execution. To ensure there are no sleeping threads; at the end of your script simply close the connection:
$entityManager->getConnection()->close();
If these queries are running in a big while/for loop, you might want to read doctrine 2 batch processing documentation.
I'm using WTL to create a window containing many splitted panes. The following will be the result.
---------------------------
| | | |
| | | |
| | | |
| |--------------------
| | | |
| | | |
---------------------------
There will be 4 splitters, three vertical ones and a horizontal one.
I followed the great article : http://www.codeproject.com/KB/wtl/wtl4mfc7.aspx.
But I only can add 3 splitters as below.
---------------------------
| | | |
| | | |
| | | |
| |--------------------
| | |
| | |
---------------------------
I tried a lot of ways but still cannot add the last one.
Is is a bug of WTL? Can anybody help me?
Best regards,
Zach#Shine
What is your problem? Is it a compile error, a runtime ASSERT, something else?
I strongly suggest that you derive your CMainFrame from CSplitFrameWindowImpl<>.
---------------------------
| | | |
| | 2TL | 2TR |
| 1L | | |
| |--------------------
| | 2BL | 2BR |
| | | |
---------------------------
The right pane (including all '2' panes) should derive from CSplitterWindowImpl<CPane2, false>, the right top pane (including all '2T' panes) should derive from CSplitterWindowImpl<CPane2T, true> as well as the right bottom one.
Each split pane should be created in the OnCreate() handler of it's parent and create it's children in it's own OnCreate().