REPEAT With FRAME:
prompt-for IN-SCAN3.scan.
if input IN-SCAN3.scan="" then Do:
Message "please input date.".
undo,retry.
end.
else DO:
FIND FIRST in-scan3 USING IN-SCAN3.scan NO-LOCK NO-WAIT NO-ERROR.
if avail In-scan3 then DO:
str="OK".
display str.
next-prompt IN-SCAN3.scan.
end.
else DO:
CREATE In-scan3.
ASSIGN IN-scan3.scan=INPUT in-scan3.scan.
str="NO". DISPLAY str.
next-prompt In-scan3.scan.
END.
end.
begin=begin + 1.
end.
Question desc:
There are 20 users using scanning at the same time,first find input data, if not found, then create one record in the database. The question is, at the same time operating will appear dead lock.
I try NO-LOCK NO-WAIT with record when find,operating will appear dead lock when create a record.
thanks any answer.
You have unfortunately fallen foul of ABL's basic "gotcha" -- record locking will default to SHARE-LOCK even if you specify otherwise, and this is probably not what you want.
The basic rule is that if your transaction scope is smaller than your record scope then the record will drop back to a SHARE-LOCK when you leave the transaction. But I recommend you read the relevant chapter in the ABL guide.
There are a number of ways to fix this. The RELEASE keyword is one. But I tend to favour the idea that you should use a separate buffer for actually locking the record. That way you make things very clear to other programmers.
For example:
def buffer b-in-scan3 for in-scan3.
repeat:
prompt-for in-scan3.scan.
/*** etc. ***/
else
do for b-in-scan3 transaction:
find first b-in-scan3 using in-scan3.scan
exclusive no-wait no-error.
if not avail b-in-scan3
and not locked b-in-scan3
then
do:
create b-in-scan3.
b-in-scan3.scan = input in-scan3.scan.
end.
end. /* of transaction */
end. /* of repeat */
This way if another programmer uses b-inscan3 outside of your transaction block, the program will fail to compile rather than start dropping back to SHARE-LOCK.
Remember that you can check LOCKED as well as AVAILABLE, but if a record is locked then it's not available.
Related
UPDATE: I had each node write to a separate file, and when the separate files were concatenated together the result was correct. I also updated the code to attempt a channel flush and file sync after each write of a single record, but there are still issues between nodes 0 and 1, now. If I make Node 0 sleep for a few seconds before it starts its iteration of the coforall loop, the records come out correct. If not, the last few hundred bytes of Node 0's records seem to be reliably overwritten with NULL bytes, up to the start of Node 1's records. The issues between Node 1 and Node 2, and Node 2 and Node 3, seem to not show up anymore.
Additionally, if I suppress either Node 0 or Node 1 from writing, I see the fully-formed records from the un-suppressed node written correctly to the file. In the case that Node 1 is suppressed, I see 9,997 100B records (or 999,700) correct bytes followed by NULL bytes in the file where Node 1's suppressed records would go. In the case that Node 0 is suppressed, I see exactly 999,700 NULL bytes in the file, after which Node 1's records begin.
Original Post:
I'm trying to troubleshoot an issue with parallel writes from different nodes to a shared NFS-backed file on disk. At the moment, I suspect that something is wrong with the way writes to the disk happen on the NFS server.
I'm working on adapting MPI+C code that uses pwrite to write to coordinated chunks of a file. If I try to have the equivalent locales in Chapel write to the file inside of a coforall loop, I end up with the bits of the file around the node boundaries messed up - usually the final few hundred bytes of each node's data are garbled. However, if I have just one locale iterate through the data on all locales and write it, the data comes out correctly. That is, I use the same data structures to calculate the offsets, but only Locale 0 seeks to that offset and performs the writes.
I've verified that the offsets into the file that each locale runs do not overlap, and I'm using a single channel per task, defined from within the on loc do block, so that tasks don't share a single channel.
Are there known issues with writing to a file from different locales? A lot of the documentation makes it seem like this is known to be safe, but an unsubstantiated guess seems to indicate that there are issues with caching of file contents; when examining the incorrect data, the bits that are incorrect seem to be the original data from the file in that location at the beginning of the program.
I've included the relevant routine below, in case you easily spot something I missed. To make this serial, I convert the coforall loc in Locales and on loc do block into a for j in 0..numLocales-1 loop, and replace here.id with j. Please let me know what else would help get to the bottom of this. Thanks!
proc write_share_of_data(data_filename: string, ref record_blocks) throws {
coforall loc in Locales {
on loc do {
var data_file: file = open(data_filename, iomode.cwr);
var data_writer = data_file.writer(kind=ionative, locking=false);
var line: [1..100] uint(8);
const indices = record_blocks[here.id].D;
var local_record_offset = + reduce record_blocks[0..here.id-1].D.size;
writeln("Loc ", here.id, ": record offset is ", local_record_offset);
var local_bytes_offset = terarec.terarec_width_disk * local_record_offset;
data_writer.seek(start=local_bytes_offset);
for i in indices {
var write_rec: terarec_t = record_blocks[here.id].records[i];
line[1..10] = write_rec.key;
line[11..98] = write_rec.value;
line[99] = 13; // \r
line[100] = 10; // \n
data_writer.write(line);
lines_written += 1;
}
data_file.fsync();
data_writer.close();
data_file.close();
}
}
return;
}
Adding an answer here that solved my particular problem, though it doesn't explain the behavior seen. I ended up changing the outer loop from coforall loc in Locales to for loc in Locales. This isn't too big of an issue since it's all writing to one file anyway - I doubt that multiple locales can actually make much headway in all attempting to write concurrently to a single file on an NFS server. As a result, the change still allows nodes to write the data they have locally to NFS, rather than forcing Node 0 to collect and then write the data on behalf of all locales. This amounts to only adding idle time to the write operation commensurate with the time it takes Locale 0 to start the remote task on other nodes when the previous node has finished writing, which for the application at hand is not a concern.
Have you tried specifying start/end in file.writer instead of using seek? Does that change anything? What about specifying the end offset for the channel.seek call? Does it matter if the file is created and has the appropriate size before you start?
Other than that, I wonder if this issue would appear for both NFS and Lustre. If it appears for both it might well be a Chapel bug. It sounds from your description that the C program was using this pattern, which points to it being a bug. But, have you run C code doing this on your setup? If it being a Chapel bug seems most likely after further investigation, we would appreciate a bug report issue with a reproducer.
I know that NFS does not always do what one would like, in terms of data consistency. It's my understanding that it has "close to open" semantics but it's unclear to me what that means in the context of opening a file and writing to a particular region within it, in parallel from different locales.
From Why NFS Sucks by Olaf Kirch:
An NFS client is permitted to cache changes locally and send them to
the server whenever it sees fit. This sort of lazy write-back greatly
helps write performance, but the flip side is that everyone else will
be blissfully unaware of these change before they hit the server. To
make things just a little harder, there is also no requirement for a
client to transmit its cached write in any particular fashion, so
dirty pages can (and often will be) written out in random order.
I read two implications from this paragraph that are relevant to your situation here:
The writes you do on different locales can be observed by the NFS server in an arbitrary order. (However as I understand it, the data should be sent to the server by the time your fsync call returns).
These writes are done at an OS page granularity (usually 4k). (Note that this is more a hypothesis I am making than it is a fact. It should be tested or further investigated).
It would be interesting to check if 2. is a plausible explanation for the behavior you are seeing. For example, you could explore having each locale operate on a multiple of 4096 records (or potentially try writing records of 4096 bytes each) and see if that changes the behavior. If 2 is indeed the explanation, it should be possible to create a C program that demonstrates the behavior as well.
I have a LTTNg trace, which i am parsing using babeltrace API. So i was wondering if I could count all events in trace (or stream) without iterating over them. What functions from publilc API I can use to do that ?
The very nature of CTF makes it impossible to count the event records of a given packet in constant time. The packet's context could include an event record count field somehow, but it's not specified, so generic tools would not use it.
Thus the only way to count events is to iterate the event records, unfortunately. The easiest way is to count the number of lines that the text format of the babeltrace(1) tool prints:
babeltrace /path/to/ctf/trace/directory | wc --lines
This works as long as there's one line per printed event record, which is the case unless an event record contains a string field which has a newline (currently not escaped in the text output).
You may also wish to consider discarded event records. They are not printed to the standard output by babeltrace(1), but the tool prints a message including the count to the standard error when they are detected.
There's no way with the current babeltrace(1) tool to only print the event records which belong to the packets of a given data stream. If you need this, what I suggest is that you remove all the data stream files except the one for which you need an event record count, and run the command above again.
Also consider the Babeltrace Python bindings, for example (not tested):
import babeltrace
def count_ctf_event_records(path):
trace_collection = babeltrace.TraceCollection()
trace_collection.add_trace(path, 'ctf')
return sum(1 for event in trace_collection.events)
if __name__ == '__main__':
import sys
print(count_ctf_event_records(sys.argv[1]))
Saved as count.py, you can try this:
python3 count.py /path/to/ctf/trace/directory
Counting the event records of a specific data stream with the Python bindings is left as an exercise for the reader.
Having said this, I don't know if the Python bindings approach is faster than the babeltrace(1) one.
for example, that is data:
1,1470732420000,0
2,1470732421000,0
3,1470732422000,0
4,1470732423000,86
5,1470732424000,87
6,1470732425000,88
7,1470732426000,84
8,1470732427000,0
9,1470732428000,0
10,1470732429000,0
11,1470732430000,89
12,1470732431000,89
13,1470732432000,87
14,1470732433000,89
15,1470732434000,85
16,1470732435000,89
17,1470732436000,89
18,1470732437000,87
19,1470732438000,86
20,1470732439000,88
21,1470732440000,0
22,1470732441000,0
23,1470732442000,0
24,1470732443000,87
25,1470732444000,85
26,1470732445000,86
27,1470732446000,0
28,1470732447000,0
29,1470732448000,0
30,1470732449000,0
column one is id,column two is timestamp,column three is value,1 sec interval between the timestamp.
i want monitoring the value of event,if i found out value>=85(e.g. id=4), i will starting counting,if the next two consecutive value>=85(e.g. id=5/id=6),then i will put the third value of event to OutputStream.(e.g. id=6,value=88,timestamp=1470732425000)
at the same time i clear the counting and wait value lower than 85(e.g. id=7,value=84), then i will monitoring again,when i found out value>=85(e.g. id=11,value=89) i will starting counting,if the next two consecutive value>=85(e.g. id=12/id=13),then i will put the third value of event to OutputStream.(e.g. id=13,value=87,timestamp=1470732432000)...
all this is i wanna do,before i post this ask, i've got an answer in this post,i've tried this code:
from every a1=InputStream[value>=85], a2=InputStream[value>=85]+, a3=InputStream[value<85]
select a2[1].id, a2[1].value
having (not (a2[1] is null))
insert into OutPutStream;
and it works,but i found out it will insert the value into OutputStream after the value<=85,and what i want is if i got three consecutive value>=85 then i insert into the value immediately.(i don't want to wait if the next value>=85 all the times)
in fact, i just wanna record value of third seconds in three consecutive seconds value(>=85) .
i'm using wso2das-3.1.0-SNAPSHOT.
Though DAS (Siddhi) supports sequence/pattern processing, for your requirement you might need to write a custom extension. I have written a sample window processor extension to cater your requirement (source code). Download and place siddhi-extension-condition-window-1.0.jar in <das_home>/repository/components/lib/ directory and restart the server. Refer to the test case to get an idea of the usage of the extension.
When I make a OOT block in gnuradio
class mod(gr.sync_block):
"""
docstring for block mod
"""
def __init__(self):
gr.sync_block.__init__(self,
name="mod",
in_sig=[np.byte],
out_sig=[np.complex64])
def work(self, input_items, output_items):
in0 = input_items[0]
out = output_items[0]
result=do(....)
out[:]=result
return len(output_items[0])
I get:
ValueError: could not broadcast input array from shape (122879) into shape (4096)
How can I solve it?
GRC is as below:
selector :input index and output index are controlled by WX GUI Chooser block
FSK4 MOD: modulate fsk4 signal and write data to raw.bin
FSK4 DEMOD : read data from raw.bin and demodulate
file source -> /////// -> FSK4 MOD -> FSK4 DEMOD -> NULL SINK
selector
file source -> ////// -> GMKS MOD -> GMSK DEMOD ->NULL SINK
when the input index or output index is changed,the whole flow graph will be not responding.
There's two things:
You have a bug somewhere, and the solution is not to change something, but fix that bug. The full Python error message will tell you exactly in which line the error is.
noutput_items is a variable that GNU Radio sets at runtime to let you know how much output you might produce in this call to work. Hence, it's not something you can set, but it's something your work method must respect.
I think it's fair to assume that you're not very aware of how GNU Radio works:
GNU Radio is based on calling your block's work function when there is enough output space available and enough input items to process. The amount of output space that your block can use is passed to your work as a parameter, and will change between calls to work.
I very strongly recommend going through chapters 1-3 of the official Guided Tutorials if you haven't already. We always try to keep these tutorials up-to-date.
EDIT: Your command shows that you have not really understood what I meant, sorry. So: GNU Radio calls your work method over and over again while it's executing.
For example, it might call work with 4000 input items and 4000 output items space (you have a sync block, therefore number of input==number of output). Your work processes the first 1000 of that, and therefore return 1000. So there's 3000 items left.
Now, the upstream block does something, so there's 100 new items. Because the 3000 from before are still there, your block's work will get called with 3100 items.
Your work processes any number of items, and returns that number. GNU Radio makes sure that the "remaining" items stay available and will call your work again if there is enough in- our output.
I'm using DirectSound's DirectSoundBuffer to play voice data that I am streaming over a network. The nature of voice/speech is that it doesn't necessarily constantly stream (it starts and stops) and it also can come in varying packet sizes (which makes it difficult to predict where the buffer should stop). I keep a StoppingPoint variable that keeps track of where in my buffer I have written up to (it also takes into account the circular nature of this buffer). If the StoppingPoint is reached, I would like to stop playing my buffer. Also, I StoppingPoint would also signify the point at which I'd also like to start writing from.
My buffer
|==============================|--------------------------|
^ ^ ^
| | |
Voice Data Stopping point Old/Garbage
data
Also, I can't use notifications, because in order to use a notification the buffer must be stopped. But in my case, it is very likely for more data to come in as the buffer is playing, thus pushing back my 'StoppingPoint` value.
What I currently have now is a function that is called every frame. This function, amongst other things, is checking where the Play cursor is. If the Play cursor is passed the StoppingPoint location, it stops the buffer, and moves the Play cursor back to the StoppingPoint location. This seems to work ok so far, but as you'd expect, the Play cursor quite often overshoots the StoppingPoint. This means a little bit of old/garbage data is played every time the end of the streamed data is reached.
I'm just curious as to if there is a way to stop playing a DirectSoundBuffer at a specific offset? What I would like is to write data to the buffer, then play, then have it stop precisely at the location described by my StoppingPoint variable without overshooting it.
Note: I haven't included any code because this is more of a high-level solution that I need. My implementation is incredibly straight forward, typical and for the most part it DOES work. I just need a nudge in the right direction to remove the overshooting of my StoppingPoint. Perhaps there is a function I can use? Or some other algorithm that is commonly used to achieve this?
I've come up with a solution, but I'm still interested in any feedback or alternative solutions. I'm not sure if the way I've done it is ok, but it seems to be producing the desired results. Although, my solution feels a little too convoluted...
1) Whenever I write data, I now write at either the Buffer's Write Cursor or the StoppingPoint - whichever is later. This is to avoid stopping, then later writing in that "untouchable" space between the Play Cursor and Write Cursor, whose data has already been dedicated to playback.
DWORD writeCursorOffset = 0;
buffer->GetCurrentPosition(NULL, &writeCursorOffset);
//if write cursor has passed stopping point, then we need to write from there.
//So update m_StoppingPoint to reflect the new writing position.
m_StoppingPoint = m_StoppingPoint > writeCursorOffset ? m_StoppingPoint : writeCursorOffset;
2) I added some silence after every single write, but I left StoppingPoint to point at the end of the actual voice data. Eg.
|==============================|*********|---------------------|
^ ^ ^ ^
| | | |
Voice data Stopping Silence Old/Garbage
Point Data
3) If the Buffer's Play Cursor passed the StoppingPoint, I would then stop playing the buffer. Even if the Play Cursor overshoots here, all it will play is silence.
//error checking removed for demonstration purposes
buffer->Stop();
4) Immediately after stopping I would update StoppingPoint to be equal to the end of the silence. This would ensure that when more speech data comes in, the buffer will not play any silence first.
//don't forget that modulo at the end - circular buffer!
m_StoppingPoint = (m_StoppingPoint + SILENCE_BUFFER_SIZE) % BufferSize;
|==============================|*********|-------------------|
^
|
Move Stopping
Point here
Again, if I've done anything glaringly evil here, please let me know!