Here's a minimal working example:
program test_stuff
implicit none
real :: b
b = 10000.0
write(*,'(A10,1PE12.4,F12.4)') "b, b: ", b, b
end program
which I simply compile with gfortran test_stuff.f90 -o test_stuff
However, running the program gives the following output:
$ ./test_stuff
b, b: 1.0000E+04 100000.0000
The second real written to the screen is wrong by a factor of 10.
This happens with gfotran 9.3.0 as well as 10.2.0, so I definitely must be doing something wrong, but I can't see what it is. Can anybody spot what I'm doing wrong?
The P control edit descriptor "temporarily changes" (Fortran 2018 13.8.5) the scale factor connection mode of the connection.
However, what is meant by temporary is until the mode is changed again or until the end of the data transfer statement: (Fortran 2018 12.5.2)
Edit descriptors take effect when they are encountered in format processing. When a data transfer statement terminates, the values for the modes are reset to the values in effect immediately before the data transfer statement was executed.
In the case of the question, both output values are thus processed with the scale factor having value 1.
This scale factor is responsible for the "wrong" second value: there is a difference in interpretation of the scale factor for E and F editing. For E editing the scale factor simply changes the representation, with the external and internal values the same (with the significand scaled up by 10 and the exponent reduced by 1), but for F editing the output value is itself scaled:
On output, with F output editing, the effect is that the externally represented number equals the internally represented number multiplied by 10k
So while 10000 would be represented by 0.1000E+05 with scale factor 0 and 1.0000E+04 with scale factor 1 under E12.4, under F12.4 the value 10000 is scaled to 100000 with the scale factor in place.
As a style note: although the comma is optional between 1P and E12.4 (and similar), many would regard it much better to include the comma, precisely to avoid this apparent tight coupling of the two descriptors (or looking like one descriptor). As the scale factor has a different effect for each of E and F, has no effect for EN and sometimes but not always has an effect with G, I'm not going to argue with anyone who calls P evil.
You are looking for section 12.5.2 of the Fortran 2018 standard.
A connection for formatted input/output has several changeable modes: these are ... and scale factor (13.8.5).
Values for the modes of a connection are established when the connection is initiated. If the connection is initiated by an OPEN statement, the values are as specified, either explicitly or implicitly, by the OPEN statement. If the connection is initiated other than by an OPEN statement (that is, if the file is an internal file or pre-connected file) the values established are those that would be implied by an initial OPEN statement without the corresponding keywords.
The scale factor cannot be explicitly specified in an OPEN statement; it is implicitly 0.
The modes of a connection can be temporarily changed by ... or by an edit descriptor. ... Edit descriptors take effect when they are encountered in format processing. When a data transfer statement terminates, the values for the modes are reset to the values in effect immediately before the data transfer statement was executed.
So when you used 1P in your format, you changed the mode for the connection. This applies to all output items after the 1P has been processed. When the write statement completes the scale factor is reset to 0.
Related
I'm trying to play with Kafka Stream to aggregate some attribute of People.
I have a kafka stream test like this :
new ConsumerRecordFactory[Array[Byte], Character]("input", new ByteArraySerializer(), new CharacterSerializer())
var i = 0
while (i != 5) {
testDriver.pipeInput(
factory.create("input",
Character(123,12), 15*10000L))
i+=1;
}
val output = testDriver.readOutput....
I'm trying to group the value by key like this :
streamBuilder.stream[Array[Byte], Character](inputKafkaTopic)
.filter((key, _) => key == null )
.mapValues(character=> PersonInfos(character.id, character.id2, character.age) // case class
.groupBy((_, value) => CharacterInfos(value.id, value.id2) // case class)
.count().toStream.print(Printed.toSysOut[CharacterInfos, Long])
When i'm running the code, I got this :
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 1
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 2
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 3
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 4
[KTABLE-TOSTREAM-0000000012]: CharacterInfos(123,12), 5
Why i'm getting 5 rows instead of just one line with CharacterInfos and the count ?
Doesn't groupBy just change the key ?
If you use the TopologyTestDriver caching is effectively disabled and thus, every input record will always produce an output record. This is by design, because caching implies non-deterministic behavior what makes itsvery hard to write an actual unit test.
If you deploy the code in a real application, the behavior will be different and caching will reduce the output load -- which intermediate results you will get, is not defined (ie, non-deterministic); compare Michael Noll's answer.
For your unit test, it should actually not really matter, and you can either test for all output records (ie, all intermediate results), or put all output records into a key-value Map and only test for the last emitted record per key (if you don't care about the intermediate results) in the test.
Furthermore, you could use suppress() operator to get fine grained control over what output messages you get. suppress()—in contrast to caching—is fully deterministic and thus writing a unit test works well. However, note that suppress() is event-time driven, and thus, if you stop sending new records, time does not advance and suppress() does not emit data. For unit testing, this is important to consider, because you might need to send some additional "dummy" data to trigger the output you actually want to test for. For more details on suppress() check out this blog post: https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers
Update: I didn't spot the line in the example code that refers to the TopologyTestDriver in Kafka Streams. My answer below is for the 'normal' KStreams application behavior, whereas the TopologyTestDriver behaves differently. See the answer by Matthias J. Sax for the latter.
This is expected behavior. Somewhat simplified, Kafka Streams emits by default a new output record as soon as a new input record was received.
When you are aggregating (here: counting) the input data, then the aggregation result will be updated (and thus a new output record produced) as soon as new input was received for the aggregation.
input record 1 ---> new output record with count=1
input record 2 ---> new output record with count=2
...
input record 5 ---> new output record with count=5
What to do about it: You can reduce the number of 'intermediate' outputs through configuring the size of the so-called record caches as well as the setting of the commit.interval.ms parameter. See Memory Management. However, how much reduction you will be seeing depends not only on these settings but also on the characteristics of your input data, and because of that the extent of the reduction may also vary over time (think: could be 90% in the first hour of data, 76% in the second hour of data, etc.). That is, the reduction process is deterministic but from the resulting reduction amount is difficult to predict from the outside.
Note: When doing windowed aggregations (like windowed counts) you can also use the Suppress() API so that the number of intermediate updates is not only reduced, but there will only ever be a single output per window. However, in your use case/code you the aggregation is not windowed, so cannot use the Suppress API.
To help you understand why the setup is this way: You must keep in mind that a streaming system generally operates on unbounded streams of data, which means the system doesn't know 'when it has received all the input data'. So even the term 'intermediate outputs' is actually misleading: at the time the second input record was received, for example, the system believes that the result of the (non-windowed) aggregation is '2' -- its the correct result to the best of its knowledge at this point in time. It cannot predict whether (or when) another input record might arrive.
For windowed aggregations (where Suppress is supported) this is a bit easier, because the window size defines a boundary for the input data of a given window. Here, the Suppress() API allows you to make a trade-off decision between better latency but with multiple outputs per window (default behavior, Suppress disabled) and longer latency but you'll get only a single output per window (Suppress enabled). In the latter case, if you have 1h windows, you will not see any output for a given window until 1h later, so to speak. For some use cases this is acceptable, for others it is not.
I made a simple model of a heat pump which uses sensor data to calculate its COP.
while COP = heat / power
sometimes there is no power so the system does a (cannot divide by zero). I would like these values to just be zero. So i tried an IF-statementif-statement. if power(u) = 0 then COP(y) = 0. somehow this does not work (see time 8)COP output + data. Anyone who seems to notice the problem?
edit(still problems at time 8.1
edit(heat and power)
To make the computation a bit more generally applicable (e.g. the sign of power can change), take a look at the code below. It could also be a good idea to build a function from it (for the function the noEvent()-statements can be left out)...
model DivNoZeroExample
parameter Real eps = 1e-6 "Smallest number to be used as divisor";
Real power = 0.5-time "Some artificial value for power";
Real heat = 1 "Some artificial value for heat";
Real COP "To be computed";
equation
if noEvent(abs(power) < abs(eps)) then
COP = if noEvent(power>= 0) then heat/eps else heat/(-eps);
else
COP = heat/power;
end if;
end DivNoZeroExample;
Relational operations work a bit differently in Modelica.
If you replace if u>0 by if noEvent(u>0) it should work as you expected.
For details see section 8.5 Events and Synchronization in the Modelica specification https://modelica.org/documents/ModelicaSpec34.pdf
I'm running a distributed model stripped to its bare minimum below:
integer, parameter :: &
nx = 1200,& ! Number of columns in grid
ny = 1200,& ! Number of rows in grid
nt = 6000 ! Number of timesteps
integer :: it ! Loop counter
real :: var1(nx,ny), var2(nx,ny), var3(nx,ny), etc(nx,ny)
! Create netcdf to write model output
call check( nf90_create(path="out.nc",cmode=nf90_clobber, ncid=nc_out_id) )
! Loop over time
do it = 1,nt
! Calculate a lot of variables
...
! Write some variables in out.nc at each timestep
CALL check( nf90_put_var(ncid=nc_out_id, varid=var1_varid, values=var1, &
start = (/ 1, 1, it /), count = (/ nx, ny, 1 /)) )
! Close the netcdf otherwise it is not readable:
if (it == nt) call check( nf90_close(nc_out_id) )
enddo
I'm in the development stage of the model so, it inevitably crashes at unexpected points (usually at the Calculate a lot of variables stage), which means that, if the model crashes at timestep it =3000, 2999 timesteps will be written to the netcdf output file, but I will not be able to read the file because the file has not been closed. Still, the data have been written: I currently have a 2GB out.nc file that I can't read. When I ncdump the file it shows
netcdf out.nc {
dimensions:
x = 1400 ;
y = 1200 ;
time = UNLIMITED ; // (0 currently)
variables:
float var1 (time, y, x) ;
data:
}
My questions are: (1) Is there a way to close the file retrospectively, even outside Fortran, to be able to read the data that have already been written? (2) Alternatively, is there another way to write the file in Fortran that would make the file readable even without closing it?
When nf90_close is called, buffered output is written to disk and the file ID is relinquished so it can be reused. The problem is most likely due to buffered output not having been written to the disk when the program terminates due to a crash, meaning that only the changes you made in "define mode" are present in the file (as shown by ncdump).
You therefore need to force the data to be written to the disk more often. There are three ways of doing this (as far as I am aware).
nf90_sync - which synchronises the buffered data to disk when called. This gives you the most control over when to output data (every loop step, or every n loop steps, for example), which can allow you to optimize for speed vs robustness, but introduces more programming and checking overhead for you.
Thanks to #RussF for this idea. Creating or opening the file using the nf90_share flag. This is the recommended approach if the netCDF file is intended to be used by multiple readers/writers simultaneously. It is essentially the same as an automatic implementation of nf90_sync for writing data. It gives less control, but also less programming overhead. Note that:
This only applies to netCDF-3 classic or 64-bit offset files.
Finally, an option I wouldn't recommend, but am including for completeness (and I guess there may be situations where this is the best option, although none spring to mind) - closing and reopening the file. I don't recommend this, because it will slow down your program, and adds greater possibility of causing errors.
I am trying to use REFPROPs HSFLSH subroutine to compute properties for steam.
When the same state property is calculated over multiple iterations
(fixed enthalpy and entropy (Enthalpy = 50000 J/mol & Entropy = 125 J/mol),
the time taken to compute using HSFLSH after every 4th/5th iteration increases to about 0.15 ms against negligible amount of time for other iterations. This is turning problematic because my program places call to this subroutine over several thousand times. Thus leading to abnormally huge program run times.
The program used to generate the above log is here:
C refprop check
program time_check
parameter(ncmax=20)
dimension x(ncmax)
real hkj,skj
character hrf*3, herr*255
character*255 hf(ncmax),hfmix
C
C SETUP FOR WATER
C
nc=1 !Number of components
hf(1)='water.fld' !Fluid name
hfmix='hmx.bnc' !Mixture file name
hrf='DEF' !Reference state (DEF means default)
call setup(nc,hf,hfmix,hrf,ierr,herr)
if (ierr.ne.0) write (*,*) herr
call INFO(1,wm,ttp,tnbp,tc,pc,dc,zc,acf,dip,rgas)
write(*,*) 'Mol weight ', wm
h = 50000.0
s = 125.0
c
C
DO I=1,NCMAX
x(I) = 0
END DO
C ******************************************************
C THIS IS THE ACTUAL CALL PLACE
C ******************************************************
do I=1,100
call cpu_time(tstrt)
CALL HSFLSH(h,s,x,T_TEMP,P_TEMP,RHO_TEMP,dl,dv,xliq,xvap,
& WET_TEMP,e,
& cv,cp,VS_TEMP,ierr,herr)
call cpu_time(tstop)
write(*,*),I,' time taken to run hsflsh routine= ',tstop - tstrt
end do
stop
end
(of course you will need the FORTRAN FILES, which unfortunately I cannot share since REFPROP isn't open source)
Can someone help me figure out why is this happening.?
P.S : The above code was compiled using gfortran -fdefault-real-8
UPDATE
I tried using system_clock to time my computations as suggested by #Ross below. The results are uniform across the loop (image below). I will have to find alternate ways to improve computation speed I guess (Sigh!)
I don't have a concrete answer, but this sort of behaviour looks like what I would expect if all calls really took around 3 ms, but your call to CPU_TIME doesn't register anything below around 15 ms. Do you see any output with time taken less than, say 10 ms? Of particular interest to me is the approximately even spacing between calls that return nonzero time - it's about even at 5.
CPU timing can be a tricky business. I recommended in a comment that you try system_clock, which can be higher precision than CPU_TIME. You said it doesn't work, but I'm unconvinced. Did you pass a long integer to system_clock? What was the count_rate for your system? Were all the times still either 15 or 0 ms?
I have an audio signal that has a kind of FM encoded signal on it. The encoded signal is using this Biphase mark coding technique <-- see at the end of this page.
This signal is a digital representation of a timecode, in hours, minutes, seconds and frames. It basically works like this:
lets consider that we are working in 25 frames per second;
we know that the code is transmitting 80 bits of information every frame (that is 80 bits per frame x 25 frames per second = 2000 bits per second);
The wave is being sampled at 44100 samples per second. So, if we divide 44100/2000 we see that every bit uses 22,05 samples;
A bit happens when the signal changes sign.
If the wave changes sign and keeps its sign during the whole bit period it is a ZERO. If the wave changes sign two times over one bit period it is a ONE;
What my code does is this:
detects the first zero crossing, that is the clock start (to)
measures the level for to = to + 0.75*bitPeriod... 0.75 to give a tolerance.
if that second level is different, we have a 1, if not we have a 0;
This is the code:
// data is a C array of floats representing the audio levels
float bitPeriod = ceil(44100 / 2000);
int firstZeroCrossIndex = findNextZeroCross(data);
// firstZeroCrossIndex is the value where the signal changed
// for example: data[0] = -0.23 and data[1] = 0.5
// firstZeroCrossIndex will be equal to 1
// if firstZeroCrossIndex is invalid, go away
if (firstZeroCrossIndex < 0) return
float firstValue = data[firstZeroCrossIndex];
int lastSignal = sign(firstValue);
if (lastSignal == 0) return; // invalid, go away
while (YES) {
float newValue = data[firstZeroCrossIndex + 0.75* bitPeriod];
int newSignal = sign(newValue);
if (lastSignal == newSignal)
printf("0");
else
printf("1");
firstZeroCrossIndex += bitPeriod;
// I think I must invert the signal here for the next loop interaction
lastSignal = -newSignal;
if (firstZeroCrossIndex > maximuPossibleIndex)
break;
}
This code appears logical to me but the result coming from it is a total nonsense. What am I missing?
NOTE: this code is executing over a live signal and reads values from a circular ring buffer. sign returns -1 if the value is negative, 1 if the value is positive or 0 if the value is zero.
Cool problem! :-)
The code fails in two independent ways:
You are searching for the first (any) zero crossing. This is good. But then there is a 50% chance, that this transition is the one which occurs before every bit (0 or 1) or whether this transition is one which marks a 1 bit. If you get it wrong in the beginning you end up with nonsense.
You keep on adding bitPeriod (float, 22.05) to firstZeroCrossIndex (int). This means that your sampling points will slowly run out of phase with your analog signal and you will see strange effects when you sample point gets near the signal transitions. You will get nonsense, periodically at least.
Solution to 1: You must search for at least one 0 first, so you know which transition indicates just the next bit and which indicates a 1 bit. In practice you will want to re-synchronize your sampler at every '0' bit.
Solution to 2: Do not add bitPeriod to your sampling point. Instead search for the next transition, like you did in the beginning. The next transition is either 'half a bit' away, or a 'complete bit' away, which gives you the information you want. After a 'half a bit' period you must see another 'half a bit' period. If not, you must re-synchronize since you took a middle transition for a start transition by accident. This is exactly the re-sync I was talking about in 1.