I am trying to find days matching to a reference number of days given or else to find the number of days close to the reference days.
I coded till here, however not sure how to go forward.
ID Date ref_days lags total_days
1 2017-02-02 224 . 0
1 2017-02-02 224 84 84
1 2017-02-02 224 84 168
2 2015-01-21 213 300 388
3 2016-02-12 560 95 .
3 2016-02-12 560 86 181
3 2016-02-12 560 82 263
3 2016-02-12 560 69 332
3 2016-02-12 560 77 409
So now I want to bring out the last value close to the reference days.
and the next total_days should start from ZERO again to find the next window. How can I do this?
Here is a code that I wrote
data want;
do until (totaldays <= ref_days);
set have;
by ID ref_days notsorted;
if first.id then totaldays=0;
else totaldays+lags;
end;
run;
Required Output:
ID Date ref_days lags total_days
1 2017-02-02 224 . 0
1 2017-02-02 224 84 84
1 2017-02-02 224 84 168
2 2015-01-21 213 300 388
3 2016-02-12 560 95 .
3 2016-02-12 300 86 181
3 2016-02-12 300 82 263
3 2016-02-12 300 69 .
3 2016-02-12 300 77 146
A while ago I did similar to this via Proc sql. It calculates all the distances and takes the closest one. It works with moderate size dataset. Hopefully it is of some use.
proc sql;
select * from
(
select *,
abs(t1.link-t2.link) as dist /*In your case these would be dateVars*/
from test1 t1
left join test2 t2
on 1=1) group by system1 having dist=min(dist);
;
quit;
There was some talk that the left join on 1=1 is a bit silly (as full outter join would suffice, or something.) However this worked for the problem in question.
Not quite a homework problem, but it may as well be:
You have a long list of positive integer values stored in column A. These are packets in unit U.
A Drum can fit up to 500 U, but you cannot break up packets.
How many drums are required for any given list of values in column A?
This does not have to be the most efficient answer, processing in row order is absolutely fine.
I Think you should be able to solve this with a formula, but the closest I got was
=CEILING(SUM(A1:A1000)/500;1)
Of course, this breaks up packets.
Additionally, this problem requires me to be able to find the room left in each drum used, but emphasis for this question should remain on just the number required.
This cannot be done with a single simple formula. Each drum and packet needs to be counted. However contrary to my comment, for this particular problem a spreadsheet works well, and there is no need for a macro.
First, set B2 to 500 for use in other formulas. If column A is not yet filled, use the formula =RANDBETWEEN(1,B$2) to add some values.
Column C is the main formula that determines how full each drum is. Set C2 to =A2. C3 is =IF(C2+A3>B$2,A3,C2+A3). Fill C3 down to fill the remaining rows.
For column D, use =IF(C2+A3>B$2,B$2-C2,""). However the last row of column D is shorter: =B$2-C21 and change 21 to whatever the last row is.
Finally in column E we find the answer, which is simply =COUNT(D2:D21).
Packets Drum Size How Full Room left in each drum used Number of filled drums
------- --------- -------- --------------------------- ----------------------
206 500 206 294 13
309 309
68 377
84 461 39
305 305 195
387 387 113
118 118
8 126 374
479 479 21
492 492 8
120 120
291 411 89
262 262
108 370 130
440 440 60
88 88
100 188
102 290 210
478 478 22
87 87 413
For OpenOffice Calc, use semicolons ; instead of commas , in formulas.
I'm trying to capture a collection of paragraphs that look like those shown below.
I would like to capture each paragraph in a separate collection. I have figured out how to capture each line independently, but not the full paragraph.
I'm using PCRE engine.
Any help would be greatly appreciated. I think there are may be new lines/line breaks at the end of each line also...if that makes a difference. Some paragraphs may be 5 lines long, or as short as 2 lines.
FORECAST VALID 04/0000Z 33.8N 77.3W
MAX WIND 85 KT...GUSTS 105 KT.
64 KT... 20NE 20SE 0SW 20NW.
50 KT... 40NE 50SE 20SW 40NW.
34 KT...100NE 110SE 70SW 60NW.
FORECAST VALID 04/1200Z 36.3N 74.4W
MAX WIND 90 KT...GUSTS 110 KT.
64 KT... 30NE 30SE 0SW 20NW.
50 KT... 50NE 50SE 30SW 40NW.
34 KT...100NE 110SE 80SW 70NW.
FORECAST VALID 05/0000Z 39.4N 70.2W
MAX WIND 60 KT...GUSTS 75 KT.
50 KT... 60NE 80SE 60SW 60NW.
34 KT...100NE 130SE 110SW 90NW.
I need to pull values out when at the following numbers (in a pipe delimited .txt file):
221
262
303
344
385
to infinity...
The pattern is 221 + 41(x), therefore the logic for the when needs to be when FieldNum is both greater than 220 and FieldNum mod 41 = 16.
How do I do this conditional?
This is what I have:
<xsl:when test ="$FieldNum > 220 and $FieldNum mod 41 = 16">
I'm getting an XSLT compile error when doing this.
All that's missing is the semi-colon at the end of the entity. You want ">".
I have a data set that looks like this:
000 100 200 300 010 020 030 001 002 003
001 101 201 301 011 021 031 000 002 003
002 102 202 302 012 022 032 001 000 003
003 103 203 303 013 023 033 001 002 000
010 110 210 310 000 020 030 011 012 013
020 120 220 320 010 000 030 021 022 023
030 130 230 330 010 020 000 031 032 033
033 133 233 333 013 023 003 031 032 030
100 000 200 300 110 120 130 101 102 103
133 033 233 333 113 123 103 131 132 130
200 100 000 300 210 220 230 201 202 203
233 133 033 333 213 223 203 231 232 230
300 100 200 000 310 320 330 301 302 303
303 103 203 003 313 323 333 301 302 300
313 113 213 013 303 323 333 311 312 310
330 130 230 030 310 320 300 331 332 333
331 131 231 031 311 321 301 330 332 333
332 132 232 032 312 322 302 331 330 333
333 133 233 033 313 323 303 331 332 330
What I intend to do is to generate list of unique strings from it, yielding:
000
001
002
003
010
011
012
013
020
021
022
023
030
031
032
033
100
101
102
103
110
113
120
123
130
131
132
133
200
201
202
203
210
213
220
223
230
231
232
233
300
301
302
303
310
311
312
313
320
321
322
323
330
331
332
333
The code I have to generate that is this. But it is very memory consumptive.
Because in reality the string is of length >36 and there are more than 35 million
lines in a file. Each line with >36*3 number of columns/entries.
Is there a memory efficient way to do it?
#include <iostream>
#include <vector>
#include <fstream>
#include <sstream>
#include <map>
using namespace std;
int main ( int arg_count, char *arg_vec[] ) {
if (arg_count !=2 ) {
cerr << "expected one argument" << endl;
return EXIT_FAILURE;
}
string line;
ifstream myfile (arg_vec[1]);
map <string,int> Tags;
if (myfile.is_open())
{
while (getline(myfile,line) )
{
stringstream ss(line);
string Elem;
while (ss >> Elem) {
Tags[Elem] = 1;
}
}
myfile.close();
}
else { cout << "Unable to open file";}
for (map <string,int>::iterator iter = Tags.begin(); iter !=
Tags.end();iter++) {
cout << (*iter).first << endl;
}
return 0;
}
This depends a bit on the characteristics of your dataset. In the worse case, where all strings are unique, you will need either O(n) memory to record your seen-set, or O(n^2) time to re-scan the entire file on each word. However, there are improvements that can be made.
First off, if your dataset only consists of 3-digit integers, then a simple array of 1000 bools will be much more memory effieicnt than a map.
If you're using general data, then another good approach would be to sort the set, so copies of the same string end up adjacent, then simply remove adjacent words. There are well-researched algorithms for sorting a dataset too large to fit in memory. This is most effective when a large percentage of the words in the set are unique, and thus holding a set of seen words in memory is prohibitively expensive.
Incidentally, this can be implemented easily with a shell pipeline, as GNU sort does the external sort for you:
tr " " "\n" < testdata | LC_ALL=C sort | uniq
Passing LC_ALL=C to sort disables locale processing and multibyte character set support, which can give a significant speed boost here.
O(1) memory [ram]:
If you want to use no memory at all (besides a couple temp variables) you could simply read 1 entry at a time and add it to the output file if it doesn't already exist in the output file. This would be slower on time though since you'd have to read 1 entry at a time from the output file.
You could insert the entry into the output file in alphabetical order though and then you would be able to see if the entry already exists or not in O(logn) time via binary search per entry being inserted. To actually insert you need to re-create the file though which is O(nlogn). You do this n times for each input string, so overall the algorithm would run in O(n^2logn) (which includes lookup to find insertion pos + insertion) and use O(1) RAM memory.
Since your output file is already in alphabetical order though future lookups would also only be O(logn) via binary search.
You could also minimize the re-creation phase of the file by leaving excessive space between entries in the file. WHen the algorithm was done you could do a vacuum on the file. This would bring it down to O(nlogn).
Another way to reduce memory:
If it's common that your strings share common prefixes then you can use a trie and probably use less memory overall since you mentioned your strings are > length 36. This would still use a lot of memory though.
Well, std::set might be slightly faster and use less memory than std::map.
It seems given that large number of entries, there will be a reasonable amount of overlap in the sequences of symbols. You could build tree using the entries at each position each sequence as nodes. Say you have an entry 12345 and 12346 then the first four entries in the sequence overlap and thus could be stored in a tree with 6 nodes.
You could walk the tree to see if a given symbol is contained at a given position, if not you would add it. When you reach the end of the given string you would just mark that node as terminating a string node. Reproducing the unique entries would be a matter of a depth first traversal of the tree the path from the root node to each terminator represents a unique entry.
If you partition the dataset, say into X thousand line chunks and aggregate the trees it would make a nice map-reduce job.
You're looking at a node space of 10^36 so if the data is entirely random you're looking at a large possible number of nodes. If there's a good deal of overlap and a smaller number of unique entries you will probably find you use a good deal less
You could probably just write an in-place sort for it. You're still going to have to load the file to memory though, because in-place sorting with IO wouldn't be efficient. You'd have to read and write for every comparison.
std::set would be better, but you should look into hash sets. Those are not available in the current C++ standard library (although it is supposed to be in C++0x's) but some compilers have implementations. Visual C++ has stdext::hash_set and boost has some kind of stuff for this, see Boost Multi-index Containers Library.
Try STXXL as an external stl for huge datasets.
The solution I would suggest is to use memory mapped file to access the data and radix sort to sort the list. This is trivial if the strings are of same length.
If the strings are of different lengths, use radix sort for a fast presorting using the n first characters, then sort the unsorted sub lists with whatever method is most appropriate. With very short sublists, a simple bubble sort would do it. With longer lists use quick sort or use the STL set with a custom class providing the compare operator.
With memory mapping and radix sort, the memory requirement and performance would be optimal. All you need is the mapping space (~size of file) and a table of 32bit values with one value per string to hold the linked list for radix sort.
You could also save memory and speed up the sorting by using a more compact encoding of the strings. A simple encoding would use 2 bits per character, using values 1,2 and 3 for the three letters and 0 to signal the end of string. Or more efficient, use a base 3 encoding, packed into integers and encode the length of string in front. Let say you have characters c1, c2, C3, c4 the encoding would yield the integer c1*3^3 + c2*3^2 + c3*3^1 + c4*3^0. This suppose you assign a value from 0 to 2 to each character. By using this encoding you'll save memory and will optimize sorting if the number of strings to sort is huge.
You can look at the excellent library Judy arrays. A compact trie based, very fast and memory efficient data structure witch scale to billons strings. Better than any search-tree.
You can use the JudySL functions to your purpose. You can use it similar to your program, but it is much faster and more memory efficient.