I have a text file in which there is a line with the category and then all items of that category in lines below it. This is followed by 2 empty lines and then the title of the next category and more items in the category. I want to know how I could use regular expressions (specifically with Notepad++) in order to put the category at the start of each of the item's lines so I can save the file as a CSV or TAB file.
I started by isolating one of the categories as such:
Городищенский поссовет 1541
Арабовщина 535
Болтичи 11
Бриксичи 59
Великое Село 160
Гарановичи 34
Грибовщина 3
Душковцы 5
Зеленая 182
Кисели 97
Колдычево 145
Конюшовщина 16
Микуличи 31
Мостытычи 18
Насейки 5
Новоселки 45
Омневичи 53
Поручин 43
Пруды 24
Станкевичи 42
Ясенец 33
I then got as far as getting to be finding for
(.+)(поссовет)(\t\d{4}\r\n)(^.*$\r\n)
and replacing with
$1$2\t$4
which makes the first line
Арабовщина 535
turn into
Городищенский поссовет Арабовщина 535
which is what I want to happen to the rest of the lines but I couldn't get any farther.
Not quite a homework problem, but it may as well be:
You have a long list of positive integer values stored in column A. These are packets in unit U.
A Drum can fit up to 500 U, but you cannot break up packets.
How many drums are required for any given list of values in column A?
This does not have to be the most efficient answer, processing in row order is absolutely fine.
I Think you should be able to solve this with a formula, but the closest I got was
=CEILING(SUM(A1:A1000)/500;1)
Of course, this breaks up packets.
Additionally, this problem requires me to be able to find the room left in each drum used, but emphasis for this question should remain on just the number required.
This cannot be done with a single simple formula. Each drum and packet needs to be counted. However contrary to my comment, for this particular problem a spreadsheet works well, and there is no need for a macro.
First, set B2 to 500 for use in other formulas. If column A is not yet filled, use the formula =RANDBETWEEN(1,B$2) to add some values.
Column C is the main formula that determines how full each drum is. Set C2 to =A2. C3 is =IF(C2+A3>B$2,A3,C2+A3). Fill C3 down to fill the remaining rows.
For column D, use =IF(C2+A3>B$2,B$2-C2,""). However the last row of column D is shorter: =B$2-C21 and change 21 to whatever the last row is.
Finally in column E we find the answer, which is simply =COUNT(D2:D21).
Packets Drum Size How Full Room left in each drum used Number of filled drums
------- --------- -------- --------------------------- ----------------------
206 500 206 294 13
309 309
68 377
84 461 39
305 305 195
387 387 113
118 118
8 126 374
479 479 21
492 492 8
120 120
291 411 89
262 262
108 370 130
440 440 60
88 88
100 188
102 290 210
478 478 22
87 87 413
For OpenOffice Calc, use semicolons ; instead of commas , in formulas.
I am trying to use nearmrg on my data files and I keep getting the same error:
using data not sorted
As I wanted to break down the problem I used very simple test data instead of my real data and the error message still shows up. Now I have the following:
Master.dta:
Group Date
A 15.01.2012
A 15.02.2012
B 15.01.2012
B 15.02.2012
C 15.01.2012
C 15.02.2012
Using.dta:
Group Date SVarOfInterest1 SVarOfInterest2
A 01.01.2012 1 201
A 15.01.2012 2 202
A 03.02.2012 3 203
A 23.02.2012 4 204
B 03.01.2012 11 211
B 19.01.2012 12 212
B 03.02.2012 13 213
C 20.01.2012 21 221
C 25.01.2012 22 222
C 04.02.2012 23 223
C 03.01.2012 24 224
This is the code:
nearmrg Group using Using.dta, nearvar(Date) genmatch(SourceDate) lower
using data not sorted
r(5);
It appears that Stata thinks that your using data is not sorted. Even if if looks sorted to you, run the sort command on each data file prior to running nearmrg.
tempfile myTemp
<read in Using file>
sort Group
* save temporary file
save "`myTemp'"
<read in master file>
sort Group
nearmrg Group using `myTemp', nearvar(Date) genmatch(SourceDate) lower
As a side note, nearmrg is not part of base Stata. It is helpful if you mention that it is a user written package in your question.
I did my best to include all of (and only) the information that is needed. Let me know if seeing more could help you help me.
I am having problems with my regex. My program prints to a .txt file lexeme - token pairs. Everything is working fine except for identifiers and integers. For example, the token for an identifier is 262.
Sample input file:
function main a: integer returns integer;
b: integer is a * 2;
Sample output:
function - 269
maina - 262
integer - 272
returns - 274
integer - 272
; - 59
b - 262
integer - 272
is - 271
a - 262
2 - 263
; - 59
Correct output would be:
function - 269
main - 262
a - 262
: - 58
integer - 272
returns - 274
integer - 272
; - 59
b - 262
: - 58
integer - 272
is - 271
a - 262
* - 261
2 - 263
; - 59
To summarize, it combines two identifiers separated by a white space, and it also deletes any token after the identifier.
You've just forgotten to put curly braces around some of the names. Instead of
{letter}(letter|digit)*
you should write
{letter}({letter}|{digit})*
You also need to move the rules for function, integer and returns before the rule for identifier, since the rule for identifier match those too.
I have a data set that looks like this:
000 100 200 300 010 020 030 001 002 003
001 101 201 301 011 021 031 000 002 003
002 102 202 302 012 022 032 001 000 003
003 103 203 303 013 023 033 001 002 000
010 110 210 310 000 020 030 011 012 013
020 120 220 320 010 000 030 021 022 023
030 130 230 330 010 020 000 031 032 033
033 133 233 333 013 023 003 031 032 030
100 000 200 300 110 120 130 101 102 103
133 033 233 333 113 123 103 131 132 130
200 100 000 300 210 220 230 201 202 203
233 133 033 333 213 223 203 231 232 230
300 100 200 000 310 320 330 301 302 303
303 103 203 003 313 323 333 301 302 300
313 113 213 013 303 323 333 311 312 310
330 130 230 030 310 320 300 331 332 333
331 131 231 031 311 321 301 330 332 333
332 132 232 032 312 322 302 331 330 333
333 133 233 033 313 323 303 331 332 330
What I intend to do is to generate list of unique strings from it, yielding:
000
001
002
003
010
011
012
013
020
021
022
023
030
031
032
033
100
101
102
103
110
113
120
123
130
131
132
133
200
201
202
203
210
213
220
223
230
231
232
233
300
301
302
303
310
311
312
313
320
321
322
323
330
331
332
333
The code I have to generate that is this. But it is very memory consumptive.
Because in reality the string is of length >36 and there are more than 35 million
lines in a file. Each line with >36*3 number of columns/entries.
Is there a memory efficient way to do it?
#include <iostream>
#include <vector>
#include <fstream>
#include <sstream>
#include <map>
using namespace std;
int main ( int arg_count, char *arg_vec[] ) {
if (arg_count !=2 ) {
cerr << "expected one argument" << endl;
return EXIT_FAILURE;
}
string line;
ifstream myfile (arg_vec[1]);
map <string,int> Tags;
if (myfile.is_open())
{
while (getline(myfile,line) )
{
stringstream ss(line);
string Elem;
while (ss >> Elem) {
Tags[Elem] = 1;
}
}
myfile.close();
}
else { cout << "Unable to open file";}
for (map <string,int>::iterator iter = Tags.begin(); iter !=
Tags.end();iter++) {
cout << (*iter).first << endl;
}
return 0;
}
This depends a bit on the characteristics of your dataset. In the worse case, where all strings are unique, you will need either O(n) memory to record your seen-set, or O(n^2) time to re-scan the entire file on each word. However, there are improvements that can be made.
First off, if your dataset only consists of 3-digit integers, then a simple array of 1000 bools will be much more memory effieicnt than a map.
If you're using general data, then another good approach would be to sort the set, so copies of the same string end up adjacent, then simply remove adjacent words. There are well-researched algorithms for sorting a dataset too large to fit in memory. This is most effective when a large percentage of the words in the set are unique, and thus holding a set of seen words in memory is prohibitively expensive.
Incidentally, this can be implemented easily with a shell pipeline, as GNU sort does the external sort for you:
tr " " "\n" < testdata | LC_ALL=C sort | uniq
Passing LC_ALL=C to sort disables locale processing and multibyte character set support, which can give a significant speed boost here.
O(1) memory [ram]:
If you want to use no memory at all (besides a couple temp variables) you could simply read 1 entry at a time and add it to the output file if it doesn't already exist in the output file. This would be slower on time though since you'd have to read 1 entry at a time from the output file.
You could insert the entry into the output file in alphabetical order though and then you would be able to see if the entry already exists or not in O(logn) time via binary search per entry being inserted. To actually insert you need to re-create the file though which is O(nlogn). You do this n times for each input string, so overall the algorithm would run in O(n^2logn) (which includes lookup to find insertion pos + insertion) and use O(1) RAM memory.
Since your output file is already in alphabetical order though future lookups would also only be O(logn) via binary search.
You could also minimize the re-creation phase of the file by leaving excessive space between entries in the file. WHen the algorithm was done you could do a vacuum on the file. This would bring it down to O(nlogn).
Another way to reduce memory:
If it's common that your strings share common prefixes then you can use a trie and probably use less memory overall since you mentioned your strings are > length 36. This would still use a lot of memory though.
Well, std::set might be slightly faster and use less memory than std::map.
It seems given that large number of entries, there will be a reasonable amount of overlap in the sequences of symbols. You could build tree using the entries at each position each sequence as nodes. Say you have an entry 12345 and 12346 then the first four entries in the sequence overlap and thus could be stored in a tree with 6 nodes.
You could walk the tree to see if a given symbol is contained at a given position, if not you would add it. When you reach the end of the given string you would just mark that node as terminating a string node. Reproducing the unique entries would be a matter of a depth first traversal of the tree the path from the root node to each terminator represents a unique entry.
If you partition the dataset, say into X thousand line chunks and aggregate the trees it would make a nice map-reduce job.
You're looking at a node space of 10^36 so if the data is entirely random you're looking at a large possible number of nodes. If there's a good deal of overlap and a smaller number of unique entries you will probably find you use a good deal less
You could probably just write an in-place sort for it. You're still going to have to load the file to memory though, because in-place sorting with IO wouldn't be efficient. You'd have to read and write for every comparison.
std::set would be better, but you should look into hash sets. Those are not available in the current C++ standard library (although it is supposed to be in C++0x's) but some compilers have implementations. Visual C++ has stdext::hash_set and boost has some kind of stuff for this, see Boost Multi-index Containers Library.
Try STXXL as an external stl for huge datasets.
The solution I would suggest is to use memory mapped file to access the data and radix sort to sort the list. This is trivial if the strings are of same length.
If the strings are of different lengths, use radix sort for a fast presorting using the n first characters, then sort the unsorted sub lists with whatever method is most appropriate. With very short sublists, a simple bubble sort would do it. With longer lists use quick sort or use the STL set with a custom class providing the compare operator.
With memory mapping and radix sort, the memory requirement and performance would be optimal. All you need is the mapping space (~size of file) and a table of 32bit values with one value per string to hold the linked list for radix sort.
You could also save memory and speed up the sorting by using a more compact encoding of the strings. A simple encoding would use 2 bits per character, using values 1,2 and 3 for the three letters and 0 to signal the end of string. Or more efficient, use a base 3 encoding, packed into integers and encode the length of string in front. Let say you have characters c1, c2, C3, c4 the encoding would yield the integer c1*3^3 + c2*3^2 + c3*3^1 + c4*3^0. This suppose you assign a value from 0 to 2 to each character. By using this encoding you'll save memory and will optimize sorting if the number of strings to sort is huge.
You can look at the excellent library Judy arrays. A compact trie based, very fast and memory efficient data structure witch scale to billons strings. Better than any search-tree.
You can use the JudySL functions to your purpose. You can use it similar to your program, but it is much faster and more memory efficient.