two way merge sort for 2 file c++ - c++

I was asked to apply Two way merge sort on two files(files of records) ,
the algorithm explains the steps as follows :
Sort phase
1)The records on the file to be sorted are divided into several groups.
Each group is called a run, and each run fits into main memory.
2)An internal sort is applied to each run, 3)and the resulting sorted runs are distributed to two external files.
Merge Sort: 1) One run from each of the external files created in the sort phase merge into a larger runs of sorted records.
2)The result is stored in a third file.
3)The data is distributed back into the first two files, and merging continues until all records are in one large run.
I was able to apply Sort Phase only , so the current files is :
(supposed run contains 3 keys only )
file 1:
50 95 110 | 40 120 153 | 22 80 140
file 2:
10 36 100 | 60 70 130
here's the steps of merge phase
so if i will solve it theoretically will perform the following :
Merge Phase:
step1 :
file 3 :
10 36 50 95 100 110 | 40 60 70 120 130 153 | 22 80 140
file 1:
10 36 50 95 100 100 | 22 80 140
file 2 :
40 60 70 120 130 153
step 2 :
file 3 :
10 36 40 50 60 70 95 100 110 120 130 135 | 22 80 140
file 1 :
10 36 40 50 60 70 95 100 110 120 130 135
file 2:
22 80 140
step 3 :
file 3 :
10 22 36 40 50 60 70 80 95 100 110 120 130 135 140
one run stop sort complete
Now i need to apply merge phase so each key from each file compared to each other and output the smaller to file 3 , and in step 2 redistribute file 3 into two file then merge and sort until have one sort run
How i can apply such algorithm in c++ , i'm little bit confused about how can i determine the size of each run in every step.

As commented by Amdt Jonasson, the program needs to keep track of the run sizes and the end of data for each file. In your example it appears the initial run size is a fixed run size of 3 elements. A merge of two runs of size 3 will result in a single run of size 6 as shown in your steps. In this case, only a single instance of run size and the end of data in each file needs to be tracked.
If the sort is to be a stable sort (the original order preserved on equal keys), and the run size is variable, then an array of run counts for each file will be needed, or some way to denote the end of runs in the file, such as a text file, with a special character sequence used as a end of run indicator.
If the sort is not required to be stable, then an out of order sequence (smaller key value after a larger key value) can be used to indicate the end of a run. The risk here is that two or more runs will appear to be a single run if the runs happen to be in order, which will lose stability and unbalance the run count on the files.
This is a two way merge sort using 3 files. If you use a 4th file, then each merge of runs can alternate between 2 output files, eliminating the need split up the runs after each merge pass.
An alternative for doing a 2 way merge sort with 3 files is a polyphase merge sort, but it's complicated, probably beyond what would be expected for a class assignment, and more of a "legacy" algorithm used back in the days of tape based sorts.

Related

Looping through csv files to append together in stata

I have 50 csv files I am trying to append together. They all take on the name mortyear. They currently look like
mort70
year cause sex
1970 HA M
1970 HA F
mort71
year cause sex
1971 HA M
1971 ST M
I am currently using the following code:
local years "70 71 72 73 74 75 76 77 78"
local file "mort"
foreach file of local years {
clear
import "file_path/`file'"
keep year cause
append
}
This doesn't seem to work. Is there a better way to accomplish this?
This code is based on a series of guesses on what the syntax might be or might do. Unravelling all the guesses would take a long answer. Here is a start.
Your code defines a local macro file containing text mort and then overwrites it in the loop. As you want the text mort always as the first element of a filename, and a indicator of year that differs around the loop, the filename is going to be assembled wrongly.
The first step is not to do that.
local years "70 71 72 73 74 75 76 77 78"
foreach file of local years {
clear
import "file_path/mort`file'"
keep year cause
append
}
Now your next problems include
There is no import command with that syntax.
You read in data and change it but never save the result.
A bare append isn't legal either.
At the frustrating early stage where everything is new, and if you don't have a mentor, then everything has to be looked up.

Increasing speed of a for loop through a List and dataframe

Currently I am dealing with a massive amount of Data in the original form of a list through combination. I am running conditions on each set of list through a for loop. Problem is this small for loop is taking hours with the data. I'm looking to optimize the speed by changing some functions or vectorizing it.
I know one of the biggest NO NOs is don't do Pandas or Dataframe operations in for loops but I need to sum up the columns and organize it a little to get what I want. It seems unavoidable.
So you have a better understanding, each list looks something like this when its thrown into a dataframe:
0
Name Role Cost Value
0 Johnny Tsunami Driver 1000 39
1 Michael B. Jackson Pistol 2500 46
2 Bobby Zuko Pistol 3000 50
3 Greg Ritcher Lookout 200 25
Name Role Cost Value
4 Johnny Tsunami Driver 1000 39
5 Michael B. Jackson Pistol 2500 46
6 Bobby Zuko Pistol 3000 50
7 Appa Derren Lookout 250 30
This is the current loop, any ideas?
for element in itertools.product(*combine_list):
combo = list(element)
df = pd.DataFrame(np.array(combo).reshape(-1,11))
df[[2,3]] = df[[2,3]].apply(pd.to_numeric)
if (df[2].sum()) <= 5000 and (df[3].sum()) > 190:
df2 = pd.concat([df2, df], ignore_index=True)
Couple things I've done that have sliced off some time but not enough.
*df[2].sum() to df[2].values.sum----its faster
*where the concat is in the if statement I've tried using append and also adding the dataframe together as a list...concat is actually 2 secs faster normally or it will end up being about the same speed.
*by the .apply(pd.to_numeric) changed it to .astype(np.int64) its faster as well.
I'm currently looking at PYPY and Cython as well but I want to start here first before I go through the headache.

Handle Time Baseline using Statistical approach

I want to perform an analysis on average handle time (AHT) from a population and come up with a baseline which can be set as a goal.
AHT: The total time taken to resolve a specific task.
Sample Table:
task aht_in_seconds
123 2
234 10
456 20
678 5
789 3000
454 47867
213 6
998 60
547 135
565 776
325 342
567 223
565 334
666 360
As per my analysis, I am taking the mean and standard deviation of the population and trying to remove the outliers by using Z-score and box zenkin method.
As per Z-score (x-mean/sigma) method (-2 to +2 range (95%)) the standard deviation comes very high even after removing outliers.
Box-zenkin method: (for outlier removal)
lower limit: Q1-1.5*Interquartile Range
Upper limit: Q3+1.5*Interquartile Range
(Interquartile Range is derived by SAS Univariate)
As per the box-zenkin method, the SD is minimal but the final AHT value does not look realistic i.e very low as 150 seconds whereas the time & motion study gives a higer value of 250 seconds for a task
I am wondering if my methods are correct or missing something. Are there any other methods to remove outliers and perform the same analysis. Any guidance would be a great help.

MIP GAP couldn't be set properly

I am solving a MIP in IBM ILOG Cplex. I have set the relative MIP GAP and absolute MIP gap to 0 ,but the gap was reported in engine log was upper than 0. also when I run the model by default values (1.0E-4,1.0E-6), the gap is reported in engine log is upper than 1.0E-4 (sometimes even 6%). and the surprising thing is that even the time of calculation was small(below 1sec). I think maybe other settings is needed besides mip gap to set it to zero to obtain optimal value of objective function. one another thing is that my other settings are as default. I appreciate if anyone can help me.
this is the result of one of my run(relative MIP GAP is set to 0 but the reported gap is 1.13% as you can see):
Nodes Cuts/
Node Left Objective IInf Best Integer Best Node ItCnt Gap
0 0 15619.2777 30 15619.2777 204
0 0 21532.4345 31 Cuts: 92 300
0 0 22240.7958 65 Cuts: 50 389
0 0 22374.7172 46 Cuts: 63 452
0 0 22428.5062 28 Cuts: 31 475
0 0 22447.7754 48 Cuts: 28 517
0 0 22486.3137 39 Cuts: 34 542
0 0 22486.3137 40 Cuts: 13 557
0 0 22486.3137 30 ZeroHalf: 4 558
0 0 22486.3137 28 Cuts: 15 583
* 0+ 0 23225.6696 22486.3137 583 3.18%
0 2 22486.3137 28 23225.6696 22486.3137 583 3.18%
Elapsed real time = 0.36 sec. (tree size = 0.01 MB, solutions = 1)
* 26 20 integral 0 22743.1173 22486.3137 1126 1.13%
GUB cover cuts applied: 2
Clique cuts applied: 23
Cover cuts applied: 9
Implied bound cuts applied: 105
Flow cuts applied: 1
Mixed integer rounding cuts applied: 30
Zero-half cuts applied: 74
Gomory fractional cuts applied: 3
Root node processing (before b&c):
Real time = 0.31
Parallel b&c, 4 threads:
Real time = 0.25
Sync time (average) = 0.02
Wait time (average) = 0.06
-------
Total (root+branch&cut) = 0.56 sec.
beforehand thanks for your help.
As Tim said in the comment, it is the last line of the log file that actually shows the final solution (upper and lower bounds) and not the last line of the tree log part.
Here is an example:
It seems that the problem is as follows:
CPLEX reports an optimal solution, with a default gap of 10e-4, but it reports 1.13%
You have found a better solution, in which you have checked that it is feasible
For the first bullet, CPLEX does not actually report that the gap is 1.13%. The gap was 1.13% when it was reported during the search.
The fact that it returns a solution, that should obey the default tolerances, is a proof that, using your formulation the optimal cannot be less than what is reported.
Since you are sure that there is a better feasible solution, you have a few options.
Try to inject your known solution to CPLEX before it starts the optimization. One way to do this is using goals. This might be somewhat complicated so you might want to ..
Print out the model and evaluate manually that the solution you claim is better is feasible with respect to the constraints you have entered (recommended method).
Add a constraint that the objective function value should be less than or equal to your solution's objective value augmented by a small quantity (something around 10e-3 should be OK). If your solution is feasible for formulation, you should get it, otherwise there is a bug and the solution is infeasible. I do not like this method because if there are multiple optimal solution you can get funny results, but usually it works).
All in all, try to debug the model and let us know. It is a fairly complicated model and it is easy to have missed something (i.e., adding a constraint that you did not mean to add).
If all this fail and you still find yourself wondering what is going on, you might want to submit your model to IBM's official forum. If there is indeed a bug in the solver they will take care of it and let you know.
I hope this helps

Streaming mulitple mp3 files to Icecast

I have a few thousand mp3 files on a web server that I need to stream to an Icecast 2.3.3 server instance running on the same server.
Each file is assigned to one or more categories. There are 7 categories in total. I'd like to have one mount per category.
I need to be able to add and remove files to categories. When a file is added / removed, I need to somehow merge the file into the category or shuffle the files in the category, after which I assume I'll need to restart the mount.
My question is: Is there a source application I could use that runs as a service on Windows OS that can automate this kind of thing?
Alternatively I could write a program to shuffle and merge these files together as one big "category" mp3 file, but would like to know if there's another way.
Any advice is very much appreciated.
Since you're just dealing with MP3 files, SHOUTcast sc_trans might be a good option for you.
http://wiki.winamp.com/wiki/SHOUTcast_DNAS_Transcoder_2
You can configure it to use a playlist (which you can generate programmatically), or have it read the directory and just run with the files as-is. Note that sc_trans doesn't support mount points, so you will have to configure Icecast to accept a SHOUTcast-style connection. This works, but will require you to run multiple instances of Icecast. If you'd like to stream everything on a single port later on, you can set up a master Icecast instance which relays all of the streams from the others.
There are plenty of other choices out there depending on your needs. Tools like SAM DJ allow full control over playlists and advertisements but can be overkill depending on what you need to do.
I typically find myself working with a diverse set of inputs, so I use VLC to playback and then some custom software to get this encoded and off to the streaming server. This isn't difficult to do, and you can even use VLC to do the encoding for you if you're crafty in configuring it.
I know it's old, and you most likely already found your solution. However, there may be more folks with this issue, so I throw in a a few considerations when you decide to write an own "shuffler" for MP3 files.
I would not use pure random for the task at hand: the likeliness of titles being played multiple times consecutively exists; you don't want that.
Also, you most likely have your titles sorted in some way, say
Artist A - Title 1
Artist A - Title 2
...
Artist B - Title 1
...
You most likely aim for diversity when shuffling, so you don't want to play the same artist twice consecutively.
I would read all filenames into an array with indices 0...n.
Find the artist with the most number of files, let the number be m.
Then find the next prime p, which is co-prime to n, but larger than m.
The generate a pseudo random number s in [0...n] just ONCE to find a starting song; this avoids to play the same starting sequence each time.
In a loop, do play song s, then set
s := (s + p) mod n
This is guaranteed to play all songs, and they will play just once, and multiple consecutive songs of the same artist are avoided.
Here's a little example for just 16 songs, capital letters are Artist, small letters song titles.
Aa Ab Ac Ba Bb Bc Bd Ca C2 Da Db Dc Dd Ea Fa Fb
n 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
The artists B and E have the most songs, hence
m := 4
You search for a prime number co-prime to 16 = 2 * 2 * 2 * 2, but larger than 4, and you find:
p := 5
You invoke the PRNG function once and obtain, say, 11, so s = 11 is the first song (s = 0) to be played. Then you loop:
Aa Ab Ac Ba Bb Bc Bd Ca Cb Da Db Dc Dd Ea Fa Fb
n 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
s 1 14 11 8 5 2 15 12 9 6 3 0 13 10 7 4
s is the played sequence:
Dc Aa Bc Db Fb Bb Da Fa Ba Cb Ea Ac Ca Dd Ab Bd
No artist repetition, no two songs after each other, much diversity.