I want to perform an analysis on average handle time (AHT) from a population and come up with a baseline which can be set as a goal.
AHT: The total time taken to resolve a specific task.
Sample Table:
task aht_in_seconds
123 2
234 10
456 20
678 5
789 3000
454 47867
213 6
998 60
547 135
565 776
325 342
567 223
565 334
666 360
As per my analysis, I am taking the mean and standard deviation of the population and trying to remove the outliers by using Z-score and box zenkin method.
As per Z-score (x-mean/sigma) method (-2 to +2 range (95%)) the standard deviation comes very high even after removing outliers.
Box-zenkin method: (for outlier removal)
lower limit: Q1-1.5*Interquartile Range
Upper limit: Q3+1.5*Interquartile Range
(Interquartile Range is derived by SAS Univariate)
As per the box-zenkin method, the SD is minimal but the final AHT value does not look realistic i.e very low as 150 seconds whereas the time & motion study gives a higer value of 250 seconds for a task
I am wondering if my methods are correct or missing something. Are there any other methods to remove outliers and perform the same analysis. Any guidance would be a great help.
Related
Currently I am dealing with a massive amount of Data in the original form of a list through combination. I am running conditions on each set of list through a for loop. Problem is this small for loop is taking hours with the data. I'm looking to optimize the speed by changing some functions or vectorizing it.
I know one of the biggest NO NOs is don't do Pandas or Dataframe operations in for loops but I need to sum up the columns and organize it a little to get what I want. It seems unavoidable.
So you have a better understanding, each list looks something like this when its thrown into a dataframe:
0
Name Role Cost Value
0 Johnny Tsunami Driver 1000 39
1 Michael B. Jackson Pistol 2500 46
2 Bobby Zuko Pistol 3000 50
3 Greg Ritcher Lookout 200 25
Name Role Cost Value
4 Johnny Tsunami Driver 1000 39
5 Michael B. Jackson Pistol 2500 46
6 Bobby Zuko Pistol 3000 50
7 Appa Derren Lookout 250 30
This is the current loop, any ideas?
for element in itertools.product(*combine_list):
combo = list(element)
df = pd.DataFrame(np.array(combo).reshape(-1,11))
df[[2,3]] = df[[2,3]].apply(pd.to_numeric)
if (df[2].sum()) <= 5000 and (df[3].sum()) > 190:
df2 = pd.concat([df2, df], ignore_index=True)
Couple things I've done that have sliced off some time but not enough.
*df[2].sum() to df[2].values.sum----its faster
*where the concat is in the if statement I've tried using append and also adding the dataframe together as a list...concat is actually 2 secs faster normally or it will end up being about the same speed.
*by the .apply(pd.to_numeric) changed it to .astype(np.int64) its faster as well.
I'm currently looking at PYPY and Cython as well but I want to start here first before I go through the headache.
I was asked to apply Two way merge sort on two files(files of records) ,
the algorithm explains the steps as follows :
Sort phase
1)The records on the file to be sorted are divided into several groups.
Each group is called a run, and each run fits into main memory.
2)An internal sort is applied to each run, 3)and the resulting sorted runs are distributed to two external files.
Merge Sort: 1) One run from each of the external files created in the sort phase merge into a larger runs of sorted records.
2)The result is stored in a third file.
3)The data is distributed back into the first two files, and merging continues until all records are in one large run.
I was able to apply Sort Phase only , so the current files is :
(supposed run contains 3 keys only )
file 1:
50 95 110 | 40 120 153 | 22 80 140
file 2:
10 36 100 | 60 70 130
here's the steps of merge phase
so if i will solve it theoretically will perform the following :
Merge Phase:
step1 :
file 3 :
10 36 50 95 100 110 | 40 60 70 120 130 153 | 22 80 140
file 1:
10 36 50 95 100 100 | 22 80 140
file 2 :
40 60 70 120 130 153
step 2 :
file 3 :
10 36 40 50 60 70 95 100 110 120 130 135 | 22 80 140
file 1 :
10 36 40 50 60 70 95 100 110 120 130 135
file 2:
22 80 140
step 3 :
file 3 :
10 22 36 40 50 60 70 80 95 100 110 120 130 135 140
one run stop sort complete
Now i need to apply merge phase so each key from each file compared to each other and output the smaller to file 3 , and in step 2 redistribute file 3 into two file then merge and sort until have one sort run
How i can apply such algorithm in c++ , i'm little bit confused about how can i determine the size of each run in every step.
As commented by Amdt Jonasson, the program needs to keep track of the run sizes and the end of data for each file. In your example it appears the initial run size is a fixed run size of 3 elements. A merge of two runs of size 3 will result in a single run of size 6 as shown in your steps. In this case, only a single instance of run size and the end of data in each file needs to be tracked.
If the sort is to be a stable sort (the original order preserved on equal keys), and the run size is variable, then an array of run counts for each file will be needed, or some way to denote the end of runs in the file, such as a text file, with a special character sequence used as a end of run indicator.
If the sort is not required to be stable, then an out of order sequence (smaller key value after a larger key value) can be used to indicate the end of a run. The risk here is that two or more runs will appear to be a single run if the runs happen to be in order, which will lose stability and unbalance the run count on the files.
This is a two way merge sort using 3 files. If you use a 4th file, then each merge of runs can alternate between 2 output files, eliminating the need split up the runs after each merge pass.
An alternative for doing a 2 way merge sort with 3 files is a polyphase merge sort, but it's complicated, probably beyond what would be expected for a class assignment, and more of a "legacy" algorithm used back in the days of tape based sorts.
So we have a timeline of T days in which some tasks have to be performed.
Every task has a penalty score. If the task is not performed in the given timeline , it's score adds up in the final penalty score. Every task can be performed only after it's given starting time.
The input will be given in the format:
T
Score Quantity_of_task Starting_time
For eg :
T = 10
140 5 4
This means that 5 tasks with penalty score 140 have to be performed from 4th day onwards.
You can perform at most 1 task on a particular day.
The goal is to minimize the final penalty score.
What I tried to do:
Example -
T = 10
Input size = 5
150 4 1
120 4 3
200 2 7
100 10 5
50 5 1
I sorted the list according to the penalty score , and greedily assigned the tasks with high penalty score to their corresponding days,i.e
2 tasks with highest score 200 are assigned to days 7 and 8
4 tasks with next highest score 150 are assigned to 1,2,3,4 days
4 tasks with next highest score 120 are assigned to 5,6,9,10 days
which gives the schedule as
150 150 150 150 120 120 200 200 120 120
Left out tasks:
10 tasks with 100 score = 1000 penalty
5 tasks with 50 score = 250 penalty
Final penalty = 1250.
This requires O(T * input_size). Is there a more elegant and optimized way of doing it?
Both input size and T have a constraint of 10^5.
Thanks.
If you store the available days in an ordered set, then you can perform your algorithm much faster.
For example, C++ provides an ordered set with a lower_bound method that will find in O(logn) time the first available day after the starting time.
Overall this should give an O(nlogn) algorithm where n = T+input_size.
For example, I suspect that when you have your 4 tasks of penalty 120 to assign from day 3 onwards, your current code will loop over days 3,4,5,etc. until you find a day that has not been assigned. You can now replace this O(n) loop with a single O(logn) call to lower_bound to find the first unassigned day. When you greedily assign the days, you should also remove them from the set so they won't be assigned twice.
Note that there are only T days so there will be at most T day assignments. For example, suppose all tasks have starting time 1, and quantity T. Then the first task will take O(Tlogn) time to assign, but all subsequent tasks will only need a single call to lower_bound (because there are no days left to assign), so will take O(logn) each.
My DataFrame 3 fields are account ,month and salary.
account month Salary
1 201501 10000
2 201506 20000
2 201506 20000
3 201508 30000
3 201508 30000
3 201506 10000
3 201506 10000
3 201506 10000
3 201506 10000
I am doing groupby on Account and Month and calculating sum of salary for group. Then removing duplicates.
MyDataFrame['salary'] = MyDataFrame.groupby(['account'], ['month'])['salary'].transform(sum)
MyDataFrame = MyDataFrame.drop_duplicates()
Expecting output like below:
account month Salary
1 201501 10000
2 201506 40000
3 201508 60000
3 201506 40000
It works well for few records. I tried same for 600 Million records and it is in progress since 4-5 hours. Initially when I loaded data using pd.read_csv() data acquired 60 GB RAM, till 1-2 hour RAM usages was in between 90 to 120 GB. After 3 hours process is taking 236 GB RAM and it is still running.
Please suggest if any other alternative faster way is available for this.
EDIT:
Now 15 Minutes in df.groupby(['account', 'month'], sort=False)['Salary'].sum()
Just to follow up on chrisb's answer and Alexander's comment, you indeed will get more performance out of the .sum() and .agg('sum') methods. Here's a Jupyter %%timeit output for the three:
So, the answers that chrisb and Alexander mention are about twice as fast on your very small example dataset.
Also, according to the Pandas API documentation, adding the kwarg sort=False will also help performance. So, your groupby should look something like df.groupby(['account', 'month'], sort=False)['Salary'].sum(). Indeed, when I ran it, it was about 10% faster than the runs shown in the above image.
Unless I'm misunderstanding something, you're really doing an aggregation - transform is for when you need the data in the shape as the original frame. This should be somewhat faster and does it all in one step.
df.groupby(['account', 'month'])['Salary'].agg('sum')
Might be worth downloading the development version of Pandas 0.17.0. They are unlocking the GIL, which controls multi threading. It's going to be natively implemented in the groupby and this blog post suggested speed increases of 3x on a group-mean example.
http://continuum.io/blog/pandas-releasing-the-gil
http://pandas.pydata.org/
I am solving a MIP in IBM ILOG Cplex. I have set the relative MIP GAP and absolute MIP gap to 0 ,but the gap was reported in engine log was upper than 0. also when I run the model by default values (1.0E-4,1.0E-6), the gap is reported in engine log is upper than 1.0E-4 (sometimes even 6%). and the surprising thing is that even the time of calculation was small(below 1sec). I think maybe other settings is needed besides mip gap to set it to zero to obtain optimal value of objective function. one another thing is that my other settings are as default. I appreciate if anyone can help me.
this is the result of one of my run(relative MIP GAP is set to 0 but the reported gap is 1.13% as you can see):
Nodes Cuts/
Node Left Objective IInf Best Integer Best Node ItCnt Gap
0 0 15619.2777 30 15619.2777 204
0 0 21532.4345 31 Cuts: 92 300
0 0 22240.7958 65 Cuts: 50 389
0 0 22374.7172 46 Cuts: 63 452
0 0 22428.5062 28 Cuts: 31 475
0 0 22447.7754 48 Cuts: 28 517
0 0 22486.3137 39 Cuts: 34 542
0 0 22486.3137 40 Cuts: 13 557
0 0 22486.3137 30 ZeroHalf: 4 558
0 0 22486.3137 28 Cuts: 15 583
* 0+ 0 23225.6696 22486.3137 583 3.18%
0 2 22486.3137 28 23225.6696 22486.3137 583 3.18%
Elapsed real time = 0.36 sec. (tree size = 0.01 MB, solutions = 1)
* 26 20 integral 0 22743.1173 22486.3137 1126 1.13%
GUB cover cuts applied: 2
Clique cuts applied: 23
Cover cuts applied: 9
Implied bound cuts applied: 105
Flow cuts applied: 1
Mixed integer rounding cuts applied: 30
Zero-half cuts applied: 74
Gomory fractional cuts applied: 3
Root node processing (before b&c):
Real time = 0.31
Parallel b&c, 4 threads:
Real time = 0.25
Sync time (average) = 0.02
Wait time (average) = 0.06
-------
Total (root+branch&cut) = 0.56 sec.
beforehand thanks for your help.
As Tim said in the comment, it is the last line of the log file that actually shows the final solution (upper and lower bounds) and not the last line of the tree log part.
Here is an example:
It seems that the problem is as follows:
CPLEX reports an optimal solution, with a default gap of 10e-4, but it reports 1.13%
You have found a better solution, in which you have checked that it is feasible
For the first bullet, CPLEX does not actually report that the gap is 1.13%. The gap was 1.13% when it was reported during the search.
The fact that it returns a solution, that should obey the default tolerances, is a proof that, using your formulation the optimal cannot be less than what is reported.
Since you are sure that there is a better feasible solution, you have a few options.
Try to inject your known solution to CPLEX before it starts the optimization. One way to do this is using goals. This might be somewhat complicated so you might want to ..
Print out the model and evaluate manually that the solution you claim is better is feasible with respect to the constraints you have entered (recommended method).
Add a constraint that the objective function value should be less than or equal to your solution's objective value augmented by a small quantity (something around 10e-3 should be OK). If your solution is feasible for formulation, you should get it, otherwise there is a bug and the solution is infeasible. I do not like this method because if there are multiple optimal solution you can get funny results, but usually it works).
All in all, try to debug the model and let us know. It is a fairly complicated model and it is easy to have missed something (i.e., adding a constraint that you did not mean to add).
If all this fail and you still find yourself wondering what is going on, you might want to submit your model to IBM's official forum. If there is indeed a bug in the solver they will take care of it and let you know.
I hope this helps