I am a complete to newbie to machine learning but I am trying to achieve following:
I have a series of data, let's say:
ie:
[ {'day1', 5}, {'day2', 3}, {'day3', 8} ... {'day100', 8}]
that at the end has a final result:
ie: pass/fail/other result
I would like to predict the most likely end result by using a partial set of data
ie: [{'day1', 3}, {'day2', 1}, {'day3', 6} ... {'day50', 8}]
What would be the best way to achieve this and if possible, do you have any examples or hint of where to look into that could help someone like me with no experience in machine learning
It is a time series prediction.
I'd suggest you check if your time series contain trend or seasonality. Then try conventional methods of time-series prediction as moving average (or sliding window) when you, for example, predict day 6 based on days 1-5, then day 7 based on days 2-6 etc. You may also try Exponential Smoothing and ARIMA.
Concerning ml.net, the following tutorial is the best solution for you
as it uses a Singular Spectrum Analysis and thus takes into consideration a trend and possible seasonality to forecast values. You can employ it by using
ssaforecastingestimator: put window = 25 and horizon=1 if you want to use the first 25 days to predict the value for day 26 (prediction one step forward). I suggest you take a smaller window though (5-7 days).
Related
For a project I am working on, which uses annual financial reports data (of multiple categories) from companies which have been successful or gone bust/into liquidation, I previously created a (fairly well performing) model on AWS Sagemaker using a multiple linear regression algorithm (specifically, the AWS stock algorithm for logistic regression/classification problems - the 'Linear Learner' algorithm)
This model just produces a simple "company is in good health" or "company looks like it will go bust" binary prediction, based on one set of annual data fed in; e.g.
query input: {data:[{
"Gross Revenue": -4000,
"Balance Sheet": 10000,
"Creditors": 4000,
"Debts": 1000000
}]}
inference output: "in good health" / "in bad health"
I trained this model by just ignoring what year for each company the values were from and pilling in all of the annual financial reports data (i.e. one years financial data for one company = one input line) for the training, along with the label of "good" or "bad" - a good company was one which has existed for a while, but hasn't gone bust, a bad company is one which was found to have eventually gone bust; e.g.:
label
Gross Revenue
Balance Sheet
Creditors
Debts
good
10000
20000
0
0
bad
0
5
100
10000
bad
20000
0
4
100000000
I hence used these multiple features (gross revenue, balance sheet...) along with the label (good/bad) in my training input, to create my first model.
I would like to use the same features as before as input (gross revenue, balance sheet..) but over multiple years; e.g take the values from 2020 & 2019 and use these (along with the eventual company status of "good" or "bad") as the singular input for my new model. However I'm unsure of the following:
is this an inappropriate use of logistic regression Machine learning? i.e. is there a more suitable algorithm I should consider?
is it fine, or terribly wrong to try and just use the same technique as before, but combine the data for both years into one input line like:
label
Gross Revenue(2019)
Balance Sheet(2019)
Creditors(2019)
Debts(2019)
Gross Revenue(2020)
Balance Sheet(2020)
Creditors(2020)
Debts(2020)
good
10000
20000
0
0
30000
10000
40
500
bad
100
50
200
50000
100
5
100
10000
bad
5000
0
2000
800000
2000
0
4
100000000
I would personally expect that a company which has gotten worse over time (i.e. companies finances are worse in 2020 than in 2019) should be more likely to be found to be a "bad"/likely to go bust, so I would hope that, if I feed in data like in the above example (i.e. earlier years data comes before later years data, on an input line) my training job ends up creating a model which gives greater weighting to the earlier years data, when making predictions
Any advice or tips would be greatly appreciated - I'm pretty new to machine learning and would like to learn more
UPDATE:
Using Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNN) is one potential route I think I could try taking, but this seems to commonly just be used with multivariate data over many dates; my data only has 2 or 3 dates worth of multivariate data, per company. I would want to try using the data I have for all the companies, over the few dates worth of data there are, in training
I once developed a so called Genetic Time Series in R. I used a Genetic Algorithm which sorted out the best solutions from multivariate data, which were fitted on a VAR in differences or a VECM. Your data seems more macro economic or financial than user-centric and VAR or VECM seems appropriate. (Surely it is possible to treat time-series data in the same way so that we can use LSTM or other approaches, but these are very common) However, I do not know if VAR in differences or VECM works with binary classified labels. Perhaps if you would calculate a metric outcome, which you later label encode to a categorical feature (or label it first to a categorical) than VAR or VECM may also be appropriate.
However you may add all yearly data points to one data points per firm to forecast its survival, but you would loose a lot of insight. If you are interested in time series ML which works a little bit different than for neural networks or elastic net (which could also be used with time series) let me know. And we can work something out. Or I'll paste you some sources.
Summary:
1.)
It is possible to use LSTM, elastic NEt (time points may be dummies or treated as cross sectional panel) or you use VAR in differences and VECM with a slightly different out come variable
2.)
It is possible but you will loose information over time.
All the best,
Patrick
This question already has answers here:
Given a set of intervals, how to find the maximum number of intersections among them,
(3 answers)
Closed 3 years ago.
On the input, we are given a number N stating how many presentations are to be given, which is followed by N rows of start and end times given in HHMM format
Example:
3
0800 0900
0830 1000
0900 1030
The code must calculate the maximum nuber of occupied rooms (one room can host only one presentation at a time, therefore expected output to the provided example is 2.
My first idea was to create a table of bools 1440 x N (number of minutes in a day by number of presentations) and fill each minute when a presentation is being held, and later go column by column and find the maximum value of presentations at a time. It works, but I'm sure it can be done faster and better. Can someone suggest how to do it in a better manner?
Pretty simple actually: we just simulate the process. First of all, it doesn't matter which presentations are going on right now. All we care about is the number of presentations happening. So we'll just have a counter which we update when a presentation starts or ends.
We could iterate over every minute for the simulation, but our counter only changes when a presentation starts or ends, so we can just make a big list of all the start and end events, sort the list by time, and iterate through the list adjusting our counter appropriately.
The way you propose needs 1440 x 3 = 4320 values. As you can tell, that's very inefficient. A better way is to store just the minutes that are actually required. To save even less values, divide the time into slots of 30 minutes.
Now use a dictionary (std::map) to keep a count of how many presentations are during the same time slot. For your example this gives:
std::map<std::string, int> slots =
{
{"0800", 1},
{"0830", 2},
{"0900", 2},
{"0930", 2},
{"1000", 1},
{"1030", 1},
}
I'll let you figure out how to implement this.
I have a dataset from which I would like to detect recurring patterns (i.e: daily, weekly, monthly). The dataset only contains a time stamp (datetime), and the spacing is non-uniform.
The observations in the data reflect the exact time when this one person passes my window. He does this several times a day (on a single day he walks by my window approx 10-30 times), and I am trying to see, if there is any pattern (there might also be some seasonality, sudden changes in previous behavior and other interesting stuff going on).
Does anyone have a suggestion for a statistical model/approach that might be helpful in figuring out if there is any pattern in this behavior? Hopefully, I’ll be able to predict when he will pass my window again ;)
How would you approach this?
Any help would really be appreciated.
I am generating alerts by reading dataset for KPI (key performance indicator) . My algorithm is looking into historical data and based on that I am able to capture if there's sudden spike in data. But I am generating false alarms . For example KPI1 is historically at .5 but reaches value 12, which is kind of spike .
Same way KPI2 also reaches from .5 to 12. But I know that KPI reaching from .5 to 12 is not a big deal and I need not to capture that . same way KPI2 reaching from .5 to 12 is big deal and I need to capture that.
I want to train my program to understand what is high value , low value or normal value for each KPI.
Could you experts tell me which is best ML algorithm is for this and any package in python I need to explore?
This is the classification problem. You can use classic logistic regression algorithm to classify any given sample into either high value, low value or normal value.
Quoting from the Wikipedia,
In statistics, multinomial logistic regression is a classification
method that generalizes logistic regression to multiclass problems,
i.e. with more than two possible discrete outcomes. That is, it is
a model that is used to predict the probabilities of the different
possible outcomes of a categorically distributed dependent variable,
given a set of independent variables (which may be real-valued,
binary-valued, categorical-valued, etc.)
To perform multi-class classification in python, sklearn library can be useful.
http://scikit-learn.org/stable/modules/multiclass.html
I'm currently implementing a histogram that will show a very large scale data using Qt and I have some doubts about which data structure(s) I should be using for my problem. I will be displaying the amount of queries received from users of an application and the way I should display is as follows -in a single application that will show different histograms upon clicking different "show me this data etc." buttons-
1) Display the histogram of total queries per every month -4 months of data here, I
kept four variables and incremented them as I caught queries belonging to those months
in the CSV file-
2) Display the histogram of total queries per every single day in a selected month -I was thinking of using 4 QVectors to represent the months for this one, incrementing every element of the vector (day), as I come by that specific day -e.g. the vector represents the month of August and whenever I come across a data with 2011-08-XY , I will increment the (XY + 1)th element of that vector by 1- my second alternative is to use 4 QLinkedList's for the sake of better complexity but I'm not sure if the ways I've come up with are efficient enough and I'm willing to listen to any other idea.
3) Here's where things get a bit complicated. Display the histogram of total queries per every hour in a selected day and month. The data represented is multiplied in a vast manner and I don't know which data structure -or combination of structures- I should use to implement this one. A list of lists perhaps?
Any ideas on my problems at 2) and 3) would be helpful, Thanks in advance.
Actually, it shouldn't be too unmanageable to always do queries per hour. Assuming that the number of queries per hour is never greater than the maximum int value, that's only 24 ints per day = 32 bits or 64 depending on your machine. Assuming 32 bits, then you could get up to 28 years worth of data per MB.
There's no need to transfer the month/year - your program can work that out. Just assign hour 0 to the earliest point in your data, which you keep as a constant, then work out the date based on hours passed since then.
This avoids having to have a list of lists or anything fancy - just use an array where each address contains the number of hours since hour 0, and the number of queries for that hour.
Why don't you simply use a classical database?
When you start asking these kind of question I think it is a good time to consider a more robust structure.There are multiple data structures implemented inside any DB, optimized either for different access type. You should considerate at least lookup, insertion, deletion, range queries. There is no structure which is better than the others in all costs, so there is always a trade-off.
Qt has some database classes you can use. I never used the Qt SQL library, but I think you should give it a shot. Fortunately, there is a Qt SQL programming guide at the end of the page linked.