Complex MapReduce Query with RavenDB [closed] - mapreduce

Question

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
Hope you can help me !!
I am collecting tweets, which have a created_at date (DataPublicacao), and some Hashtags. Each tweet refers to a broadcaster (redeId), and a show (programaId).
I want to query the database for the 20 most used hashtags in a certain period.
I have to map each hashtag, when it was used, and to which broadcaster and tv show it refers to.
Then, I need to be able to count the occurrences of each hashtag in a certain period (I dont know how).
public class Tweet : IModelo
{
public string Id { get; set; }
public string RedeId { get; set; }
public string ProgramaId { get; set; }
public DateTime DataPublicacao { get; set; }
public string Conteudo { get; set; }
public string Aplicacao { get; set; }
public Autor Autor { get; set; }
public Twitter.Monitor.Dominio.Modelo.TweetJson.Geo LocalizacaoGeo { get; set; }
public Twitter.Monitor.Dominio.Modelo.TweetJson.Place Localizacao { get; set; }
public Twitter.Monitor.Dominio.Modelo.TweetJson.Entities Entidades { get; set; }
public string Imagem { get; set; }
public Autor Para_Usuario { get; set; }
public string Retweet_Para_Status_Id { get; set; }
}
And the "entities" are hashtags, usermentions, and urls.
I tried to group the hashtags by broadcaster, tv show, and text, and listing the dates of the occurrences. Then, I have to transform the results, so I can count the occurrences on that period.
public class EntityResult
{
public string hashtagText { get; set; }
public string progId { get; set; }
public string redeId { get; set; }
public int listCount { get; set; }
}
public class HashtagsIndex : AbstractIndexCreationTask<Tweet, HashtagsIndex.ReduceResults>
{
public class ReduceResults
{
public string hashtagText { get; set; }
public DateTime createdAt { get; set; }
public string progId { get; set; }
public string redeId { get; set; }
public List<DateTime> datesList { get; set; }
}
public HashtagsIndex()
{
Map = tweets => from tweet in tweets
from hts in tweet.Entidades.hashtags
where tweet.Entidades != null
select new
{
createdAt = tweet.DataPublicacao,
progId = tweet.ProgramaId,
redeId = tweet.RedeId,
hashtagText = hts.text,
datesList = new List<DateTime>(new DateTime[] { tweet.DataPublicacao })
};
Reduce = results => from result in results
group result by new { result.progId, result.redeId, result.hashtagText }
into g
select new
{
createdAt = DateTime.MinValue,
progId = g.Key.progId,
redeId = g.Key.redeId,
hashtagText = g.Key.hashtagText,
datesList = g.ToList().Select(t => t.createdAt).ToList()
};
}
}
And the query I made so far is:
var hashtags2 = session.Query<dynamic, HashtagsIndex>().Customize(t => t.TransformResults((query, results) =>
results.Cast<dynamic>().Select(g =>
{
Expression<Func<DateTime, bool>> exp = o => o >= dtInit && o <= dtEnd;
int count = g.Where(exp);
return new EntityResult
{
redeId = g.redeId,
progId = g.progId,
hashtagText = g.hashtagText,
listCount = count
};
}))).Take(20).ToList();
Now I need to OrderByDescending(t=>t.count), so I cant Take(20) most used hashtags on that period.
How do I do that?

Answer 1

Is it possible to filter items before the mapreduce process?
A map/reduce index is just like any other index. All documents are processed through all indexes, always. So when phrased with "before" like you asked, the answer is clearly "no".
But I think you are just interested in filtering items during the indexing, and that is easily done in the map:
Map = items => from item in items
where item.foo == whatever // this is how you filter
select new
{
// whatever you want to map
}
This index will process all documents, but the resulting index will only contain items that match the filter you specified in the where clause.
Is it possible to subsequently group by features, like users by age, and then by region
Grouping is done in the reduce step. That is what map/reduce is all about.
My advice to you (and I mean no disrespect by this), is to walk before you try to run. Build a simple prototype or set of unit tests, and try first just basic storage and retrieval. Then try basic indexing and querying. Then try a simple map reduce, such as counting all your tweets. Only then should you attempt an advance map/reduce with other groupings. And if you run into trouble, then you will have code you can post here for help.
Is it possible?
Of course. Anything is possible. :)