I have an actor which receives WeatherConditions and pushes it (by using OfferAsync) it to source. Currently it is setup to run for each item it receives (it stores it to db).
public class StoreConditionsActor : ReceiveActor
{
public StoreConditionsActor(ITemperatureDataProvider temperatureDataProvider)
{
var materializer = Context.Materializer();
var source = Source.Queue<WeatherConditions>(10, OverflowStrategy.DropTail);
var graph = source
.To(Sink.ForEach<WeatherConditions>(conditions => temperatureDataProvider.Store(conditions)))
.Run(materializer);
Receive<WeatherConditions>(i =>
{
graph.OfferAsync(i);
});
}
}
What I would like to achieve is:
Run it only once every N minutes and store average value of WeatherConditions from all items received in this N minutes time window
If item received matches certain condition (i.e. item temperature is 30% higher than previous item's temperature) run it despite of being "hidden" in time window.
I've been trying ConflateWithSeed, Buffer, Throttle but neither seems to be working (I'm newbie in Akka / Akka Streams so I may be missing something basic)
This answer uses Akka Streams and Scala, but perhaps it will inspire your Akka.NET solution.
The groupedWithin method could meet your first requirement:
val queue =
Source.queue[Int](10, OverflowStrategy.dropTail)
.groupedWithin(10, 1 second)
.map(group => group.sum / group.size)
.toMat(Sink.foreach(println))(Keep.left)
.run()
Source(1 to 10000)
.throttle(10, 1 second)
.mapAsync(1)(queue.offer(_))
.runWith(Sink.ignore)
In the above example, up to 10 integers per second are offered to the SourceQueue, which groups the incoming elements in one-second bundles and calculates the respective averages of each bundle.
As for your second requirement, you could use sliding to compare an element with the previous element. The below example passes an element downstream only if it is at least 30% greater than the previous element:
val source: Source[Int, _] = ???
source
.sliding(2, 1)
.collect {
case Seq(a, b) if b >= 1.3 * a => b
}
.runForeach(println)
Related
I have a column of 12.000 VAT numbers, and I add a custom Column with code to get data from web API. I want it to pause for 1 second after every 100th row. how do i manage this?
current code:
Function.InvokeAfter(()=>
Json.Document(
Web.Contents("https://data.brreg.no/enhetsregisteret/api/enheter/"&
Number.ToText([Kunde ID]), [Headers=[Accept="application/json"]]))
,#duration(0,0,0,1))
See this
https://community.powerbi.com/t5/Desktop/Query-to-Scrap-URLs-with-a-pause-every-xx-URLs/td-p/352809
which basically involves a wait function you call every 100 times before hitting the API
let Wait = (seconds as number, action as function) =>
if (List.Count(List.Generate(() => DateTimeZone.LocalNow() + #duration(0,0,0,seconds), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0)
then null else action()
in Wait
I have this pre-request script and using runner to send bulk requests each second
const moment = require('moment');
postman.setEnvironmentVariable("requestid", moment().format("0223YYYYMMDDHHmmss000000"));
I need the “requestid” to be unique every time.
first request: "022320221115102036000001"
second request: "022320221115102037000002"
third request: "022320221115102038000003"
.
.
.
and so on until let’s say 1000 requests.
Basically, I need to make the last 6 digits dynamic.
Your answer can be found on this postman request I've created for you. There's many ways to achieve this, given the little information provided, I've defaulted to:
Set a baseline prefix (before the last 6 numbers)
Give a start number for the last 6 numbers
If there IS NOT a previous variable stored initialized with the values above
If there IS a previous variable stored just increment it by one.
The variable date is your final result, current is just the increment
You can see here sequential requests:
And here is the code, but I would test this directly on the request I've provided above:
// The initial 6 number to start with
// A number starting with 9xxxxxx will be easier for String/Number converstions
const BASELINE = '900000'
const PREFIX = '022320221115102036'
// The previous used value if any
let current = pm.collectionVariables.get('current')
// If there's a previous number increment that, otherwise use the baseline
if (isNaN(current)) {
current = BASELINE
} else {
current = Number(current) + 1
}
const date = PREFIX + current
// Final number you want to use
pm.collectionVariables.set('current', current)
pm.collectionVariables.set('date', PREFIX + date)
console.log(current)
console.log(date)
Another question is if there is any better way to write this method?
Public decimal CalculateTotalPrice(List<product> items)
{
decimal totalPrice = 0.m;
foreach(Product p in items)
{
if(p.Offer == "")
calc = new DefaultCalc();
else if(p.Offer == "BuyOneGetOneFree")
calc = new BuyOneGetOneFreeCalc();
else if(p.Offer == "ThreeInPriceOfTwo")
calc = new ThreeInPriceOfTwoCalc()
totalPrice += calc.Calculate(p.Quantity, p.UnitPrice);
}
return totalPrice;
}
You should probably review Polly Want a Message, by Sandi Metz
How to unit test a method that is having multiple object creation in switch statement?
An important thing to notice here is that the switch statement is an implementation detail. From the point of view of the caller, this thing is just a function
Public decimal CalculateTotalPrice(List<product> items);
If the pricing computations are fixed, you can just use the usual example based tests:
assertEquals(expectedPrice, CalculateTotalPrice(items));
But if they aren't fixed, you can still do testing based on the properties of the method. Scott Wlaschin has a really good introduction to property based testing. Based on the logic you show here, there are some things we can promise about prices, without knowing anything about the strategies in use
the price should always be greater than zero.
the price of a list of items is the same as the sum of the prices of the individual items.
if there is any better way to write this method?
You might separate choosing the pricing strategy from using the strategy. As Sandi notes, that sort of construct often shows up in more than once place.
foreach(Product p in items)
{
calc = pricing(p.Offer);
totalPrice += calc.Calculate(p.Quantity, p.UnitPrice);
}
"pricing" would then become something that you pass into this function (either as an argument, or as a dependency).
In effect, you would end up with three different kinds of test.
Checks that pricing returns the right pricing strategy for each offer.
Checks that each strategy performs its own calculation correctly.
Checks that CalculateTotalPrice compute the sum correctly.
Personally, I prefer to treat the test subject as a single large black box, but there are good counter arguments. Horses for courses.
Constructors can not be mocked (at least with free mocking frameworks).
Write tests without mocking as far as your tests run fast and test case setup is not very very complicated.
In your particular case you should be able to write tests without mocking.
Prepare data
var products = new List<Product>
{
new Product { Quantity = 10, UnitPrice = 5.0m, Offer = "" },
new Product { Quantity = 2, UnitPrice = 3.0m , Offer = "BuyOneGetOneFree" },
new Product { Quantity = 3, UnitPrice = 2.0m , Offer = "ThreeInPriceOfTwo" },
}
// prepare expected total
var expected = 57.0m; // 10 * 50.0 + 1 * 3.0 + 2 * 2.0
// Run the test
var actual = CalculateTotalPrice(products);
actual.Should().Be(expected); // pass Ok.
With this approach tests will not depend on implementation details.
You will be able to freely play with designs without rewriting tests every time you change your implementation logic.
The other answers are technically fine, but I would suggest one thing:
if(p.Offer == "")
calc = new DefaultCalc();
else if(p.Offer == "BuyOneGetOneFree")
calc = new BuyOneGetOneFreeCalc();
else if(p.Offer == "ThreeInPriceOfTwo")
calc = new ThreeInPriceOfTwoCalc()
should absolutely go into its own method/scope/whatever.
You are mapping a string to a specific calculator. That should happen in one place, and one place only. You see, first you do that here. Then some method method comes around that needs the same mapping. So you start duplicating.
We are using DynamoDB for counting user actions and an item must be either inserted or updated, depending on whatever it's already exists. The code must also update a counter. Right now we do this with 2 steps:
using (var client = AWSClientFactory.CreateAmazonDynamoDBClient(RegionEndpoint.USEast1))
{
var table = Table.LoadTable(client, TableName);
var item = await table.GetItemAsync(id);
if (item == null)
{
// row not exists -> insert & return 1
var document = new Document();
document["Id"] = id;
document["Counter"] = 1;
await table.PutItemAsync(document);
return 1;
}
// row exists -> increment counter & update
var counter = item["Counter"].AsInt();
item["Counter"] = counter + 1;
await table.UpdateItemAsync(item);
return counter + 1;
}
The problem with the code is that it increases latency times & server load. I would prefer to do this with a single operation. I think this should be possible with conditional expressions but I cannot figure out how to do this using .NET SDK.
Be careful about incrementing counters yourself, as you could have race conditions if multiple instances of your app can increment the counter. Instead use DynamoDB Atomic Counters. For example, my ruby code calls the UpdateItem API with the following (older) way of incrementing counters:
{"counter" => {value: {n: "1"}, action: "ADD"}}
The newer way is to use an Update Expression, which I haven't implemented yet. Also, if the counter/item doesn't already exist, it will assume the value is 0 and increment the counter to 1.
you have a race condition in your code.
it's possible that 2 different worker create the item at the same time.
the recommended pattern for what you are trying to do is:
create if not exists operation for the item.
atomic counter update on "Count"
So instead of 3 operations (get, put, update) - that also have a race condition - in this case you will only have 2 operations (and the correct behavior)
hope this helps.
*Basically I'm trying to order objects by their score over the last hour.
I'm trying to generate an hourly votes sum for objects in my database. Votes are embedded into each object. The object schema looks like this:
{
_id: ObjectId
score: int
hourly-score: int <- need to update this value so I can order by it
recently-voted: boolean
votes: {
"4e4634821dff6f103c040000": { <- Key is __toString of voter ObjectId
"_id": ObjectId("4e4634821dff6f103c040000"), <- Voter ObjectId
"a": 1, <- Vote amount
"ca": ISODate("2011-08-16T00:01:34.975Z"), <- Created at MongoDate
"ts": 1313452894 <- Created at timestamp
},
... repeat ...
}
}
This question is actually related to a question I asked a couple of days ago Best way to model a voting system in MongoDB
How would I (can I?) run a MapReduce command to do the following:
Only run on objects with recently-voted = true OR hourly-score > 0.
Calculate the sum of the votes created in the last hour.
Update hourly-score = the sum calculated above, and recently-voted = false.
I also read here that I can perform a MapReduce on the slave DB by running db.getMongo().setSlaveOk() before the M/R command. Could I run the reduce on a slave and update the master DB?
Are in-place updates even possible with Mongo MapReduce?
You can definitely do this. I'll address your questions one at a time:
1.
You can specify a query along with your map-reduce, which filters the set of objects which will be passed into the map phase. In the mongo shell, this would look like (assuming m and r are the names of your mapper and reducer functions, respectively):
> db.coll.mapReduce(m, r, {query: {$or: [{"recently-voted": true}, {"hourly-score": {$gt: 0}}]}})
2.
Step #1 will let you use your mapper on all documents with at least one vote in the last hour (or with recently-voted set to true), but not all the votes will have been in the last hour. So you'll need to filter the list in your mapper, and only emit those votes you wish to count:
function m() {
var hour_ago = new Date() - 3600000;
this.votes.forEach(function (vote) {
if (vote.ts > hour_ago) {
emit(/* your key */, this.vote.a);
}
});
}
And to reduce:
function r(key, values) {
var sum = 0;
values.forEach(function(value) { sum += value; });
return sum;
}
3.
To update the hourly scores table, you can use the reduceOutput option to map-reduce, which will call your reducer with both the emitted values, and the previously saved value in the output collection, (if any). The result of that pass will be saved into the output collection. This looks like:
> db.coll.mapReduce(m, r, {query: ..., out: {reduce: "output_coll"}})
In addition to re-reducing output, you can use merge which will overwrite documents in the output collection with newly created ones (but leaving behind any documents with an _id different than the _ids created by your m-r job), replace, which is effectively a drop-and-create operation and is the default, or use {inline: 1}, which will return the results directly to the shell or to your driver. Note that when using {inline: 1}, your results must fit in the size allowed for a single document (16MB in recent MongoDB releases).
(4.)
You can run map-reduce jobs on secondaries ("slaves"), but since secondaries cannot accept writes (that's what makes them secondary), you can only do this when using inline output.