How to delete mass records using Map/reduce script? - mapreduce

I have created a Map/Reduce script which will fetch customer invoices and delete it. If I am creating saved search in UI based on the below criteria, it shows 4 million records. Now, if I run the script, execution stops before completing the "getInputData" stage as maximum storage limit of this stage is 200Mb. So, I want to fetch first 4000 records out of 4 million and execute it and schedule the script for every 15 mins. Here is the code of first stage (getInputData) -
var count=0;
var counter=0;
var result=[];
var testSearch = search.create({
type: 'customrecord1',
filters: [ 'custrecord_date_created', 'notonorafter', 'sta​rtO​fLa​stM​ont​h' ],
columns: [ 'internalid' ]
});
do{
var resultSearch = testSearch.run().getRange({
start : count,
end : count+1000
});
for(var arr=0;arr<resultSearch.length;arr++){
result.push(resultSearch[arr]);
}
counter = count+counter;
}while(resultSearch.length >= 1000 && counter != 4000);
return result;
During creating the saved search, it is taking long time, is there any work around where we can filter first 4000 records during saved search creation?

Why not a custom mass update?
It would be a 5-10 line script that grabs the internal id and record type of the current record in the criteria of the mass update then deletes the record.

I believe this is what search.runPaged() and pagedData.fetch() is for.
search.runPaged runs the current search and returns summary information about paginated results - it does not give you the result set or save the search.
pagedData.fetch retrieves the data within the specified page range.

If you are intent on the Map/Reduce you can just return your created search. Netsuite will run it and pass each line to the next phase. You can even use a saved search where you limit the number of lines and then in your summarize phase re-trigger the script if there's anything left to do.
The 4k record syntax though is:
var toDelete = [];
search.run().each(function(r){
toDelete.push(r.id);
return toDelete.length < 4000;
});
return toDelete;
finally I normally do this as scheduled mass update. It will tend to interfere less with any production scheduled and map/reduce scripts.
/**
* #NApiVersion 2.x
* #NScriptType MassUpdateScript
*/
define(["N/log", "N/record"], function (log, record) {
function each(params) {
try {
record.delete({
type: params.type,
id: params.id
});
log.audit({ title: 'deleted ' + params.type + ' ' + params.id, details: '' });
}
catch (e) {
log.error({ title: 'deleting: ' + params.type + ' ' + params.id, details: (e.message || e.toString()) + (e.getStackTrace ? (' \n \n' + e.getStackTrace().join(' \n')) : '') });
}
}
return {
each:each
};
});

Related

Netsuite reconfigure inventory detail error

On a custom record creation i am calling a map reduce script that in creating a negative inventory adjustment.
The script is called on after submit.
I am using following code to populate the inventory detail subrecords
for (var n = 0; n < cp_lotsearch.length; n++) {
if (cree_onhand > 0) {
var cp_lotid = cp_lotsearch[n].getValue({name: 'internalid'});
var cp_parentcasenum = cp_lotsearch[n].getValue({name: 'custrecord_nsts_ia_lot'});
var itemsearch = search.create({
type: 'item',
filters: ['inventorynumber.inventorynumber', 'is', cp_parentcasenum],
columns: [search.createColumn({name: "quantityonhand", join: "inventorynumber"})]
}).run().getRange({
start: 0,
end: 1000
});
var qtyonhand = itemsearch[0].getValue({name: 'quantityonhand', join: 'inventorynumber'});
var remain_qtyonhand = Number(qtyonhand) - Number(cree_onhand);
if (remain_qtyonhand >= 0) {
var reduceqty = 0 - Number(cree_onhand);
} else {
var reduceqty = 0 - Number(qtyonhand);
}
log.debug("reduceqty", reduceqty);
subrecord.insertLine({sublistId: 'inventoryassignment',line: n});
subrecord.setSublistText({ sublistId: 'inventoryassignment',fieldId: 'issueinventorynumber',line: n,text: cp_parentcasenum});
subrecord.setSublistValue({sublistId: 'inventoryassignment',fieldId: 'binnumber', line: n, value: cree_bin});
subrecord.setSublistValue({sublistId: 'inventoryassignment',fieldId: 'quantity',line: n,value: reduceqty});
log.debug("reduceqty added to inventory detail", reduceqty);
cree_onhand = Number(cree_onhand) + Number(reduceqty);
}
}
var invadjid = parentcase_inv_Adj.save({
enableSourcing: true,
ignoreMandatoryFields: true
});
When i try to run this code i am getting following error:
"type":"error.SuiteScriptError","name":"USER_ERROR","message":"You still need to reconfigure the inventory detail record after changing the quantity."
Same code is running perfect in sandbox account but in production it is throwing error
Can anyone help me to solve this error?
I Got one solution in suite answer that i already tried but it is not working here is link to it:
https://netsuite.custhelp.com/app/answers/detail/a_id/80790/kw/reconfigure%20error
I've discovered this error can be triggered server-side if :
multiple units of measure feature is enabled.
the stock units of measure are not 1-to-1, i.e. the receipt unit is in packs of 20, but the base unit of the UOM is 1's.
the item is serialised.
However, if the same is attempted client-side, the error becomes "The total inventory detail quantity must be XX" (XX being the quantity set in "adjustqtyby" on the inventory adjustment line).

Unable to display highcharts graphs with setextremes

I am collecting info from weather sensors(every 5 mins) and through the days and months, the amount of records is huge and render the chart takes for ever....It is working ,but the delay is too high... So instead of displaying all the values at once, I want to make the graph based on range of dates selected, and get data accordingly.
I am using python , sqlite3 and Flask and trying to copy this example from Highcharts
https://jsfiddle.net/gh/get/library/pure/highcharts/highcharts/tree/master/samples/stock/demo/lazy-loading/
In the example they use php , which here is the code
https://github.com/highcharts/highcharts/blob/master/samples/data/from-sql.php
This is my setextreme function in the html page
function setExtremes(e)
// detect zoom button
{
if(typeof(e.rangeSelectorButton)!== 'undefined')
{
if (e.rangeSelectorButton.type=='all')
{
var chart = Highcharts.charts[0];
chart.showLoading('Loading data from server...');
$.getJSON('http://192.168.0.74:5000/rango?start=' + Math.round(e.min) +'&end=' + Math.round(e.max) + '&callback=?', function (data) {
// Arrange data
alert('I am in')
for (i = 0; i< data.length; i++)
{
processed_json_temperatura.push(data[i][0], data[i][1]);
processed_json_presion.push(data[i][0], data[i][2]);
processed_json_humedad.push(data[i][0], data[i][3]);
processed_json_lluvia.push(data[i][0], data[i][4]);
processed_json_horas_frio.push(data[i][0], data[i][5]);
}
temperatura_matriz = listToMatrix( processed_json_temperatura,2);
presion_matriz = listToMatrix( processed_json_presion,2);
humedad_matriz = listToMatrix( processed_json_humedad,2);
lluvia_matriz = listToMatrix( processed_json_lluvia,2);
horas_frio_matriz = listToMatrix(processed_json_horas_frio,2);
//update graphs with new values
chart.series[0].setData(temperatura_matriz);
chart.series[1].setData(presion_matriz);
chart.series[2].setData(humedad_matriz);
chart.series[3].setData(lluvia_matriz);
chart.series[4].setData(horas_frio_matriz);
chart.hideLoading();
});
}
}
}
On server side, this is my python code with flask
#app.route("/rango")
def rango():
start=request.args.get('start')
end=request.args.get('end')
range = int(end)/1000 - int(start)/1000
# find the right table
# two days range loads minute data
if (range < 2 * 24 * 3600 * 1000):
connection = sqlite3.connect("/var/www/nueva_estacion.db")
cursor = connection.cursor()
cursor.execute("select (julianday(timestamp)-2440587.5)*86400000, round(temp1,2) as temp1, presion, round(humedad,2) as humedad, lluvia,aux1 from raw_all_sensors where datetime(timestamp) >=datetime('now', '-6 Day')")
results = cursor.fetchall()
print json.dumps(results)
return json.dumps(results)
When I click the zoom button(on highstock actually) I do see the message " Loading data from Server ..." but never shows the new graphs and never get "I am in" console message..
When monitoring the messaging on server side, I do get
192.168.0.122 - - [22/May/2019 19:47:22] "GET /rango?start=1558050049000&end=1558551372000&callback=jQuery18307335687220269529_1558565228146&_=1558565238543 HTTP/1.1" 200 -
where is clear start date, end date and then I do not understand the callback and the numbers that follow.
Just in case, this is the info coming from server
[1558546562999.989, 17.82, 1021, 76.15, 0, 1]
Can anyone bring some light here?

How to use casperjs evaluate to diff 2 strings and assert

I have a simple test that checks to see a user's quota correctly changes after they upload a file.
casper.then(function() {
quota_begin = this.evaluate(function() {
return document.querySelector('.storage_used p').textContent;
});
});
casper.then(function() {
common.ACTIONS.uploadFile(casper);
});
casper.then(function() {
quota_changed = this.evaluate(function() {
return document.querySelector('.storage_used p').textContent;
});
this.echo('Storage quota change: ' + quota_begin + ' => ' + quota_changed);
});
That last echo's output gives me:
Storage quota change: Upload quota 0B of 1GB used => Upload quota 192 KB of 1GB used
I'd like to include an assert in the test that fails when quota_begin and quota_changed do not actually change.
Something like:
test.assert(parseFloat(quota_changed) > parseFloat(quota_begin), "Quota was increased by file");
(doesn't work)
Is there an easy way to assert a diff on the two? regex?
Write a simple function to parse used bytes from that string will do that task:
function get_used_bytes(input) {
var unit_dict = {'B':1,'KB':1024,'MB':1024*1024,'GB':1024*1024*1024}
var ret = /Upload quota ([\d.]+)(\S+) of ([\d.]+)(\S+) used/g.exec(input)
return ret[1] * unit_dict[ret[2]]
}
// get_used_bytes("Upload quota 192KB of 1GB used")
// 196608
test.assert(get_used_bytes(quota_changed) > get_used_bytes(quota_begin), "Quota was increased by file");

Creating a cached counter path for large data lists in Firebase

Using Firebase to count the total records is done this way:
var table = new Firebase('http://beta.firebase.com/user/tablename');
table.on('value', function(snapshot) {
var count = 0;
snapshot.forEach(function() {
count++;
});
//count is now safe to use.
});
Is there a way to avoid enumeration by having a cached counter in a different path?
I was thinking in some "counter" object which keeps the history of changes and the last computed value.
counter:
{
value: 672,
history:
{
+2, -4, +1, +1, +1
}
}
in a transaction then:
pick one history item, update the value, remove the history item.
Also who would be responsible of doing this?
Here's an example that combines the idea of a counter with an incremental, numeric ID. For your use case, you could skip the ID portion, but the principles are still the same.
The core of this is a transaction that, when you create a new record, adds one to your counter:
var fb = new Firebase('http://beta.firebase.com');
// stores incremental id before adding record
function incRecord(data) {
// increment the counter
fb.child('counter/value').transaction(function(currentValue) {
return (currentValue||0) + 1
}, function(err, committed, ss) {
if( err ) {
console.error(err);
}
else if( committed ) {
// if you want to pass the counter into the data,
// just use ss.val() here to fetch it
addRecord(data);
// could also store an audit history about changes to the counter, assuming we had a user ID or something to that effect with this:
// but you don't need this history to increment it
// fb.child('counter/history/'+ss.val()).set(userId);
}
});
}
// creates new record
function addRecord(data) {
// you could pass the record value here, I just set the value to "record #<id>"
fb.child('records').push(data, function(err) {
err && console.error(err);
});
}
Then invoke it by calling something like this:
incRecord({ hello: 'world' });

ColdFusion - Get next scheduled task due to run

This thread was useful in finding out the next run-time for a scheduled task.
How do I find out the next run time for a Scheduled Task?
But, is there also a way to simply get the next scheduled task due to run?
If I can get the date and name of the next task due to run, I can plug that date into a jQuery countdown timer, which will display a countdown to the next scheduled task, something like:
TaskABC due to run in:
12 03 20
hrs min sec
. This is for an admin interface in case you're wondering how geeky can people get:-)
EDIT
I had the same thought as Bill. But was curious if there was another way.
I poked around and apparently the internal Scheduler class maintains a list of upcoming tasks. The list is private, but you can use the same reflection technique to access it. Interestingly the list also includes system tasks like the mail spooler, session/application trackers, watchers, etecetera. So you must iterate through it until you find a "scheduled task" ie CronTabEntry
Below is a very lightly tested function that seems to do the trick in CF9. (Note, includes the CreateTimeStruct function from http://www.cflib.org).
Rules:
Returns a structure containing the name and time remaining until the next task. If no tasks were found, result.task is an empty string.
Excludes paused tasks
Usage:
result = new TaskUtil().getNextTask();
WriteDump(result);
CFC
component {
public struct function getNextTask() {
// get list of upcoming tasks from factory (UNDOCUMENTED)
local.scheduler = createObject("java", "coldfusion.server.ServiceFactory").getSchedulerService();
local.taskField = local.scheduler.getClass().getDeclaredField("_tasks");
local.taskField.setAccessible( true );
local.taskList = local.taskField.get(local.scheduler);
// taskList contains system jobs too, so we must iterate
// through the tasks to find the next "scheduled task"
local.nextTask = "";
local.tasks = local.taskList.iterator();
while ( local.tasks.hasNext() ) {
local.currTask = local.tasks.next();
local.className = local.currTask.getRunnable().getClass().name;
// exit as soon as we find a scheduled task that is NOT paused
if (local.className eq "coldfusion.scheduling.CronTabEntry"
&& !local.currTask.getRunnable().paused) {
local.nextTask = local.currTask;
break;
}
}
// if we found a task, calculate how many days, hours, etcetera
// until its next run time
local.details = { task="", remaining={} };
if ( isObject(local.nextTask) ) {
local.secondsToGo = (local.nextTask.getWhen() - now().getTime()) / 1000;
local.details.task = local.nextTask.getRunnable().task;
local.details.remaining = createTimeStruct(local.secondsToGo);
local.details.nextDate = dateAdd("s", local.nextTask.getWhen() / 1000
, "January 1 1970 00:00:00" );
}
return local.details;
}
/**
* Abbreviated version of CreateTimeStruct by Dave Pomerance
* See http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=421
*
* #param timespan The timespan to convert.
* #return Returns a structure.
* #author Dave Pomerance
* #version 1, January 7, 2002
*/
public struct function CreateTimeStruct(required numeric timespan) {
var timestruct = StructNew();
var mask = "s";
// only 4 allowed values for mask - if not one of those, return blank struct
if (ListFind("d,h,m,s", mask)) {
// compute seconds
if (mask eq "s") {
timestruct.s = (timespan mod 60) + (timespan - Int(timespan));
timespan = int(timespan/60);
mask = "m";
} else timestruct.s = 0;
// compute minutes
if (mask eq "m") {
timestruct.m = timespan mod 60;
timespan = int(timespan/60);
mask = "h";
} else timestruct.m = 0;
// compute hours, days
if (mask eq "h") {
timestruct.h = timespan mod 24;
timestruct.d = int(timespan/24);
} else {
timestruct.h = 0;
timestruct.d = timespan;
}
}
return timestruct;
}
}
My first thought is to iterate Leigh's getNextRunTime(string taskName) function over the collection of tasks. You can get an array of structs containing the details of all scheduled tasks using taskArray = createobject("java","coldfusion.server.ServiceFactory").getCronService().listAll();
The key in the struct containing the task name is "task". So you can extract all the task names as an array for example, run Leigh's function on each element and determine which one will run next.