I wanted to move some files to the input folder and tried the required code by placing it in configure() method (I had to use Old mapred API, due to some constraints i have).
But, some how the code in configure is not being executed.
I have achieved my requirement though in other better way. Though this is a stupid idea, I wanted to know why it is not being executed. I have checked the job tracker and all variables got the right values
Code:
in main:
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
String date = sdf.format(new Date());
if(args[3].toString().equalsIgnoreCase("net")){
imgInpPath = "/user/mapreduce/output/net/"+date+"/";
inputDir = inputDir+"net/";
}
conf.set("imgInpPath", imgInpPath);
conf.set("inputDir", inputDir);
FileInputFormat.setInputPaths(conf,inputDir);
in configure()
inputPath = conf.get("inputDir");
Path inputImgPath = new Path(conf.get("imgInpPath"));
Configuration config = new Configuration();
FileSystem fileSystem = FileSystem.get(config);
inputImgPath = fileSystem.makeQualified(inputImgPath);
FileStatus[] status = fileSystem.listStatus(inputImgPath, new PathFilter(){
#Override
public boolean accept(Path name) {
return name.getName().contains("part");
}});
for(int i=0; i <status.length;i++)
{
Path inpPath = status[i].getPath();
FileUtil.copy(fileSystem, inpPath, fileSystem, new Path(inputPath), true , conf);
}
As i said, required thing is achieved in other way. But, I wanted to know, why this is not being performed, irrespective of the requirement.
Related
I am creating a payables invoices using Web Services on GP2013. Optionally, my users can provide line item distributions. I can create invoices, but unless I modify the CreatePayablesInvoice policy to "Distributions Will Be Provided" in the dynamics security console, the invoice gets both the system provided distributions as well as the distribution lines I am creating. I want to have the ability to provide distributions if necessary, otherwise I want the system to handle it.
The documentation suggests I should be able to alter the policy in code, but when I get the policy object back from GetPolicyByOperation, the Behaviors array is empty. I have tried creating the behavior manually in code and it doesn't alter what happens when the invoice is created. The only thing that impacts the result is editing the property in the security console.
My code for altering the the policy is below:
payablesInvoiceCreatePolicy = wsDynamicsGP.GetPolicyByOperation("CreatePayablesInvoice", context);
BehaviorKey bk = new BehaviorKey();
bk.Id = new Guid("e476a157-ecf0-4dae-8cef-317dd2cfbe41");
Behavior b = new Behavior();
b.Key = bk;
BehaviorOption opt0 = new BehaviorOption();
opt0.Key = new BehaviorOptionKey();
opt0.Key.Id = 0;
opt0.Name = "Distributions Will Be Provided";
BehaviorOption opt1 = new BehaviorOption();
opt1.Key = new BehaviorOptionKey();
opt1.Key.Id = 1;
opt1.Name = "Automatically Create Distributions";
b.Options = new BehaviorOption[] { opt0, opt1 };
b.SelectedOption = b.Options[1];
payablesInvoiceCreatePolicy.Behaviors = new Behavior[]{b};
wsDynamicsGP.CreatePayablesInvoice(payablesInvoice, context, payablesInvoiceCreatePolicy);
Documentation seems to be sparse on what should or shouldn't work here. I have to assume I should be able to update the policy as I see fit at runtime based on whether or not my user has decided to provide line item distributions.
Does anyone know what I am missing?
Yes. It took me 2 days to fix this myself. After adding the behaviour to the policy object UpdatePolicy before calling the create invoice as below:
wsDynamicsGP.UpdatePolicy(payablesInvoiceCreatePolicy, new RoleKey { Id = "00000000-0000-0000-0000-000000000000" }, context)
wsDynamicsGP.CreatePayablesInvoice(payablesInvoice, context, payablesInvoiceCreatePolicy);
Note that you selected the behavior that automatically creates distribution lines on invoice creation. The behavior's Internal property also must be set to true.
Here is a working example also utilizing FlowerKing's answer:
BehaviorKey bk = new BehaviorKey();
bk.Id = new Guid("e476a157-ecf0-4dae-8cef-317dd2cfbe41");
bk.PolicyKey = payablesInvoiceCreatePolicy.Key;
Behavior b = new Behavior();
b.Key = bk;
b.Internal = true;
BehaviorOption opt0 = new BehaviorOption();
opt0.Key = new BehaviorOptionKey();
opt0.Key.Id = 0;
opt0.Name = "Distributions Will Be Provided";
BehaviorOption opt1 = new BehaviorOption();
opt1.Key = new BehaviorOptionKey();
opt1.Key.Id = 1;
opt1.Name = "Automatically Create Distributions";
b.Options = new BehaviorOption[] { opt0, opt1 };
b.SelectedOption = b.Options[0];
policy.Behaviors = new Behavior[] { b };
client.UpdatePolicy(payablesInvoiceCreatePolicy, new RoleKey { Id = "00000000-0000-0000-0000-000000000000" }, context);
client.CreatePayablesInvoice(payablesInvoice, context, payablesInvoiceCreatePolicy);
I want to use "GATE" through web. Then I decide to create a SOAP web service in java with help of GATE Embedded.
But for the same document and saved Pipeline, I have a different run-time duration, when GATE Embedded runs as a java web service.
The same code has a constant run-time when it runs as a Java Application project.
In the web service, the run-time will be increasing after each execution until I get a Timeout error.
Does any one have this kind of experience?
This is my Code:
#WebService(serviceName = "GateWS")
public class GateWS {
#WebMethod(operationName = "gateengineapi")
public String gateengineapi(#WebParam(name = "PipelineNumber") String PipelineNumber, #WebParam(name = "Documents") String Docs) throws Exception {
try {
System.setProperty("gate.home", "C:\\GATE\\");
System.setProperty("shell.path", "C:\\cygwin2\\bin\\sh.exe");
Gate.init();
File GateHome = Gate.getGateHome();
File FrenchGapp = new File(GateHome, PipelineNumber);
CorpusController FrenchController;
FrenchController = (CorpusController) PersistenceManager.loadObjectFromFile(FrenchGapp);
Corpus corpus = Factory.newCorpus("BatchProcessApp Corpus");
FrenchController.setCorpus(corpus);
File docFile = new File(GateHome, Docs);
Document doc = Factory.newDocument(docFile.toURL(), "utf-8");
corpus.add(doc);
FrenchController.execute();
String docXMLString = null;
docXMLString = doc.toXml();
String outputFileName = doc.getName() + ".out.xml";
File outputFile = new File(docFile.getParentFile(), outputFileName);
FileOutputStream fos = new FileOutputStream(outputFile);
BufferedOutputStream bos = new BufferedOutputStream(fos);
OutputStreamWriter out;
out = new OutputStreamWriter(bos, "utf-8");
out.write(docXMLString);
out.close();
gate.Factory.deleteResource(doc);
return outputFileName;
} catch (Exception ex) {
return "ERROR: -> " + ex.getMessage();
}
}
}
I really appreciate any help you can provide.
The problem is that you're loading a new instance of the pipeline for every request, but then not freeing it again at the end of the request. GATE maintains a list internally of every PR/LR/controller that is loaded, so anything you load with Factory.createResource or PersistenceManager.loadObjectFrom... must be freed using Factory.deleteResource once it is no longer needed, typically using a try-finally:
FrenchController = (CorpusController) PersistenceManager.loadObjectFromFile(FrenchGapp);
try {
// ...
} finally {
Factory.deleteResource(FrenchController);
}
But...
Rather than loading a new instance of the pipeline every time, I would strongly recommend you explore a more efficient approach to load a smaller number of instances of the pipeline but keep them in memory to serve multiple requests. There is a fully worked-through example of this technique in the training materials on the GATE wiki, in particular module number 8 (track 2 Thursday).
I'm really suffering through the WinRT Windows::Storage namespace with all it's asyncronousness.
I have the following private members in my header file:
//Members for copying the SQLite db file
Platform::String^ m_dbName;
Windows::Storage::StorageFolder^ m_localFolder;
Windows::Storage::StorageFolder^ m_installFolder;
Windows::Storage::StorageFile^ m_dbFile;
And I have the following code block in my implementation file:
//Make sure the SQLite Database is in ms-appdata:///local/
m_dbName = L"DynamicSimulations.db";
m_localFolder = ApplicationData::Current->LocalFolder;
m_installFolder = Windows::ApplicationModel::Package::Current->InstalledLocation;
auto getLocalFileOp = m_localFolder->GetFileAsync(m_dbName);
getLocalFileOp->Completed = ref new AsyncOperationCompletedHandler<StorageFile^>([this](IAsyncOperation<StorageFile^>^ operation, AsyncStatus status)
{
m_dbFile = operation->GetResults();
if(m_dbFile == nullptr)
{
auto getInstalledFileOp = m_installFolder->GetFileAsync(m_dbName);
getInstalledFileOp->Completed = ref new AsyncOperationCompletedHandler<StorageFile^>([this](IAsyncOperation<StorageFile^>^ operation, AsyncStatus status)
{
m_dbFile = operation->GetResults();
m_dbFile->CopyAsync(m_localFolder, m_dbName);
});
}
});
I get a memory access violation when it gets to m_dbFile = operation->GetResults();
What am I missing here? I come from a c# background in which this is really easy stuff to do :/
I've tried using '.then' instead of registering the event but I couldn't even get those to compile.
thank you for your help!
If you are interested in the WinRT solution, here it is:
It seems all you want to do is to copy the DB file from the installed location into the local folder. For that the following code should suffice:
//Make sure the SQLite Database is in ms-appdata:///local/
m_dbName = L"DynamicSimulations.db";
m_localFolder = ApplicationData::Current->LocalFolder;
m_installFolder = Windows::ApplicationModel::Package::Current->InstalledLocation;
create_task(m_installFolder->GetFileAsync(m_dbName)).then([this](StorageFile^ file)
{
create_task(file->CopyAsync(m_localFolder, m_dbName)).then([this](StorageFile^ copiedFile)
{
// do something with copiedFile
});
});
I've tried this thing before. Don't do this:
if(m_dbFile == nullptr)
Instead verify the value of "status".
if(status == AsyncStatus::Error)
My goal is to get a listing of all the DataStores in a specific datacenter. I'm able to list all of the Hosts, and VM's, but not the Datastores, and I don't understand why (I'm still learning the API's). Any insight would be appreciated.
Here's the code for grabbing all of the VM's (this works as expected):
public List<VM> getVMsInDatacenter(String datacenter, IEnumerable<String> properties)
{
List<VM> VMs = null;
this.joinConnection((appUtil) =>
{
var svcUtil = appUtil.getServiceUtil();
var dcMoRef = svcUtil.GetDecendentMoRef(null, "Datacenter", datacenter);
var typeinfo = buildTypeInfo("VirtualMachine", properties.ToList());
VMs = buildVMsFromObjectContent(svcUtil.GetContentsRecursively(null, dcMoRef, typeinfo, true));
});
return VMs;
}
Here is the analogous code for the Datastore (which does not work as expected):
public List<DataStore> getDataStoresInDatacenter(String datacenter, IEnumerable<String> properties)
{
List<DataStore> DataStores = null;
this.joinConnection((appUtil) =>
{
var svcUtil = appUtil.getServiceUtil();
var dcMoRef = svcUtil.GetDecendentMoRef(null, "Datacenter", datacenter);
var typeinfo = buildTypeInfo("Datastore", properties.ToList());
DataStores = buildDataStoresFromObjectContent(svcUtil.GetContentsRecursively(null, dcMoRef, typeinfo, true));
});
return DataStores;
}
appUtil is an instantiation of the AppUtil class that came with the VIM SDK samples. It houses functionality for connecting, querying, etc.
joinConnection is a method for connecting, or re-using a connection if we've already connected.
If there are any other questions about the code, please let me know.
Also, if there's a better way, I'd like to know that too :)
Found the problem. The method getContentsRecursively is calling a method called 'buildFullTraversal' that builds a traversal/selection spec. This method was not adding a traversal for the datastore. I added one that like so:
TraversalSpec vmToDs = new TraversalSpec();
vmToDs.name = "vmToDs";
vmToDs.type = "VirtualMachine";
vmToDs.path = "datastore";
HToVm.skip = false;
HToVm.skipSpecified = true;
And then I modified the visitFolders traversal like so:
// Recurse through the folders
TraversalSpec visitFolders = new TraversalSpec();
visitFolders.name = "visitFolders";
visitFolders.type = "Folder";
visitFolders.path = "childEntity";
visitFolders.skip = false;
visitFolders.skipSpecified = true;
visitFolders.selectSet = new SelectionSpec[] { new SelectionSpec(), new SelectionSpec(), new SelectionSpec(), new SelectionSpec(), new SelectionSpec(), new SelectionSpec(), new SelectionSpec(), new SelectionSpec() };
visitFolders.selectSet[0].name = "visitFolders";
visitFolders.selectSet[1].name = "dcToHf";
visitFolders.selectSet[2].name = "dcToVmf";
visitFolders.selectSet[3].name = "crToH";
visitFolders.selectSet[4].name = "crToRp";
visitFolders.selectSet[5].name = "HToVm";
visitFolders.selectSet[6].name = "rpToVm";
visitFolders.selectSet[7].name = "vmToDs";
return new SelectionSpec[] { visitFolders, dcToVmf, dcToHf, crToH, crToRp, rpToRp, HToVm, rpToVm, vmToDs };
Now, calls to getContentsRecursively will also include the datastores that belong to a VM, so the method in the question will works as expected.
I need to get the primary key for a row and then insert it into one of the other columns in a string.
So I've tried to do it something like this:
newsObj = new news();
newsObj.name = "test"
newsObj.Save();
newsObj.url = String.Format("blah.aspx?p={0}",newsObj.col_id);
newsObj.Save();
But it doesn't treat it as the same data object so newsObj.col_id always comes back as a zero. Is there another way of doing this? I tried this on another page and to get it to work I had to set newsObj.SetIsLoaded(true);
This is the actual block of code:
page p;
if (pageId > 0)
p = new page(ps => ps.page_id == pageId);
else
p = new page();
if (publish)
p.page_published = 1;
if (User.IsInRole("administrator"))
p.page_approved = 1;
p.page_section = staticParent.page_section;
p.page_name = PageName.Text;
p.page_parent = parentPageId;
p.page_last_modified_date = DateTime.Now;
p.page_last_modified_by = (Guid)Membership.GetUser().ProviderUserKey;
p.Add();
string urlString = String.Empty;
if (parentPageId > 0)
{
urlString = Regex.Replace(staticParent.page_url, "(.aspx).*$", "$1"); // We just want the static page URL (blah.aspx)
p.page_url = String.Format("{0}?p={1}", urlString, p.page_id);
}
p.Save();
If I hover the p.Save(); I can see the correct values in the object but the DB is never updated and there is no exception.
Thanks!
I faced the same problem with that :
po oPo = new po();
oPo.name ="test";
oPo.save(); //till now it works.
oPo.name = "test2";
oPo.save(); //not really working, it's not saving the data since isLoaded is set to false
and the columns are not considered dirty.
it's a bug in the ActiveRecord.tt for version 3.0.0.3.
In the method public void Add(IDataProvider provider)
immediately after SetIsNew(false);
there should be : SetIsLoaded(true);
the reason why the save is not working the second time is because the object can't get dirty if it is not loaded. By adding the SetIsLoaded(true) in the ActiveRecord.tt, when you are going to do run custom tool, it's gonna regenerate the .cs perfectly.