Files locked after indexing - solrj

I have the following workflow in my (web)application:
download a pdf file from an archive
index the file
delete the file
My problem is that after indexing the file, it remains locked and the delete-part throws an exception.
Here is my code-snippet for indexing the file:
try
{
ContentStreamUpdateRequest req = new ContentStreamUpdateRequest("/update/extract");
req.addFile(file, type);
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
NamedList<Object> result = server.request(req);
Assert.assertEquals(0, ((NamedList<?>) result.get("responseHeader")).get("status"));
}
Do I miss something?
EDIT:
I tried this way too, but with the same result...
ContentStream contentStream = null;
try
{
contentStream = new ContentStreamBase.FileStream(document);
ContentStreamUpdateRequest req = new ContentStreamUpdateRequest(UPDATE_EXTRACT_REQUEST);
// req.addFile(document, context.getProperty(FTSConstants.CONTENT_TYPE_APPLICATION_PDF));
req.addContentStream(contentStream);
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
NamedList<Object> result = server.request(req);
if (!((NamedList<?>) result.get("responseHeader")).get("status").equals(0))
{
throw new IDSystemException(LOG, "Document could not be indexed. Status returned: " +
((NamedList<?>) result.get("responseHeader")).get("status"));
}
}
catch (FileNotFoundException fnfe)
{
throw new IDSystemException(LOG, fnfe.getMessage(), fnfe);
}
catch (IOException ioe)
{
throw new IDSystemException(LOG, ioe.getMessage(), ioe);
}
catch (SolrServerException sse)
{
throw new IDSystemException(LOG, sse.getMessage(), sse);
}
finally
{
try
{
if(contentStream != null && contentStream.getStream() != null)
{
contentStream.getStream().close();
}
}
catch (IOException ioe)
{
throw new IDSystemException(LOG, ioe.getMessage(), ioe);
}
}

This seems like a bug,
a patch is proposed here
https://issues.apache.org/jira/browse/SOLR-1744
Also checkout
http://lucene.472066.n3.nabble.com/ContentStreamUpdateRequest-addFile-fails-to-close-Stream-td485429.html
you can check if the stream is not null and close it.

It may be due to lock acquired by file system. Instead of addFile(), you can try the following.
ContentStreamUpdateRequest req = new ContentStreamUpdateRequest("/update/extract");
ContentStreamBase.FileStream fileStream = new FileStream(file);
req.addContentStream(fileStream);
Shishir

Related

Exception in Concurrency task.then.wait affects further call of ::ShellExecuteEx()

Following logic is implemented to open a file by a "filename.extension" in a C++ application using managed-C++:
try
{
CoInitialize(nullptr);
auto task = Concurrency::create_task(Windows::Storage::StorageFile::GetFileFromPathAsync(filePath));
// an excpetion is thrown in the next line
Concurrency::task_status status = task.then([&](Windows::Storage::StorageFile^ file){
if (file != nullptr)
{
concurrency::task<bool> launchFileOperation(Windows::System::Launcher::LaunchFileAsync(file));
launchFileOperation.then([&](bool success)
{
if (!success)
return 0;
}).wait();
}
}).wait();
}
catch (...)
{
CoUninitialize(); // an exeption is catched
return 0;
}
Since the above code throws an exception, we go further to an alternative file open approach via ::ShellExecuteEx
SHELLEXECUTEINFO exec_info = {0};
exec_info.cbSize = sizeof exec_info;
exec_info.fMask = SEE_MASK_NOCLOSEPROCESS
| SEE_MASK_DOENVSUBST;
exec_info.fMask &= ~SEE_MASK_NOASYNC;
exec_info.lpVerb = "open";
exec_info.lpFile = full_path_str;
exec_info.nShow = SW_SHOW;
bool result_b = ::ShellExecuteEx(&exec_info) ? true : false;
The ::ShellExecuteEx fails and ends up in Microsofts ppltasks.h
_REPORT_PPLTASK_UNOBSERVED_EXCEPTION();.
::ShellExecuteEx works correctly if the managed-C++ Concurrency::create_task approach is removed.
Why does Concurrency::create_task affect the further call of ::ShellExecuteEx?
This issue appears only in release build.
Adding try/catch-blocks to the innermost .wait()-block solved the issue
try {
concurrency::task<bool> launchFileOperation(Windows::System::Launcher::LaunchFileAsync(file));
launchFileOperation.then([&](bool success) {
// logic
}).wait();
}
catch (concurrency::invalid_operation& ex)
{
...
}
catch (concurrency::task_canceled& ex)
{
...
}

Give a file as input to Pocketsphinx on Android

I am using the latest pocketsphinx android demo (mighty computer),which takes input from microphone. I want to give a wav file as input to the same. I tried using decoder.processrow() function. But I don't know how to configure the decoder using hmm, lm etc.
Code to process files in pocketsphinx-java
Config c = Decoder.defaultConfig();
c.setString("-hmm", "../../model/en-us/en-us");
c.setString("-lm", "../../model/en-us/en-us.lm.dmp");
c.setString("-dict", "../../model/en-us/cmudict-en-us.dict");
Decoder d = new Decoder(c);
URL testwav = new URL("file:../../test/data/goforward.wav");
FileInputStream stream = new FileInputStream(new File(testwav)));
d.startUtt();
byte[] b = new byte[4096];
try {
int nbytes;
while ((nbytes = stream.read(b)) >= 0) {
ByteBuffer bb = ByteBuffer.wrap(b, 0, nbytes);
// Not needed on desktop but required on android
bb.order(ByteOrder.LITTLE_ENDIAN);
short[] s = new short[nbytes/2];
bb.asShortBuffer().get(s);
d.processRaw(s, nbytes/2, false, false);
}
} catch (IOException e) {
fail("Error when reading goforward.wav" + e.getMessage());
}
d.endUtt();
System.out.println(d.hyp().getHypstr());
for (Segment seg : d.seg()) {
System.out.println(seg.getWord());
}
}
Adding to the answer from Nikolay, this is how it can be done on Android, adapting the SpeechRecognizer Android implementation example found here: http://cmusphinx.sourceforge.net/wiki/tutorialandroid
//statically load our library
static {
System.loadLibrary("pocketsphinx_jni");
}
//convert an inputstream to text
private void convertToSpeech(final InputStream stream){
new AsyncTask<Void, Void, Exception>() {
#Override
protected Exception doInBackground(Void... params) {
try {
Assets assets = new Assets(WearService.this);
File assetsDir = assets.syncAssets();
Config c = Decoder.defaultConfig();
c.setString("-hmm", new File(assetsDir, "en-us-ptm").getPath());
c.setString("-dict", new File(assetsDir, "cmudict-en-us.dict").getPath());
c.setBoolean("-allphone_ci", true);
c.setString("-lm", new File(assetsDir, "en-phone.dmp").getPath());
Decoder d = new Decoder(c);
d.startUtt();
byte[] b = new byte[4096];
try {
int nbytes;
while ((nbytes = stream.read(b)) >= 0) {
ByteBuffer bb = ByteBuffer.wrap(b, 0, nbytes);
// Not needed on desktop but required on android
bb.order(ByteOrder.LITTLE_ENDIAN);
short[] s = new short[nbytes/2];
bb.asShortBuffer().get(s);
d.processRaw(s, nbytes/2, false, false);
}
} catch (IOException e) {
fail("Error when reading inputstream" + e.getMessage());
}
d.endUtt();
System.out.println(d.hyp().getHypstr());
for (Segment seg : d.seg()) {
//do something with the result here
}
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR);
}

winrt c++/cx concurrency access violation exception

What I'm trying to do is check for the existence of a file in the local folder and then copy it there if it isn't found (the file was previously added to the project as an asset).
Here is the code:
Windows::Storage::StorageFile^ MainPage::GetCustomFileAsync(Platform::String^ fileName)
{
using Windows::Storage::StorageFile;
using Windows::Storage::StorageFolder;
auto localFolder = Windows::Storage::ApplicationData::Current->LocalFolder;
auto localTask = concurrency::create_task(localFolder->GetFileAsync(fileName));
StorageFile^ retVal = nullptr;
localTask.then([&](StorageFile^ t){
retVal = t;
}).then([](concurrency::task<void> t)
{
try
{
t.get();
OutputDebugString(L"Found\n");
}
catch (Platform::COMException^ e)
{
OutputDebugString(e->Message->Data());
}
}).wait();
return retVal;
}
StorageFile^ fileVar;
if ((fileVar = this->GetCustomFileAsync("somefile.txt")) == nullptr)
{
String^ path = Windows::ApplicationModel::Package::Current->InstalledLocation->Path + "\\Assets";
concurrency::create_task(Windows::Storage::StorageFolder::GetFolderFromPathAsync(path)).then([](StorageFolder^ folder){
return (folder->GetFileAsync("somefile.txt"));
}).then([](StorageFile^ file){
return (file->CopyAsync(Windows::Storage::ApplicationData::Current->LocalFolder));
}).then([&](StorageFile^ file){
fileVar = file;
OutputDebugString(file->DisplayName->Data());
});
}
What happens is that I get an access violation exception at the point where "file" is being assigned to "fileVar" (because of cross-thread access perhaps?). How to fix this?
Edit: I can't do all the processing there because the file will be accessed many times. In short I need to know when it has been successfully copied and get a handle to it. Here is the code that works
Windows::Storage::StorageFile^ GetFile(Platform::String^ fileName)
{
using Windows::Storage::StorageFile;
using Windows::Storage::StorageFolder;
using Windows::Foundation::AsyncOperationCompletedHandler;
using Windows::Foundation::AsyncStatus;
using Windows::Foundation::IAsyncOperation;
using Platform::String;
auto localFolder = Windows::Storage::ApplicationData::Current->LocalFolder;
bool completed = false;
StorageFile^ retVal = nullptr;
localFolder->GetFileAsync(fileName)->Completed = ref new AsyncOperationCompletedHandler<StorageFile^>([&completed, &retVal, &fileName](IAsyncOperation<StorageFile^>^ fileOperation, AsyncStatus status)
{
if (status == AsyncStatus::Error)
{
String^ path = Windows::ApplicationModel::Package::Current->InstalledLocation->Path + "\\Assets";
Windows::Storage::StorageFolder::GetFolderFromPathAsync(path)->Completed = ref new AsyncOperationCompletedHandler<Windows::Storage::StorageFolder^>(
[&completed, &retVal, &fileName](IAsyncOperation<Windows::Storage::StorageFolder^>^ folderOperation, AsyncStatus status)->void{
auto assetFolder = folderOperation->GetResults();
assetFolder->GetFileAsync(fileName)->Completed = ref new AsyncOperationCompletedHandler<Windows::Storage::StorageFile^>([&completed, &retVal, &fileName](IAsyncOperation<Windows::Storage::StorageFile^>^ fileOperation, AsyncStatus status)->void{
auto file = fileOperation->GetResults();
file->CopyAsync(Windows::Storage::ApplicationData::Current->LocalFolder)->Completed = ref new AsyncOperationCompletedHandler<Windows::Storage::StorageFile^>
([&completed, &retVal, &fileName](IAsyncOperation<Windows::Storage::StorageFile^>^ fileOperation, AsyncStatus status)->void {
retVal = fileOperation->GetResults();
completed = true;
});
});
});
}
else
{
retVal = fileOperation->GetResults();
completed = true;
}
});
while (completed == false);
return retVal;
}
Rather than passing a delegate as an argument and returning void, make your method return task<StorageFile^> and then the caller can do a .then() to continue working once the operation has succeeded.
Or if this is exposed as a public WinRT method (not an internal / private C++ method) then use IAsyncOperation<StorageFile^>^ as the return type, and wrap the whole thing in create_async():
IAsyncOperation<StorageFile^>^ DoStuff(params)
{
return concurrency::create_async([params]
{
// function body goes here
});
}
Here's a solution I put together. Two things that are important to know:
When executing an asynchronous operation using concurrency::create_task the async operation(s) can still be executing when the parent function returns. So the captured variables MUST outlive the context of the parent function. Which obviously won't happen if they are being passed by reference. It took a while to realize this.
WinRT imposes certain restrictions on the concurrency runtime. Calling concurrency::task::get() or concurrency::task::wait() will throw an exception in an STA thread, unless the call is in a task continuation.
More information in this post:
http://social.msdn.microsoft.com/Forums/windowsapps/en-US/ae54980b-41ce-4337-a059-2213b549be4b/concurrencyinvalidoperation-when-calling-tasktget?forum=winappswithnativecode
In that case how to know when the function has finished doing it's job? I opted to pass in a callback (AKA delegate).
delegate void FileOperation(Windows::Storage::StorageFile^ file);
void GetFileConcurrency(Platform::String^ fileName, FileOperation^ fileOp)
{
using Windows::Storage::StorageFile;
using Windows::Storage::StorageFolder;
using Platform::String;
auto localFolder = Windows::Storage::ApplicationData::Current->LocalFolder;
String^ assetFolderPath = Windows::ApplicationModel::Package::Current->InstalledLocation->Path + "\\Assets";
auto localFolderTask = concurrency::create_task(localFolder->GetFileAsync(fileName));
localFolderTask.then([localFolder, assetFolderPath, fileName, fileOp](concurrency::task<StorageFile^> theTask){
try
{
StorageFile^ theFile = theTask.get();
fileOp(theFile);
}
catch (Platform::Exception^ e)
{
OutputDebugString(e->Message->Data());
auto assetFolderTask = concurrency::create_task(StorageFolder::GetFolderFromPathAsync(assetFolderPath));
assetFolderTask.then([localFolder, assetFolderPath, fileName, fileOp](StorageFolder^ assetFolder){
auto assetFileTask = concurrency::create_task(assetFolder->GetFileAsync(fileName));
assetFileTask.then([localFolder, assetFolderPath, fileName, fileOp](StorageFile^ file){
auto copyFileTask = concurrency::create_task(file->CopyAsync(localFolder));
copyFileTask.then([localFolder, assetFolderPath, fileName, fileOp](StorageFile^ file){
OutputDebugString(file->Path->Data());
fileOp(file);
});
});
});
}
});
}

Shell Extension DLL - how to capture the folder path, if user clicks inside the folder empty area?

Using shell extension dll, how to capture the folder path, if user clicks inside the folder empty area?
If you're implementing a shell extension dll, then you get the path in your IShellExtInit::Initialize() method as the pidlFolder parameter.
To make sure your extension is also registered for folder backgrounds, you have to create the appropriate entries also under HKCR\Directory\Background\shellex\ContextMenuHandlers
With VC++ language please reference Winmerge souce code
http://sourceforge.net/p/winmerge/code/HEAD/tree/trunk/ShellExtension/
With C# please reference this article
http://www.codeproject.com/Articles/174369/How-to-Write-Windows-Shell-Extension-with-NET-Lang
and update some place bellow:
At FileContextMenuExt.cs file:
...............
#region Shell Extension Registration
[ComRegisterFunction()]
public static void Register(Type t)
{
try
{
ShellExtReg.RegisterShellExtContextMenuHandler(t.GUID, "Directory",
"CSShellExtContextMenuHandler.FileContextMenuExt Class");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message); // Log the error
throw; // Re-throw the exception
}
}
[ComUnregisterFunction()]
public static void Unregister(Type t)
{
try
{
ShellExtReg.UnregisterShellExtContextMenuHandler(t.GUID, "Directory");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message); // Log the error
throw; // Re-throw the exception
}
}
#endregion
...............
public void Initialize(IntPtr pidlFolder, IntPtr pDataObj, IntPtr hKeyProgID)
{
if (pDataObj == IntPtr.Zero && pidlFolder == IntPtr.Zero)
{
throw new ArgumentException();
}
FORMATETC fe = new FORMATETC();
fe.cfFormat = (short)CLIPFORMAT.CF_HDROP;
fe.ptd = IntPtr.Zero;
fe.dwAspect = DVASPECT.DVASPECT_CONTENT;
fe.lindex = -1;
fe.tymed = TYMED.TYMED_HGLOBAL;
STGMEDIUM stm = new STGMEDIUM();
try
{
if (pDataObj != IntPtr.Zero)
{
// The pDataObj pointer contains the objects being acted upon. In this
// example, we get an HDROP handle for enumerating the selected files
// and folders.
IDataObject dataObject = (IDataObject)Marshal.GetObjectForIUnknown(pDataObj);
dataObject.GetData(ref fe, out stm);
// Get an HDROP handle.
IntPtr hDrop = stm.unionmember;
if (hDrop == IntPtr.Zero)
{
throw new ArgumentException();
}
// Determine how many files are involved in this operation.
uint nFiles = NativeMethods.DragQueryFile(hDrop, UInt32.MaxValue, null, 0);
// This code sample displays the custom context menu item when only
// one file is selected.
if (nFiles == 1)
{
// Get the path of the file.
StringBuilder fileName = new StringBuilder(260);
if (0 == NativeMethods.DragQueryFile(hDrop, 0, fileName,
fileName.Capacity))
{
Marshal.ThrowExceptionForHR(WinError.E_FAIL);
}
this.selectedFile = fileName.ToString();
}
else
{
Marshal.ThrowExceptionForHR(WinError.E_FAIL);
}
}
if (pidlFolder != IntPtr.Zero) {
StringBuilder folderName = new StringBuilder(260);
if (0 == NativeMethods.SHGetPathFromIDList(pidlFolder, folderName))
{
Marshal.ThrowExceptionForHR(WinError.E_FAIL);
}
this.selectedFile = folderName.ToString();
}
}
finally
{
NativeMethods.ReleaseStgMedium(ref stm);
}
}
At ShellExtLib.cs file Add folowing source:
[DllImport("shell32.dll")]
public static extern Int32 SHGetPathFromIDList(
IntPtr pidl, // Address of an item identifier list that
// specifies a file or directory location
// relative to the root of the namespace (the
// desktop).
StringBuilder pszPath); // Address of a buffer to receive the file system
And update RegisterShellExtContextMenuHandler and UnregisterShellExtContextMenuHandler function at ShellExtLib.cs file
public static void RegisterShellExtContextMenuHandler(Guid clsid,
string fileType, string friendlyName)
{
if (clsid == Guid.Empty)
{
throw new ArgumentException("clsid must not be empty");
}
if (string.IsNullOrEmpty(fileType))
{
throw new ArgumentException("fileType must not be null or empty");
}
// If fileType starts with '.', try to read the default value of the
// HKCR\<File Type> key which contains the ProgID to which the file type
// is linked.
if (fileType.StartsWith("."))
{
using (RegistryKey key = Registry.ClassesRoot.OpenSubKey(fileType))
{
if (key != null)
{
// If the key exists and its default value is not empty, use
// the ProgID as the file type.
string defaultVal = key.GetValue(null) as string;
if (!string.IsNullOrEmpty(defaultVal))
{
fileType = defaultVal;
}
}
}
}
else {
// Create the key HKCR\<File Type>\shellex\ContextMenuHandlers\{<CLSID>}.
string keyName1 = string.Format(#"{0}\Background\shellex\ContextMenuHandlers\{1}",
fileType, clsid.ToString("B"));
using (RegistryKey key = Registry.ClassesRoot.CreateSubKey(keyName1))
{
// Set the default value of the key.
if (key != null && !string.IsNullOrEmpty(friendlyName))
{
key.SetValue(null, friendlyName);
}
}
}
// Create the key HKCR\<File Type>\shellex\ContextMenuHandlers\{<CLSID>}.
string keyName = string.Format(#"{0}\shellex\ContextMenuHandlers\{1}",
fileType, clsid.ToString("B"));
using (RegistryKey key = Registry.ClassesRoot.CreateSubKey(keyName))
{
// Set the default value of the key.
if (key != null && !string.IsNullOrEmpty(friendlyName))
{
key.SetValue(null, friendlyName);
}
}
}
public static void UnregisterShellExtContextMenuHandler(Guid clsid,
string fileType)
{
if (clsid == null)
{
throw new ArgumentException("clsid must not be null");
}
if (string.IsNullOrEmpty(fileType))
{
throw new ArgumentException("fileType must not be null or empty");
}
// If fileType starts with '.', try to read the default value of the
// HKCR\<File Type> key which contains the ProgID to which the file type
// is linked.
if (fileType.StartsWith("."))
{
using (RegistryKey key = Registry.ClassesRoot.OpenSubKey(fileType))
{
if (key != null)
{
// If the key exists and its default value is not empty, use
// the ProgID as the file type.
string defaultVal = key.GetValue(null) as string;
if (!string.IsNullOrEmpty(defaultVal))
{
fileType = defaultVal;
}
}
}
}
else {
// Remove the key HKCR\<File Type>\shellex\ContextMenuHandlers\{<CLSID>}.
string keyName1 = string.Format(#"{0}\Background\shellex\ContextMenuHandlers\{1}",
fileType, clsid.ToString("B"));
Registry.ClassesRoot.DeleteSubKeyTree(keyName1, false);
}
// Remove the key HKCR\<File Type>\shellex\ContextMenuHandlers\{<CLSID>}.
string keyName = string.Format(#"{0}\shellex\ContextMenuHandlers\{1}",
fileType, clsid.ToString("B"));
Registry.ClassesRoot.DeleteSubKeyTree(keyName, false);
}

Print an EObject?

I am writing some eclipse emf code and would like to print the content of an EObject (not store it to disk).
Here is what I try:
public static void print(EObject obj) {
Resource eResource = obj.eResource();
try {
eResource.save(System.out, null);
} catch (IOException e) {
e.printStackTrace();
}
}
but that gives a NullPointerException. I have tried this instead:
public static void print(EObject obj) {
ResourceSet resourceSet = new ResourceSetImpl();
resourceSet.getResourceFactoryRegistry().getProtocolToFactoryMap()
.put("*", new XMIResourceFactoryImpl());
Resource resource = resourceSet.createResource(URI.createURI("dummyfile.xml"));
resource.getContents().add(obj);
try {
resource.save(System.out, null);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
This works, but is it not possible to print to screen without specifying a dummy URI??
Updated to include EcoreUtil.copy()
Check this code.
Resource res = new XMLResourceImpl ();
res.getContents().add(EcoreUtil.copy(obj));
try {
resource.save(System.out, null);
} catch (IOException ioe) {
ioe.printStackTrace();
}
If that fails then yes you need a dummy URI
Resource res = new XMLResourceImpl (URI.createURI("dummyfile.xml"));
res.getContents().add(EcoreUtil.copy(obj));
try {
resource.save(System.out, null);
} catch (IOException ioe) {
ioe.printStackTrace();
}
Hm when I pass a copy:
Resource res = new XMLResourceImpl ();
res.getContents().add(ECoreUtil.copy(obj));
try {
resource.save(System.out, null);
} catch (IOException ioe) {
ioe.printStackTrace();
}
some of the xmi attributes are not printed. But if I call the above method multiple times and DON't pass a copy I get a NullPointerException. I guess I am not understanding some basic EMF/Containment functionality here?
So my updated question is:
Is it possible to print a FULL EObject model without modifying the content if the model is used in the following code?