Aurelius Titan and Elasticsearch CONTAINS_REGEX not giving results - regex

I have a basic Titan instance running on Cassandra with elastic search for full-text indexing (se below for setup).
Elastic search index does not appear to return correctly when using CONTAINS_REGEX when the string parameter does not contain any sepertors like whitespace or ..
Example:
g.V().has('name', "somename") //This exist
==> v[123456]
g.V().has('name', CONTAINS_REGEX, ".*somename.*") //This does not find the vertex.
==> null
g.V().has('name', CONTAINS_REGEX, ".*omenam.*") //Nor does this..
==> null
g.V().has('name', CONTAINS_REGEX, "somename") //Or this..
==> null
g.V().has('name', CONTAINS_REGEX, "^somename$") //String start + end does not work.
==> null
g.V().has('name', "somename.two") //If the name has a separator in it, then CONTAINS_REGEX works as expected
==> v[23456]
g.V().has('name', CONTAINS_REGEX, ".*somename.*")
==> v[23456]
g.V().has('name', CONTAINS_REGEX, ".*omenam.*")
==> v[23456]
g.V().has('name', CONTAINS_REGEX, "somename\\.two")
==> v[23456]
//Changing to the new syntax also does not work as expected.
g.query().has('name', CONTAINS_REGEX, ".*somename.*").vertices()
So, as you can see, CONTAINS_REGEX does not appear to behave correctly when the string does not contain any whitespace or other separators. This has worked on previous versions (0.4.X). Maybe something has changed that I am not aware of, and this should be done differently now. I was not able to find anything in the docs however. It could also be that there are some settings in either titan or ES I have not set up correctly, but again, I could not find anything.
Any help would be greatly appreciated :)
I am inserting the names like this:
def get_or_create_name(g, name)
{
nv = g.V('name', name);
if(!nv.hasNext())
{
ret = g.addVertexWithLabel('name');
ElementHelper.setProperties(ret, 'name', name);
}
else
{
ret = nv.next();
}
ret
}
Using:
Titan 0.5.2
Debian 7.6
Java java version "1.7.0_65", OpenJDK Runtime Environment (IcedTea 2.5.1) (7u65-2.5.1-5~deb7u1), OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
Indexing part of db schema:
name_label = mgmt.makeVertexLabel("name").make()
name = mgmt.makePropertyKey('name').dataType(String.class).make()
name_uniqueness = mgmt.buildIndex("name_uniqueness",Vertex.class).addKey(name).unique().buildCompositeIndex();
mgmt.setConsistency(name_uniqueness, ConsistencyModifier.DEFAULT); //Not sure if this is needed
mgmt.buildIndex('name_index',Vertex.class).addKey(name,Mapping.TEXT.getParameter()).buildMixedIndex("search")
Small note: I am aware that I could use Mapping.STRING.getParameter() in stead of TEXT, and then use g.V().has('name', REGEX, ".*somename.*"), but I was never able to get the string index to work at all. I am also aware CONTAINS_REGEX is intended for use on each word within a longer text, but again, I was never able to get the string version to work.

After digging through error logs and testing lots of stuff I found that the issue was that some of the java-processes were running out of memory. Specifically Rexster.
What I did was increase the available memory and tweak the java options a bit.
Thus, in bin/rexster.sh I changed this line:
JAVA_OPTIONS="-server -Xms128m -Xmx512m -Dtitan.logdir=$LOG_DIR"
To:
JAVA_OPTIONS="-server -Xms128m -Xmx2g -XX:-UseConcMarkSweepGC -Dtitan.logdir=$LOG_DIR"
And now everything seems to be working nicely! :)
To recap: The issue was that Rexster crashed between storing to cassandra and the initialization of the elasticsearch index because of lack of ram.

Related

AWS Lambda Extension throws exit status 127 (/usr/bin/env: node : No such file or directory)

I am creating a Lambda extension to get secret values from secret manager using as a template:
https://github.com/hariohmprasath/aws-lambda-extensions
I have zipped the files into the following structure.
extension.zip
--> extensions
--> secret-extension
--> secret-extension
--> node_modules
--> extensions-api.js
--> index.js
--> package.json
--> package-lock.json
--> secrets.js
Error:
{
"errorMessage": "RequestId: e5c06575-cf7d-46c0-b168-624e8e9cf572 Error: exit status 127",
"errorType": "Extension.Crash"
}
The Error is that /usr/bin/env: node : No such file or directory
At the top of the index.js file is the command #!/usr/bin/env node (in order to interpret the file in node)
The runtime environment is Nodejs 12 and have tried with 14 as well.(extension documentation says node 12 runtime is required)
What could be causing this issue?
The lambda runtime is a node runtime so node should be installed.
I have ls the folder and /env folder exists.
I know node exists within the runtime as node -v returns v14.20.0 or v12.22.11
I am on a windows machine
creating the extension (dont think the deployment could be causing
this because it was written on windows machine.
Any help would be appreciated.
So found out it has to do with a custom environment they are using for the example provided by AWS. Instead I went the route of using a runtime independent solution which has worked as expected.
Documentation
I suspect the issue you may have been encountering is the same as mine, and that issue was:
The #!/usr/bin/env node had the whitespace characters \r\n at the end of the line which obviously cannot be seen unless you have your editor display these, and this is how windows handles new lines (*nix systems use just \n); Now when the lambda reads the line, it is trying to interpret it as #!/usr/bin/env node\r which obviously won't exist, and can't run the file via node.
The problem with the logs is when you look at the logs, it won't render the \r as that, it could do 1 of 2 things depending on where you look at the logs:
It will interpret \r as a new line character, and thereby just print the whitespace, which is not obvious in the log message; OR the other situation that can occur (which is what happened to me):
It shows just : No such file or directory because it's interpreted the \r as a carriage return, which means it takes the cursor to the beginning of the line, and overwrites as it prints the new characters,
I am pretty confident this is your issue, and I will admit I didn't solve this 100% on my own as a person in my team had a similar issue with whitespace characters and only after allot of head banging did I think of it, and confirmed using hexdump -C to confirm the issue.

Why is faketime command skipping section of code

I have a cpp code that I'm trying to run with faketime command. I'm running it on two identical computers. They're both running RHEL 7. I noticed that when I run my code, on one computer, it totally skips my popen call.
My code is essentially
char ntp[]= "192.168.1.200";
FILE *jitter;
char line[100];
char *start;
char * eol;
char pps[] = "NTPS";
jitter = popen("chronyc sources", "r");
int i;
cout<<"reached here"<<endl;
while(fgets(line,sizeof(line),jitter))
{
cout<<"line is\n"<<line<<endl;
if(strstr(line,pps)){
start = strpbrk(line,"#0+-");
cout<<"PPS is "<<start<<endl;
//find new line character and replace it with comma
eol = strrchr(start,'\n');
i=eol-start;
start[i]=',';
myfile<<start;
}
if(strstr(line,ntp)){
myfile<<start;
}
}
pclose(jitter);
}
I added a print statement of
cout<<"reached here"<<endl;
but when I run it with "faketime 'last friday 5pm' ./code", on one computer it never reaches the print statement for some reason while on the other it does. I searched online to no success (I'm not running a approximating algorithm, they have the same compiler and make file, etc. I'm literally doing a git pull of the code and running it).
Does anyone know why?
Thanks
So it appears the issue was with SELinux. Apparently chrony doesn't interact well when SELinux is in enforced mode. I switched it to permissive and it behaves as expected now. I can call faketime 'last friday 5pm' chronyc sources as well as use it in my popen code

How to enumerate the installed StoreApps and their ID in Windows 8 and 10

What I want to get is the AppUserModelId of all installed StoreApp applications, so that I can pass it to IApplicationActivationManager->ActivateApplication.
In Windows 8 it was stored in the Registry, but in Windows 10 it is not anymore.
There are a lot of questions about this in internet but even after days of searching I could not find a satisfying solution.
What I have so far is the following:
I create an instance of IPackageManager,
I call FindPackagesByUserSecurityId() with the SID of the current user,
I iterate through the returned collection
I get an IPackage interface
From that I get an IPackageId interface,
Then I call IPackageId->get_FamilyName()
With that I have for example on Windows 10 for the Windows Calculator the string "Microsoft.WindowsCalculator_8wekyb3d8bbwe".
When I append to this string an "!App" I have the complete AppUserModelId to start the Windows Calculator: "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"
But not all applications use an "!App" behind the FamilyName.
For example Spartan uses the AppUserModelId
"Microsoft.Windows.Spartan_cw5n1h2txyewy!Microsoft.Spartan.Spartan" which does not end with "!App". And when I replace "!Microsoft.Spartan.Spartan" with "!App" it will not start -> "This app does not support the contract specified".
So my question is where do I get the last missing part from?
I found a PowerShell code in internet (http://poshcode.org/5702) that seems to do something very similar:
Get-AppXPackage $PackageName -pv Package |
Get-AppxPackageManifest | % {
foreach($Application in $_.Package.Applications.Application) {
if($Application.Id -like $AppId) {
if($Protocol -and !($Application.Extensions.Extension.Protocol.Name | ? { ($_ + "://") -match (($Protocol -replace '\*','.*') + "(://)?") })) {
continue
}
[PSCustomObject]#{
# Notice the secret magic property:
PSTypeName = "Microsoft.Windows.Appx.Application"
AppUserModelId = $Package.PackageFamilyName + "!" + $Application.Id
}
}
}
}
I really don't understand this cryptic PowerShell stuff, but one line seems interesting to me:
foreach($Application in $_.Package.Applications.Application)
This seems to enumerate Applications in a Package.
A comment in the same PowerShell code says:
# The full AppUserModelId is composed of the package name,
the publisher id, and the app id, such as
Microsoft.ZuneMusic_8wekyb3d8bbwe!Microsoft.ZuneMusic
so what is missing is the $Application.Id.
If I could get an IAppInfo interface anyhow I could call IAppInfo->get_Id() and I would be ready.
But I don't know how to get this from an IPackage in C++.
Incredible that nobody has an idea!
This shows how Microsoft makes us life hard.
Such a universal task like enumerating the installed StoreApps with their AppUserModelId requires a cientific research department.
I finally came to a solution that works perfectly on Windows 8 and Windows 10. But a lot of code is required.
It seems that Windows does not hold the Application ID's in memory and there is no API to determine them directly. I studied all header files in the Windows 10 SDK and could not find a corresponding interface useful for that task.
But I found out how to get them. I continue after the 6 steps in my question:
call IPackage->get_InstalledLocation() which returns an IStorageFolder.
QueryInterface for IStorageItem
call IStorageItem->get_Path()
Now you have the path were the App is installed. Windows 10 uses two base folders:
C:\Program Files\WindowsApps
C:\Windows\SystemApps
and several others like
C:\Windows\vpnplugins
C:\Windows\devicesflow
C:\Windows\MicracastView
C:\Windows\PrintDialog
C:\Windows\PrintDialog3D
C:\Windows\WinStore
In the returned folder path you will find a file "AppxManifest.xml".
This file looks like:
<?xml version="1.0" encoding="utf-8"?>
<Package xmlns=".....">
......
......
<Applications>
<Application Id="microsoft.windowslive.mail" Executable="HxMail.exe" EntryPoint="Executable">
......
......
</Application>
<Application Id="microsoft.windowslive.calendar" Executable="HxCalendarAppImm.exe" EntryPoint="Executable">
......
......
</Application>
</Applications>
</Package>
And voilĂ , there they are. This package has two application ID's: "microsoft.windowslive.mail" and "microsoft.windowslive.calendar".
Then you take the package's FamilyName from step 6 append an "!" and append this ID and you are done.
This package can be started with IApplicationActivationManager->ActivateApplication() using one of the AppUserModelId's:
"microsoft.windowscommunicationsapps_8wekyb3d8bbwe!microsoft.windowslive.calendar"
"microsoft.windowscommunicationsapps_8wekyb3d8bbwe!microsoft.windowslive.mail"
Use PackageManager APIs to enumerate packages and GetPackageApplicationIds to enumerate applications in a package e.g. pseudo-code
FOREACH p IN PackageManager.FindPackagesForUserWithPackageTypes(null,
PackageType_Main|PackageType_Optional)
{
PACKAGE_INFO_REFERENCE pir
OpenPackageInfoByFullName(p.Id.FullName, 0, &pir)
UINT32 n=0
GetPackageApplicationIds(pir, &n, null, null)
BYTE* buffer = new BYTE[n]
UINT32 count=0
GetPackageApplicationIds(pir, &n, buffer, &count)
ClosePackageInfo(pir)
PCWSTR * applicationUserModelIds = reinterpret_cast<PCWSTR*>(buffer);
FOR (i=0; i<count; ++i)
{
PCWSTR applicationUserModelId = applicationUserModelIds[i]
}
delete [] buffer
}
See GetPackageApplicationIds() on MSDN for more details including working sample code
https://msdn.microsoft.com/en-us/library/windows/desktop/dn270603(v=vs.85).aspx

Troubleshooting Emacs Tramp Connection that Used To Work--Password or Shell Regexp Wrong?

I'm trying to troubleshoot why an emacs+tramp connection that used to work doesn't now. At first I thought it might be an Aquamacs problem, but I loaded Emacs.app on OSX, based on version 23.3 (9.0), and it doesn't work either. The remote host runs FreeBSD 7.4 and the user shell defaults to /usr/local/bin/bash.
If I try to open a file such as:
Find file: /username#remote_host:~/folder/
...the cursor in the minibuffer stops at:
Find file: /username#remote_host
...and I can't type anything further. Confounding this, I can't C-g to stop it, either. Emacs just locks up. It times out after a minute or so, but until then is unusable.
The relevant lines of my .emacs file are:
(require 'tramp)
(setq tramp-debug-buffer t)
(setq tramp-verbose 9)
(setq tramp-default-method "ssh")
(setq tramp-password-prompt-regexp ".*[Pp]assword: *$")
(setq tramp-shell-prompt-pattern "^[^;$#>]*[;$#>] *")
(setq password-cache-expiry nil)
And the password and shell prompts on the machine in question (anonymized) look like:
Password:
[username#remote_host ~]$ # with a single space at the end
The value of tramp-password-prompt-regexp is ".*[Pp]assword: *$"
The value of tramp-shell-prompt-pattern is "^[^;$#>]*[;$#>] *"
And lastly, the last few lines of the debug buffer:
14:36:40.271851 tramp-get-connection-property (7) # check-remote-echo nil
14:36:41.090876 tramp-wait-for-regexp (6) #
are you awake
#$
14:36:41.091037 tramp-wait-for-regexp (1) # File error: [[Regexp `\(^\|\)[^#$
?$' not found in 60 secs]]
I'm not sure where the different version of the shell regexp are coming from, but in any case, I'm never even getting to the password prompt.
Any ideas what I might be doing wrong?
Addendum
Using GNU Emacs 22.1.1 (mac-apple-darwin, Carbon Version 1.6.0), Tramp version 2.0.55, works. I'm reluctant to rely on that, however, as it is based on the terminal window and I've shuffled Cmd, Ctrl, and Caps Lock around to make my workstation feel more like my home machine, then told Emacs to treat the Cmd key as an alternate Ctrl; unfortunately, the terminal window traps several Cmd-key combinations.
Furthermore
Downloaded the Emacs-22.3-i386-10.5.7.dmg from Emacs For OS X, which uses tramp 2.0.58-pre. That works too, and has fewer issues with the environment trapping key combinations. But I still don't understand why the Aquamacs and later Carbon Emacs versions quit working.

Why is a Remote WMI call to Win32_Printer Coming Back With An Empty Set For Some Machines?

I am using WMI (prototyping everything in VBScript first, as examples are more plentiful and it removes VBScript/Python impedence) to connect remotely to a fresh PC (we will call this PC2). Most Win32_* classes can be remotely read, yet Win32_Printer returns an empty set when queried, but only when I query remotely. The resulting SWbemObjectSet always has a .Count of zero. No error. I can connect to PC1 and receive a SWbemObjectSet with a non-zero .Count, can iterate through it, etc. If I run the script locally (after removing the superuser username and password from the .ConnectServer method, naturally), I get a non-zero .Count back and can iterate through it. Even if I foolishly use my own Domain Administrator account, the problem persists. The Script:
strComputer = "nnn.nnn.nnn.nnn"
username = "DOMAIN\superuser"
password = "thisisaverygoodpassword"
Set objSWbemLocator = CreateObject("WbemScripting.SWbemLocator")
objSWbemLocator.Security_.ImpersonationLevel = 3
objSWbemLocator.Security_.AuthenticationLevel = 6
Set objSWbemServices = objSWbemLocator.ConnectServer(strComputer, "root\cimv2", username, password)
Set colSWBemObjectSet = objSWbemServices.ExecQuery("Select * From Win32_Printer")
WScript.Echo colSWBemObjectSet.Count & " Found."
For Each objPrinter in colSWBemObjectSet
For Each Property in objPrinter.Properties_
If TypeName(Property.Value) = "Variant()" Then
Wscript.Echo """" & Property.Name & """, """ & TypeName(Property.Value) & """, ""Skipping ..."""
Else
Wscript.Echo """" & Property.Name & """, """ & TypeName(Property.Value) & """, """ & Property.Value & """"
End If
Next
Next
Commenting and error checking have been omitted for brevity.
It does not appear to be a firewall problem.
Reason 1: Where a firewall blockage does exist, I receive an error
back from SWbemLocator, "The RPC server is unavailable."
Reason 2: I can access and run through the WMI class Win32_ComputerSystem
with ease.
It does not appear to be a username/password problem.
Reason 1: I can retrieve information from Win32_ComputerSystem.
Reason 2: I ought to get an error.
It does not appear to be an OS version problem:
Reason: PC2 and PC1 are both running Windows 7 Professional. PC1 is running the 64-bit version, PC2 the 32-bit.
Although I started trying to reach a 32-bit machine from a 64-bit server, it does not appear to be a 32-bit vs. 64-bit problem.
Reason 1: I added a value of 32 for __ProviderArchitecture in a SWbemNamedValueSet prior to my .ConnectServer attempt (with that SWbemNamedValueSet in the arguments to no avail), although I was unable to later add that same context to the .ExecQuery method of the connected server without a type mismatch operator.
Reason 2: I later ran the script from a 32-bit server with the same result.
It does not appear to be a corrupted WMI problem.
Reason: Once I stop using credentials, I can run the script from the target machine itself and receive a result set with more than zero items and can iterate through it.
It does not appear to be a credential/namespace mistake within my script.
Reason: Using WBemTest.exe from the same source machine and using identical username, password, authentication level, impersonation level, namespace, and so forth, I receive the same null set for an answer.
It does not appear to be an issue of WMI Namespace security on the target machine.
Reason 1: Logging in to the target machine with the same credentials as the script uses generates results.
Reason 2: Win32_Printer is in the same namespace as Win32_ComputerSystem. Win32_ComputerSystem works.
Reason 3: After using the Wmimgmt.msc Microsoft Management Console (MMC) to give the superuser full permissions, starting in the root namespace, propagating to "This namespace and subnamespaces," rebooting, and checking again, I still receive the same empty set.
It does not appear to be the respective OUs of PC2 and PC1 that are the problem.
Reason: I swapped the OUs each machine was in and rebooted. No change.
It does not appear to be the Local Computer Groups:
Reason: I made the membership of groups in PC2 look like PC1 and rebooted. No change.
It does not appear to be something magical about Win32_Printer in that remote access does not work.
Reason: I can read PC1's Win32_Printer class.
It does not appear that my WQL is unusual.
Reason: "SELECT * FROM Win32_Printer" is my test case.
It does not appear that my DCOM settings are off.
Reason: They appear identical when I go through PC1 and PC2.
I have even gone so far as to hit the Trace logs in WMI-Activity, print them out for both PC1 and PC2, then sort by GroupOperationID, OperationID (the TimeCreated SystemTime is not granular enough and EventID seems ... out of order. I can see events from the following actions:
IWbemServices::Connect
Start IWbemServices::ExecQuery - Select * from __ClassProviderRegistration
Start IWbemServices::GetObject - __Win32Provider.Name="WmiPerfClass"
Start IWbemServices::ExecQuery - references of {__Win32Provider.Name="WmiPerfClass"}
Start IWbemServices::GetObject - Win32_Printer
Start IWbemServices::ExecQuery - Select * From Win32_Printer
Provider::CreateInstanceEnum - Win32_Printer
in both sets of logs, and if I sort by GroupOperationID, OperationID they appear to happen in identical order. Sorting by EventID shows a somewhat different order. That's the closest I can see to a difference. I'm stumped at this point.
I know this verges perilously close to a system administration issue.
Found this link in the win32_printer spec page referring to this problem http://www.lansweeper.com/forum/yaf_postsm18178_WMI-Security-PowershellLansweeper.aspx#post18178 It appears only printers installes for this user are returned, not all printers on the system. So if you've never logged on to the remote system under the credentials of the user you are using to enumerate the printers then you get an empty result.
Looks like you've had a good shot at it. ServerFault might yield something more...
It's a long shot, but I once heard terminal services being disabled aparently caused issues when issuing WMI queries...
Edit:
This may not apply, but could be worth a look: AD Delegation