Related
Not able to get desired output. Getting “Unrecognized expression ‘$regex’” error
[
{
'$lookup': {
'from': 'profiles',
'let': {
'userId': '$userId',
},
'pipeline': [
{
'$match': {
$expr: {
$and: [
{ '$eq': ['$uniqueId', '$$mcontactId'] },
{
$or: [{ 'birthDate': { '$regex': '$$today' } },
{ 'spouseBirthdate': { '$regex': '$$today' } },
{ 'weddingAnniversary': { '$regex': '$$today' } },
],
},
],
},
},
},
{
'$project': {
'userId': 1,
'uniqueId': 1,
'mobileNumber': 1,
'whatsApp': 1,
'emailId': 1,
'lastName': 1,
'firstName': 1,
'address': 1,
'signature': 1,
},
},
],
'as': 'profile',
},
},
]
$regex is a query operator, you are trying to use it within an $expr which uses the "aggregation" language as oppose to the "query" language normally used within a $match stage.
Apart from that you have some other issue's in your pipeline, for example you only define $userId as a variable for the $lookup stage but in it you're trying to use $$today and $$mcontactId which are not defined anywhere.
Regardless once you sort out those issue's you have two options:
if the regex match is not related to the input variables just use $regex outside the $expr, like so:
{
'$match': {
$and: [
{
$expr: {
'$eq': [
'$uniqueId',
'$$userId',
],
},
},
{
$or: [
{
'birthDate': {
'$regex': '03-05',
},
},
],
},
],
},
},
Mongo Playground
if the regex does not to use an input variable from the $lookup then you need to use an aggregation operator, like $regexMatch to do the match within the $expr
I have specific situation. I have something like common module that define common behaviour of all integrations and all integrations is using that as a library. In that common module, I have configuration of jest and set of tests.
Everything works OK in case that common module is linked (npm link) to integration. But when this this common module is installed directly from npm, jest is not able to find any tests. I suppose, there are some files ignored, but I was not abe to find any configuration that would solve this situation.
Final configuration from debug mode looks like this:
{
"configs": [
{
"automock": false,
"cache": true,
"cacheDirectory": "/private/var/folders/mv/v_zhxfq113qf8d6vsf4ldwq80000gn/T/jest_dx",
"clearMocks": false,
"coveragePathIgnorePatterns": [
"/node_modules/"
],
"cwd": "<path>/Workspace/integration/my_integration",
"detectLeaks": false,
"detectOpenHandles": false,
"errorOnDeprecated": false,
"extraGlobals": [],
"forceCoverageMatch": [],
"globals": {},
"haste": {
"computeSha1": false,
"throwOnModuleCollision": false
},
"moduleDirectories": [
"node_modules"
],
"moduleFileExtensions": [
"js",
"json",
"jsx",
"ts",
"tsx",
"node"
],
"moduleNameMapper": [],
"modulePathIgnorePatterns": [],
"name": "24eedccdafdba030f3d9209ab1064c8e",
"prettierPath": "prettier",
"resetMocks": false,
"resetModules": false,
"restoreMocks": false,
"rootDir": "<path>/Workspace/integration/my_integration/node_modules/common_module/tests/api",
"roots": [
"<path>/Workspace/integration/my_integration/node_modules/common_module/tests/api"
],
"runner": "jest-runner",
"setupFiles": [],
"setupFilesAfterEnv": [],
"skipFilter": false,
"slowTestThreshold": 5,
"snapshotSerializers": [],
"testEnvironment": "<path>/Workspace/integration/my_integration/node_modules/jest-environment-node/build/index.js",
"testEnvironmentOptions": {},
"testLocationInResults": false,
"testMatch": [
"**/__tests__/**/*.[jt]s?(x)",
"**/?(*.)+(spec|test).[tj]s?(x)"
],
"testPathIgnorePatterns": [
"/node_modules/"
],
"testRegex": [],
"testRunner": "<path>/Workspace/integration/my_integration/node_modules/jest-jasmine2/build/index.js",
"testURL": "http://localhost",
"timers": "real",
"transform": [],
"transformIgnorePatterns": [
"/node_modules/",
"\\.pnp\\.[^\\/]+$"
],
"watchPathIgnorePatterns": []
}
],
"globalConfig": {
"bail": 0,
"changedFilesWithAncestor": false,
"collectCoverage": false,
"collectCoverageFrom": [],
"coverageDirectory": "<path>/Workspace/integration/my_integration/node_modules/common_module/coverage",
"coverageProvider": "babel",
"coverageReporters": [
"json",
"text",
"lcov",
"clover"
],
"detectLeaks": false,
"detectOpenHandles": false,
"errorOnDeprecated": false,
"expand": false,
"findRelatedTests": false,
"forceExit": false,
"json": false,
"lastCommit": false,
"listTests": false,
"logHeapUsage": false,
"maxConcurrency": 5,
"maxWorkers": 7,
"noStackTrace": false,
"nonFlagArgs": [],
"notify": false,
"notifyMode": "failure-change",
"onlyChanged": false,
"onlyFailures": false,
"passWithNoTests": false,
"projects": [],
"rootDir": "<path>/Workspace/integration/my_integration/node_modules/common_module/tests/api",
"runTestsByPath": false,
"skipFilter": false,
"testFailureExitCode": 1,
"testPathPattern": "",
"testSequencer": "<path>/Workspace/integration/my_integration/node_modules/#jest/test-sequencer/build/index.js",
"updateSnapshot": "new",
"useStderr": false,
"verbose": true,
"watch": false,
"watchAll": false,
"watchman": true
},
"version": "26.4.0"
}
All tests are defined on path <path>/Workspace/integration/my_integration/node_modules/common_module/tests/api named as e.g. something.test.js.
Console will show allways:
No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0
No files found in <path>/Workspace/integration/my_integration/node_modules/common_module/tests/api.
Make sure Jest's configuration does not exclude this directory.
To set up Jest, make sure a package.json file exists.
Jest Documentation: facebook.github.io/jest/docs/configuration.html
Pattern: - 0 matches
npm ERR! Test failed. See above for more details.
I want to use the Layout-as-a-Service (LaaS) feature of DotCMS. This approach is documented in
https://dotcms.com/blog/post/more-than-a-headless-cms-layout-as-a-service-in-dotcms and also
in https://github.com/fmontes/dotcms-page.
Both articles suggest, that the DotCMS page API should return a field called "layout" in the response, e.g. to http://localhost:8080/api/v1/page/json/test-page
test-page is a page, which is using a standard template. By standard template, I mean template created with "Template Designer", with a 20% sidebar on the left, one 100% width column, both containing "Blank container".
No matter what I try, the "layout" field is never part of the response. All I get is this:
{
"errors": [],
"entity": {
"canCreateTemplate": true,
"containers": ...,
"numberContents": 2,
"page": ...,
"site": ...,
"template": ...,
"viewAs": ... },
"messages": [],
"i18nMessagesMap": {},
"permissions": []
}
I tried DotCMS version 5.2.0 and also 5.2.3.
Is this perhaps a feature of the Enterprise edition only?
Edit: What I expect:
{
"errors": [],
"entity": {
"canCreateTemplate": true,
"containers": ...,
"layout": {
"width": "responsive",
"title": "mytemplate1",
"header": true,
"footer": true,
"body": {
"rows": [
{
"columns": [
{
"containers": [
{
"identifier": "b5ea1513-7653-4602-a729-97cd8dd099b6",
"uuid": "1582123997023"
}
],
"widthPercent": 100,
"leftOffset": 1,
"styleClass": null,
"preview": false,
"width": 12,
"left": 0
}
],
"styleClass": null
}
]
},
"sidebar": {
"containers": [
{
"identifier": "b5ea1513-7653-4602-a729-97cd8dd099b6",
"uuid": "1582123991866"
}
],
"location": "left",
"width": "small",
"widthPercent": 20,
"preview": false
}
}
...
Nevermind, I found the answer myself.
I checked the source code. In com.dotmarketing.portlets.htmlpageasset.business.render.page.HTMLPageAssetRenderedBuilder#build there is following piece of code:
final TemplateLayout layout = template != null && template.isDrawed() && !LicenseManager.getInstance().isCommunity()
? DotTemplateTool.themeLayout(template.getInode()) : null;
So the answer is, that you can use LaaS promoted by DotCMS only if you have a non-community licence.
Your page needs to use a "layout template" in order to retrieve the layout from dotCMS - using an advanced template won't work.
Also, layouts are part of the enterprise edition. Make sure you have a license in order to use the layout manager.
I am trying to use extended grep to extract data from a JSON. The regex I use is functional on my regexr instance, but for some reason it doesn't work in bash.
I tried many things, notably the bare double dash and various minor edits to the regex for escaping.
#!/bin/bash
networks='{ "networks": [ { "admin_state_up": true, "availability_zone_hints": [], "availability_zones": [], "created_at": "2019-03-12T23:45:13Z", "description": "", "id": "7188504a-72cb-4590-a9b0-414732017837", "ipv4_address_scope": null, "ipv6_address_scope": null, "is_default": false, "mtu": 1450, "name": "BLUE", "port_security_enabled": true, "project_id": "187d635aec4c43fe8e8918afb3a5c82e", "provider:network_type": "vxlan", "provider:physical_network": null, "provider:segmentation_id": 86, "revision_number": 2, "router:external": false, "shared": false, "status": "ACTIVE", "subnets": [], "tags": [], "tenant_id": "187d635aec4c43fe8e8918afb3a5c82e", "updated_at": "2019-03-12T23:45:13Z" }, { "admin_state_up": true, "availability_zone_hints": [], "availability_zones": [], "created_at": "2019-03-12T23:45:13Z", "description": "", "id": "ed82083f-0a7c-4322-a4fb-de8db23e2bae", "ipv4_address_scope": null, "ipv6_address_scope": null, "is_default": false, "mtu": 1450, "name": "RED", "port_security_enabled": true, "project_id": "187d635aec4c43fe8e8918afb3a5c82e", "provider:network_type": "vxlan", "provider:physical_network": null, "provider:segmentation_id": 108, "revision_number": 2, "router:external": false, "shared": false, "status": "ACTIVE", "subnets": [], "tags": [], "tenant_id": "187d635aec4c43fe8e8918afb3a5c82e", "updated_at": "2019-03-12T23:45:13Z" }, { "admin_state_up": true, "availability_zone_hints": [], "availability_zones": [], "created_at": "2019-03-12T23:45:13Z", "description": "", "id": "1eb6647e-869e-4e83-9468-43e2c320bccc", "ipv4_address_scope": null, "ipv6_address_scope": null, "is_default": false, "mtu": 1450, "name": "public", "port_security_enabled": true, "project_id": "187d635aec4c43fe8e8918afb3a5c82e", "provider:network_type": "vxlan", "provider:physical_network": null, "provider:segmentation_id": 32, "revision_number": 2, "router:external": false, "shared": false, "status": "ACTIVE", "subnets": [], "tags": [], "tenant_id": "187d635aec4c43fe8e8918afb3a5c82e", "updated_at": "2019-03-12T23:45:13Z" } ] }'
result=`echo $networks | grep -oE '"(id|name)": "([\w+-]+)"'`
echo $result
The aforementioned code doesn't work but if I switch to the following regex, it works. I just need to add extraction for id field too to be able to extract ids and names using \2 back reference (group 2)
grep -oE '"(id|name)": "(\w+)"'
Can you help me understand why the script doesn't work?
Full formatted JSON
{
"networks": [{
"admin_state_up": true,
"availability_zone_hints": [],
"availability_zones": [],
"created_at": "2019-03-12T23:45:13Z",
"description": "",
"id": "7188504a-72cb-4590-a9b0-414732017837",
"ipv4_address_scope": null,
"ipv6_address_scope": null,
"is_default": false,
"mtu": 1450,
"name": "BLUE",
"port_security_enabled": true,
"project_id": "187d635aec4c43fe8e8918afb3a5c82e",
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 86,
"revision_number": 2,
"router:external": false,
"shared": false,
"status": "ACTIVE",
"subnets": [],
"tags": [],
"tenant_id": "187d635aec4c43fe8e8918afb3a5c82e",
"updated_at": "2019-03-12T23:45:13Z"
}, {
"admin_state_up": true,
"availability_zone_hints": [],
"availability_zones": [],
"created_at": "2019-03-12T23:45:13Z",
"description": "",
"id": "ed82083f-0a7c-4322-a4fb-de8db23e2bae",
"ipv4_address_scope": null,
"ipv6_address_scope": null,
"is_default": false,
"mtu": 1450,
"name": "RED",
"port_security_enabled": true,
"project_id": "187d635aec4c43fe8e8918afb3a5c82e",
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 108,
"revision_number": 2,
"router:external": false,
"shared": false,
"status": "ACTIVE",
"subnets": [],
"tags": [],
"tenant_id": "187d635aec4c43fe8e8918afb3a5c82e",
"updated_at": "2019-03-12T23:45:13Z"
}, {
"admin_state_up": true,
"availability_zone_hints": [],
"availability_zones": [],
"created_at": "2019-03-12T23:45:13Z",
"description": "",
"id": "1eb6647e-869e-4e83-9468-43e2c320bccc",
"ipv4_address_scope": null,
"ipv6_address_scope": null,
"is_default": false,
"mtu": 1450,
"name": "public",
"port_security_enabled": true,
"project_id": "187d635aec4c43fe8e8918afb3a5c82e",
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 32,
"revision_number": 2,
"router:external": false,
"shared": false,
"status": "ACTIVE",
"subnets": [],
"tags": [],
"tenant_id": "187d635aec4c43fe8e8918afb3a5c82e",
"updated_at": "2019-03-12T23:45:13Z"
}]
}
According to man grep:
The Backslash Character and Special Expressions
The symbol \w is a synonym for [[:alnum:]] and \W is a synonym for [^[:alnum:]]. ... A bracket expression is a list of characters enclosed by [ and ]. ... To include a literal ] place it first in the list. Similarly, to include a literal ^ place it anywhere but first. Finally, to include a literal - place it last.
Basically, \w is literally replaced by those characters when evaluated, giving you "([[[:alnum:]]+-]+)", which in a US standard locale gives you "([[a-zA-Z0-9]+-]+)".
Since a bracket expression is truncated by the first ] it sees (unless it is the first element of a bracket expression), the group is only [[[:alnum:]]+, or "1 or more of a digit, letter, and [. This expression is followed by -]+, meaning "exactly one hyphen and one or more ]". This is obviously pretty terrible.
If you try
echo $networks | grep -oE '"(id|name)": "([[:alnum:]+-]+)"'
I.e., \w without the outer bracket expression, the relevant part means "a group (surrounded by ") comprised of one or more digits, letters, hyphens, and plus signs", which outputs:
"id": "7188504a-72cb-4590-a9b0-414732017837"
"name": "BLUE"
"id": "ed82083f-0a7c-4322-a4fb-de8db23e2bae"
"name": "RED"
"id": "1eb6647e-869e-4e83-9468-43e2c320bccc"
"name": "public"
Using PERL (-P) instead of Extended (-E) regexp, looks like the \w is interpreted as expected, without escaping issue: note the -oP
result=$( echo $networks | grep -oP '"(id|name)": "([\w+-]+)"' ) ;
echo $result
"id": "7188504a-72cb-4590-a9b0-414732017837" "name": "BLUE" "id": "ed82083f-0a7c-4322-a4fb-de8db23e2bae" "name": "RED" "id": "1eb6647e-869e-4e83-9468-43e2c320bccc" "name": "public"
As a workaround (it does not resolve the "escaping \w issue)
result=$( echo $networks | grep -oE '"(id|name)": "([a-zA-Z_+-]+)"' ) ;
echo $result
Prints me:
"name": "BLUE" "name": "RED" "name": "public"
Note: prefer using $( ) syntax to execute sub shells rather than the backtick.
is mongo regex ignoring my index? I have a case insentive index, but by the look of things my regex search recognize it and ignores it.
db.getCollection("myCol").find({ value: /^mysearchVal/i }}).explain(...)
I have 95, 708 docs total.
output:
{
"queryPlanner": {
"plannerVersion": 1,
"namespace": "myDb.myCol",
"indexFilterSet": false,
"parsedQuery": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"winningPlan": {
"stage": "FETCH",
"filter": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"Value": 1
},
"indexName": "value_case_insensitive_and_unique",
"collation": {
"locale": "en",
"caseLevel": false,
"caseFirst": "off",
"strength": 2,
"numericOrdering": false,
"alternate": "non-ignorable",
"maxVariable": "punct",
"normalization": false,
"backwards": false,
"version": "57.1"
},
"isMultiKey": false,
"multiKeyPaths": {
"Value": []
},
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"Value": [
"[\"\", {})",
"[/^mysearchVal/i, /^mysearchVal/i]"
]
}
}
},
"rejectedPlans": []
},
"executionStats": {
"executionSuccess": true,
"nReturned": 1,
"executionTimeMillis": 1447,
"totalKeysExamined": 95708,
"totalDocsExamined": 95708,
"executionStages": {
"stage": "FETCH",
"filter": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"nReturned": 1,
"executionTimeMillisEstimate": 1270,
"works": 95709,
"advanced": 1,
"needTime": 95707,
"needYield": 0,
"saveState": 785,
"restoreState": 785,
"isEOF": 1,
"invalidates": 0,
"docsExamined": 95708,
"alreadyHasObj": 0,
"inputStage": {
"stage": "IXSCAN",
"nReturned": 95708,
"executionTimeMillisEstimate": 596,
"works": 95709,
"advanced": 95708,
"needTime": 0,
"needYield": 0,
"saveState": 785,
"restoreState": 785,
"isEOF": 1,
"invalidates": 0,
"keyPattern": {
"Value": 1
},
"indexName": "value_case_insensitive_and_unique",
"collation": {
"locale": "en",
"caseLevel": false,
"caseFirst": "off",
"strength": 2,
"numericOrdering": false,
"alternate": "non-ignorable",
"maxVariable": "punct",
"normalization": false,
"backwards": false,
"version": "57.1"
},
"isMultiKey": false,
"multiKeyPaths": {
"Value": []
},
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"Value": [
"[\"\", {})",
"[/^mysearchVal/i, /^mysearchVal/i]"
]
},
"keysExamined": 95708,
"seeks": 1,
"dupsTested": 0,
"dupsDropped": 0,
"seenInvalidated": 0
}
},
"allPlansExecution": []
},
"ok": 1.0
}
the output shows 95,708 keys and docs examined, 1 doc returned. really? did the index apply in this case or am I missing a point or two?
Case insensitive regular expression queries generally cannot use
indexes effectively. The $regex implementation is not collation-aware
and is unable to utilize case-insensitive indexes.
https://docs.mongodb.com/manual/reference/operator/query/regex/#index-use