Related
I have the flow where i want to edit the column in the csv table and replace the "," by a "."
How do I do that? Because the replace function expression in logicApp does not return the column:
It asks me to take the complete body when I use the replace function.
Where as details column is available which I want to edit:
How should I replace the "," from the details column?
I did this then, Then i don't see the variable I initialize.
For instance I've taken this as my sample .csv file which I'm retrieving from my storage account.
Firstly I have used Parse CSV file like you did the same, then initialised and used the Append the string variable connector taking the Productsname column. Lastly, have used the replace function expression to replace ' , ' with a ' . '.
NOTE: I have used '|' following productsname variable for future purpose.
Here is my Logic App workflow
THE COMPOSE CONNECTOR EXPRESSION :-
split(replace(variables('Productname'),',','.'),'|')
OUTPUT:
Here is my workflow that you can refer to:
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": "#split(replace(variables('Productname'),',','.'),'|')",
"runAfter": {
"For_each_2": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each_2": {
"actions": {
"Append_to_string_variable": {
"inputs": {
"name": "Productname",
"value": "#{items('For_each_2')?['Productname']}|"
},
"runAfter": {},
"type": "AppendToStringVariable"
}
},
"foreach": "#body('Parse_CSV')",
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "Foreach"
},
"Get_blob_content_(V2)": {
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azureblob']['connectionId']"
}
},
"method": "get",
"path": "/v2/datasets/#{encodeURIComponent(encodeURIComponent('AccountNameFromSettings'))}/files/#{encodeURIComponent(encodeURIComponent('JTJmY29udGFpbmVyMjQwOCUyZlByb2R1Y3RzLmNzdg=='))}/content"
},
"metadata": {
"JTJmY29udGFpbmVyMjQwOCUyZlByb2R1Y3RzLmNzdg==": "/container2408/Products.csv"
},
"runAfter": {},
"type": "ApiConnection"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "Productname",
"type": "string"
}
]
},
"runAfter": {
"Parse_CSV": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_CSV": {
"inputs": {
"body": {
"content": "#{base64(body('Get_blob_content_(V2)'))}",
"headers": "Productid,Productname"
},
"host": {
"connection": {
"name": "#parameters('$connections')['plumsail']['connectionId']"
}
},
"method": "post",
"path": "/flow/v1/Documents/jobs/ParseCsv"
},
"runAfter": {
"Get_blob_content_(V2)": [
"Succeeded"
]
},
"type": "ApiConnection"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/<subscription id>/resourceGroups/<Your resource group name>/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"id": "/subscriptions/<subscription id>/providers/Microsoft.Web/locations/northcentralus/managedApis/azureblob"
},
"plumsail": {
"connectionId": "/subscriptions/<subscription id >/resourceGroups/<Your resource group name>/providers/Microsoft.Web/connections/plumsail",
"connectionName": "plumsail",
"id": "/subscriptions/<subscription id>/providers/Microsoft.Web/locations/northcentralus/managedApis/plumsail"
}
}
}
}
}
I used items function express and did it directly.
#replace(item()?['details'],',','')
This was a bit strange it didn't work at first but now it is working.
Scenario:
We are using AWS ElasticSearch 6.8. We got an index (index-A) with a mapping structure consist of multiple nested objects and JSON hierarchy. We need to create new index (index-B) and move all documents from index-A to index-B.
We need to create index-B with only specific fields.
We need to rename field names while reindexing
e.g.
index-A mapping:
{
"userdata": {
"properties": {
"payload": {
"type": "object",
"properties": {
"Alldata": {
"Username": {
"type": "keyword"
},
"Designation": {
"type": "keyword"
},
"Company": {
"type": "keyword"
},
"Region": {
"type": "keyword"
}
}
}
}
}
}}
Expected structure of index-B mapping after reindexing with rename (Company-cnm, Region-rg) :-
{
"userdata": {
"properties": {
"cnm": {
"type": "keyword"
},
"rg": {
"type": "keyword"
}
}
}}
Steps we are Following:
First we are using Create index API to create index-B with above mapping structure
Once index is created we are creating an ingest pipeline.
PUT ElasticSearch domain endpoint/_ingest/pipeline/my_rename_pipeline
{
"description": "rename field pipeline",
"processors": [{
"rename": {
"field": "payload.Company",
"target_field": "cnm",
"ignore_missing": true
}
},
{
"rename": {
"field": "payload.Region",
"target_field": "rg",
"ignore_missing": true
}
}
]
}
Perform reindexing operation, payload for the same below
let reindexParams = {
wait_for_completion: false,
slices: "auto",
body: {
"conflicts": "proceed",
"source": {
"size": 8000,
"index": "index-A",
"_source": ["payload.Company", "payload.Region"]
},
"dest": {
"index": "index-B",
"pipeline": "my_rename_pipeline",
"version_type": "external"
}
}
};
Problem:
Once the reindexing is complete as expected all documents transferred to new index with renamed fields but there is one additional field which is not selected. As you can see below the "payload" object with metadata is also added to the new index after reindexing. This field is empty and consist of no data.
index-B looks like below after reindexing:
{
"userdata": {
"properties": {
"cnm": {
"type": "keyword"
},
"rg": {
"type": "keyword"
},
"payload": {
"properties": {
"Alldata": {
"type": "object"
}
}
}
}
}}
We are unable to find the workaround and need help how to stop this field from creating. Any help will be appreciated.
Great job!! You're almost there, you simply need to remove the payload field within your pipeline using the remove processor and you're good:
{
"description": "rename field pipeline",
"processors": [
{
"rename": {
"field": "payload.Company",
"target_field": "cnm",
"ignore_missing": true
}
},
{
"rename": {
"field": "payload.Region",
"target_field": "rg",
"ignore_missing": true
}
},
{
"remove": { <--- add this processor
"field": "payload"
}
}
]
}
I am getting this error after upgrading my api from .netcore2.2 to 3.1 and trying to generate using autorest with the --v3 switch
WARNING: Schema violation: Data does not match any schemas from
'oneOf'
I have tried with and without SerializeAsV2
I see from the Autorest docs that this warning is because of an supported feature.
anyOf, oneOf are not currently supported
In services.AddSwaggerGen I have
c.ParameterFilter<SwaggerEnumParameterFilter>();
c.SchemaFilter<SwaggerEnumFilter>();
where
public void Apply(OpenApiParameter parameter, ParameterFilterContext context)
{
var type = context.ApiParameterDescription.Type;
if (type.IsEnum)
parameter.Extensions.Add("x-ms-enum", new OpenApiObject
{
["name"] = new OpenApiString(type.Name),
["modelAsString"] = new OpenApiBoolean(false)
});
}
public class SwaggerEnumFilter : ISchemaFilter
{
public void Apply(OpenApiSchema model, SchemaFilterContext context)
{
if (model == null)
throw new ArgumentNullException("model");
if (context == null)
throw new ArgumentNullException("context");
if (context.Type.IsEnum)
model.Extensions.Add(
"x-ms-enum",
new OpenApiObject
{
["name"] = new OpenApiString(context.Type.Name),
["modelAsString"] = new OpenApiBoolean(false)
}
);
}
}
[update]
After upgrading to Autorest 3.0.6244 the warnings have changed to errors and the error message ends with
post > parameters > 0)
If I don't use the v3 switch I get the error
FATAL: swagger-document/individual/schema-validator - FAILED
FATAL: Error: [OperationAbortedException] Error occurred. Exiting.
Process() cancelled due to exception : [OperationAbortedException] Error occurred. Exiting.
I can see in the swagger.json that the parameters property "name" is not generating correctly. Here it contains "body" whereas previously it contained "info"
"/api/FrameLookUp": {
"post": {
"tags": [
"Frame"
],
"operationId": "FrameLookup",
"consumes": [
"application/json-patch+json",
"application/json",
"text/json",
"application/*+json"
],
"produces": [
"application/json"
],
"parameters": [
{
"in": "header",
"name": "Authorization",
"description": "access token",
"required": true,
"type": "String"
},
{
"in": "body",
"name": "body",
"schema": {
"$ref": "#/definitions/FrameRequest"
}
}
],
"responses": {
"200": {
"description": "Success",
"schema": {
"$ref": "#/definitions/FrameResponse"
}
}
}
}
},
The controller is
[Produces("application/json")]
[Authorize(AuthenticationSchemes = JwtBearerDefaults.AuthenticationScheme)]
[Route("api")]
public class FrameController : MyController
{
[ProducesResponseType(typeof(FrameResponse), StatusCodes.Status200OK)]
[HttpPost("FrameLookUp")]
public IActionResult FrameLookup([FromBody] FrameRequest info)
{
IMyResponse MyFunc(IMyRequest x) => FrameData.FrameLookUp(info);
return InnerMethod(MyFunc, info);
}
}
Update
I have also tried using the SwaggerParameter from Swashbuckle.AspNetCore.Annotations
[Update]
I am thinking that maybe I just need to try the release for issue 1766
I tried cloning the swashbuckle.aspnetcore repo but ran into this issue
[Update]
I added c.GeneratePolymorphicSchemas(); to the AddSwaggerGen options but it has not helped.
[Update]
Here is the first error message
ERROR: Schema violation: Data does not match any schemas from 'oneOf'
- https://localhost:44348/api-docs/v1/swagger.json:1951:8 ($.paths["/api/synchronise-management/get-product-images-Ids"].post.parameters)
Investigating line 1951 in swagger.json
In the working swagger ( generated from dotnet2.2 project ) the json looks very similar however the parameter order is swapped
The other difference I can see is the generated name of the parameter
I see from this question the error occurs in the same place
[Update]
when I add the --debug switch to the autorest call I get
/configuration
DEBUG: pipeline-emitter - END
DEBUG: configuration-emitter - END
DEBUG: swagger-document-override/md-override-loader - END
DEBUG: swagger-document/loader - END
DEBUG: swagger-document/individual/transform - START
DEBUG: swagger-document/individual/transform - END
DEBUG: swagger-document/individual/schema-validator - START
ERROR: Schema violation: Data does not match any schemas from 'oneOf'
- https://localhost:44348/api/v1/swagger.json:1951:8 ($.paths["/api/synchronise-management/get-product-images-Ids"].
[Update]
Here is the cut down json
{
"swagger": "2.0",
"info": {
"title": "myapi API31",
"description": "ASP.NET Core Web API",
"version": "v1"
},
"host": "localhost:44348",
"basePath": "/v1",
"schemes": [
"https"
],
"paths": {
"/api/Test": {
"get": {
"tags": [
"Auth"
],
"operationId": "Test",
"responses": {
"200": {
"description": "Success"
}
}
}
},
"/api/RequestToken": {
"post": {
"tags": [
"Auth"
],
"operationId": "RequestToken",
"consumes": [
"application/json-patch+json",
"application/json",
"text/json",
"application/*+json"
],
"produces": [
"application/json"
],
"parameters": [
{
"in": "body",
"name": "body",
"schema": {
"$ref": "#/definitions/TokenRequest"
}
}
],
"responses": {
"200": {
"description": "Success",
"schema": {
"$ref": "#/definitions/TokenResponse"
}
}
}
}
},
"/api/FrameLookUp": {
"post": {
"tags": [
"Frame"
],
"operationId": "FrameLookup",
"consumes": [
"application/json-patch+json",
"application/json",
"text/json",
"application/*+json"
],
"produces": [
"application/json"
],
"parameters": [
{
"in": "header",
"name": "Authorization",
"description": "access token",
"required": true,
"type": "String"
},
{
"in": "body",
"name": "body",
"schema": {
"$ref": "#/definitions/FrameRequest"
}
}
],
"responses": {
"200": {
"description": "Success",
"schema": {
"$ref": "#/definitions/FrameResponse"
}
}
}
}
}
},
"definitions": {
"TokenRequest": {
"required": [
"password",
"username"
],
"type": "object",
"properties": {
"username": {
"type": "string"
},
"password": {
"type": "string"
}
}
},
"TokenResponse": {
"type": "object",
"properties": {
"tokenResult": {
"type": "string"
}
}
},
"FramePackTypeEnum": {
"enum": [
"NotApplicable",
"PipeRack",
"LwBVan",
"VanTray",
"Car",
"CarryBag"
],
"type": "string",
"x-ms-enum": {
"name": "FramePackTypeEnum",
"modelAsString": false
}
},
"FrameRequest": {
"type": "object",
"properties": {
"qCodeJobId": {
"format": "int32",
"type": "integer"
},
"quantity": {
"format": "int32",
"type": "integer"
},
"widthInMm": {
"format": "int32",
"type": "integer"
},
"heightInMm": {
"format": "int32",
"type": "integer"
},
"ePackingType": {
"$ref": "#/definitions/FramePackTypeEnum"
},
"userEmail": {
"type": "string"
}
}
},
"FrameCaseEnum": {
"enum": [
"Case0_NoBraces",
"Case1_1Vertical_0Horizontal",
"Case2_2Vertical_0Horizontal",
"Case3_NVertical_0Horizontal",
"Case4_0Vertical_1Horizontal",
"Case5_1Vertical_1Horizontal",
"Case6_2Vertical_1Horizontal",
"Case7_NVertical_1Horizontal",
"Case8_0Vertical_2Horizontal",
"Case9_1Vertical_2Horizontal",
"Case10_2Vertical_2Horizontal",
"Case11_NVertical_2Horizontal",
"Case12_0Vertical_NHorizontal",
"Case13_1Vertical_NHorizontal",
"Case14_2Vertical_NHorizontal",
"Case15_NVertical_NHorizontal"
],
"type": "string",
"x-ms-enum": {
"name": "FrameCaseEnum",
"modelAsString": false
}
},
"FrameResponse": {
"type": "object",
"properties": {
"description": {
"type": "string"
},
"caseNumber": {
"$ref": "#/definitions/FrameCaseEnum"
},
"memberPriceEachExGst": {
"format": "double",
"type": "number"
},
"retailPriceEachExGst": {
"format": "double",
"type": "number"
}
}
}
}
}
With the .netcore2.2 api the request generates as
"FrameRequest": {
"type": "object",
"properties": {
"qCodeJobId": {
"format": "int32",
"type": "integer"
},
"quantity": {
"format": "int32",
"type": "integer"
},
"widthInMm": {
"format": "int32",
"type": "integer"
},
"heightInMm": {
"format": "int32",
"type": "integer"
},
"ePackingType": {
"enum": [
"NotApplicable",
"PipeRack",
"LwBVan",
"VanTray",
"Car",
"CarryBag"
],
"type": "string",
"x-ms-enum": {
"name": "FramePackTypeEnum",
"modelAsString": false
}
},
"userEmail": {
"type": "string"
}
}
}
Here is the command line I am running
autorest --input-file=.\myswagger.json --output-folder=generated --csharp --namespace=DDD --debug
Some links which the author, Kirsten Greed, put in comments:
https://github.com/domaindrivendev/Swashbuckle.AspNetCore#schema-filters
https://github.com/domaindrivendev/Swashbuckle.AspNetCore/pull/1766
https://stackoverflow.com/questions/63857310/could-not-find-a-part-of-the-path-d-dev-swashbuckle-aspnetcore-src-swashbuckle
From your swagger.json we can see the validation shows:
https://validator.swagger.io/validator/debug?url=https://raw.githubusercontent.com/heldersepu/hs-scripts/master/swagger/63783800_swagger.json
{
"schemaValidationMessages": [
{
"level": "error",
"domain": "validation",
"keyword": "oneOf",
"message": "instance failed to match exactly one schema (matched 0 out of 2)",
"schema": {
"loadingURI": "http://swagger.io/v2/schema.json#",
"pointer": "/definitions/parametersList/items"
},
"instance": {
"pointer": "/paths/~1api~1FrameLookUp/post/parameters/0"
}
}
]
}
that lead us to your code:
that type: "String" should be: type: "string" with all lower case the error goes away
I'm working on an OpenAPI 3 schema.
I would like to use a data model from the components.schemas inside the responses content and have some required nested properties inside that data model. However, it doesn't seem like the required validation is being applied. I'm testing this in Postman with a mock server.
Here is my schema:
{
"openapi": "3.0.0",
"info": {
"version": "1.0.0",
"title": "Usage stats API"
},
"servers": [
{
"url": "http://some-middleware-endpoint.com"
}
],
"paths": {
"/publishers/{publisherId}/files/{fileId}": {
"get": {
"summary": "Get single file for publisher",
"parameters": [
{
"name": "publisherId",
"in": "path",
"description": "ID of the publisher",
"required": true,
"schema": {
"type": "integer",
"format": "int64"
}
},
{
"name": "fileId",
"in": "path",
"description": "ID of the file",
"required": true,
"schema": {
"type": "integer",
"format": "int64"
}
}
],
"responses": {
"200": {
"description": "File for publisher",
"headers": {
"Content-Type": {
"description": "application/json"
}
},
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"meta"
],
"properties": {
"meta": {
"type": "object",
"required": ["page"],
"properties": {
"$ref": "#/components/schemas/Pagination"
}
}
}
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"Pagination": {
"properties": {
"page": {
"required": ["current-page", "per-page", "from", "to", "total", "last-page"],
"type": "object",
"properties": {
"current-page": {
"type": "integer"
},
"per-page": {
"type": "integer"
},
"from": {
"type": "integer"
},
"to": {
"type": "integer"
},
"total": {
"type": "integer"
},
"last-page": {
"type": "integer"
}
}
}
}
}
}
}
}
This response passes validation:
{
"meta": {
"page": {}
}
}
Even though all of the attributes I've required ("required": ["current-page", "per-page", "from", "to", "total", "last-page"]) are not present.
Basically, I would like page and all its nested properties to be required.
I guess I'm doing something wrong in defining the properties. Any help is appreciated!
Oh well, I guess my issue was pulling up the $ref one level up.
The following seems to work inside responses.content.
"meta": {
"type": "object",
"required": [
"page"
],
"$ref": "#/components/schemas/Pagination"
}
instead of
"meta": {
"type": "object",
"required": ["page"],
"properties": {
"$ref": "#/components/schemas/Pagination"
}
}
I have an index in Elasticsearch (v 1.5.0) that has a mapping that looks like this:
{
"storedash": {
"mappings": {
"outofstock": {
"_ttl": {
"enabled": true,
"default": 1296000000
},
"properties": {
"CompositeSKUProductId": {
"type": "string"
},
"Hosts": {
"type": "nested",
"properties": {
"HostName": {
"type": "string"
},
"SKUs": {
"type": "nested",
"properties": {
"CompositeSKUProductId": {
"type": "string",
"index": "not_analyzed"
},
"Count": {
"type": "long"
},
"ProductId": {
"type": "string",
"index": "not_analyzed",
"copy_to": [
"CompositeSKUProductId"
]
},
"SKU": {
"type": "string",
"index": "not_analyzed",
"copy_to": [
"CompositeSKUProductId"
]
}
}
}
}
},
"Timestamp": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
}
}
Look at how field CompositeSKUProductId is created as a composition of both the SKU and ProductId fields.
I now want to perform an aggregation on that composite field, but it doesn't seem to work; the relevant part of my query looks like this:
"aggs": {
"hostEspecifico": {
"filter": {
"term": { "Hosts.HostName": "www.example.com"}
},
"aggs": {
"skus": {
"nested": {
"path": "Hosts.SKUs"
},
"aggs": {
"valores": {
"terms": {
"field": "Hosts.SKUs.CompositeSKUProductId", "order": { "media": "desc" }, "size": 100 },
"aggs": {
"media": {
"avg": {
"field": "Hosts.SKUs.Count"
}
}
}
}
}
}
}
}
}
Thing is, this aggregation returned zero buckets, as though it weren't even there.
I checked that the very same query works if only I change CompositeSKUProductId by another field like ProductId.
Any ideas as to what I can do to solve my problem?
N.B.: I'm using the AWS Elasticsearch Service, which does not allow scripting.
The problem here is that you have misunderstood the concept of copy_to functionality. It simply copies the field values of various fields and does not combine the way you would expect.
If SKU is 123 and product id is 456 then composite field will have them as separate values and not 123 456. You can verify this by querying your field.
You would have to do this on server side, ideally with script but it is not allowed. Personally we used AWS ES service but faced multiple problems, major being not able to change elasticsearch.yml file and not able to use scripts. You might want to look at Found.
Hope this helps!
In order to copy_to another field in the nested doc, you need to supply the full path to the field you want to copy to in your mapping. You have only provided "CompositeSKUProductId", which causes the data to be copied to a field in your root document, instead of your nested SKUs type document.
Try updating your mapping for your "SKUs" type to copy_to the fully qualified field "Hosts.SKUs.CompositeSKUProductId" instead.
Like this:
{
"storedash": {
"mappings": {
"outofstock": {
"_ttl": {
"enabled": true,
"default": 1296000000
},
"properties": {
"CompositeSKUProductId": {
"type": "string"
},
"Hosts": {
"type": "nested",
"properties": {
"HostName": {
"type": "string"
},
"SKUs": {
"type": "nested",
"properties": {
"CompositeSKUProductId": {
"type": "string",
"index": "not_analyzed"
},
"Count": {
"type": "long"
},
"ProductId": {
"type": "string",
"index": "not_analyzed",
"copy_to": [
"Hosts.SKUs.CompositeSKUProductId"
]
},
"SKU": {
"type": "string",
"index": "not_analyzed",
"copy_to": [
"Hosts.SKUs.CompositeSKUProductId"
]
}
}
}
}
},
"Timestamp": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
}
}
You may find this discussion helpful, when a similar issue was opened on github.