Searching empty date fields in index - sitecore

Somewhere between Sitecore 7.2 and 8.0 the logic for how empty date fields (i.e. date fields for which the content editor has not selected a value) are stored changed. They used to be be stored as DateTime.MinValue (i.e. 00010101); however, now they are stored as an empty string. Under Sitecore 7.2 I used to be able to run the following line of code to find all items that have no value selected for a given date field:
var myQuery = _searchContext.GetQueryable<MyClass>.Where(item => item.MyDateField == DateTime.MinValue);
Which generated the follow Lucene query: +mydatefield: 00010101
This of course no longer works since the field value in the index is an empty string. I'm not quite sure how to use the ContentSearch API to setup the query since DateTime can't be compared to a null or empty string value. I'm wondering if there's a way to query this new format or if I need to look into modifying how Sitecore stores empty date values to match the old format.

One approach you can take is to define a new boolean computed field that keeps track of the presence of the date field. This will make your queries easier to read, and it does not require special knowledge of how Sitecore matches empty fields. It is also likely to be future proof if a change is made to how the values are stored.
using System;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.ComputedFields;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
namespace YourProject.ComputedFields
{
public class HasDateComputedIndexField : IComputedIndexField
{
public object ComputeFieldValue(IIndexable indexable)
{
Item item = indexable as SitecoreIndexableItem;
const string dateFieldName = "MyDateField";
return
item != null &&
item.Fields[dateFieldName] != null &&
!((DateField)item.Fields[dateFieldName]).DateTime.Equals(DateTime.MinValue) &&
!((DateField)item.Fields[dateFieldName]).DateTime.Equals(DateTime.MaxValue);
}
public string FieldName { get; set; }
public string ReturnType { get; set; }
}
}
The computed field will need to be added to your search configuration and your indexes rebuilt. From there, you can reference the computed field in your search result item class and query as follows:
public class MyClass : PageSearchResultItem
{
[IndexField("has_date")]
public bool HasDate { get; set; }
}
var myQuery = _searchContext.GetQueryable<MyClass>.Where(item => item.HasDate);

I believe you can use a nullable DateTime (DateTime?) for your field which should then have no value when the data is blank.
Your check can then be as simple as checking the HasValue property.
var myQuery = _searchContext.GetQueryable<MyClass>.Where(item => item.MyDateField.HasValue);

Related

Why LSI returns LastEvalutedKey on last record?

I am trying to implement pagination ( forward and backward) in dynamodb using LSI.
i have created LSI on abc attribute which is of type string contains characters in the form "https://mydomain/[A-Za-z1-9-_~]"
When I try to forward paginate upon reaching the last record LastEvaluated key becomes null which is expected behavior however for reverse pagination I am getting LastEvaluatedKey. referred even docs https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-response-LastEvaluatedKey
How do i find what is the last page using query operation, if i want to achieve backward pagination?
my code
params.ScanIndexForward = false;
response = await dynamoDb.query(paramsForQuery).promise();
console.log('LE', response.LastEvaluatedKey);
const arrayLength = response.Items.length;
LastEvalSortKey = {
pk: userId,
originalUrl: response.Items[0].originalUrl,
};
if (sortBy === 'createdAt')
LastEvalSortKey[sortBy] = response.Items[0].createdAt;
if (sortBy === 'updatedAt')
LastEvalSortKey[sortBy] = response.Items[0].updatedAt;
if (sortBy === 'convertedUrl')
LastEvalSortKey[sortBy] = response.Items[0].convertedUrl;
return {
items: response.Items,
nextToken: arrayLength
? Base64.encodeURI(JSON.stringify(LastEvalSortKey))
: prevToken,
prevToken: response.LastEvaluatedKey
? Base64.encodeURI(JSON.stringify(response.LastEvaluatedKey))
: undefined,
};
The documentation you linked to has the answer:
If LastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is when LastEvaluatedKey is empty.
The only way you can know that you've reached the end is to query until you get an empty LastEvaluatedKey.
If you can guarantee that the size of the data in a single page won't come close to the 1 MB query limit, one trick you can use is to query for (page + 1) items.
If the query returns page + 1 items, you can throw away the last item and use it to know that there is another page. If it returns less items and you have a LastEvaluatedKey, you can either assume that there is no additional data, or perform an additional single item query to check whether you're at the end.

Flutter : List<Document>, how to remove duplicate documents according to its title

Let's say I have a class Document like this :
class Document {
String text;
String title;
String date;
}
I created a List of Document (List Document listDocs), and I would like to delete documents that have the same title, and keep ONLY the one with the earliest date.
For example if I have 3 documents with same title (but different texts and dates), I want to keep only the document with the earliest date.
Is there an easy way ?
thanks !!
To sort the list of objects by its date, you can do MyDocumentList.sort((a, b) => a.date.compareTo(b.date));
To make sure there is no second object with the same title you can use a Set. which is
A collection of objects in which each object can occur only once.
In this set you would store all the seen titles. Then you would go over MyDocumentList, try to store its title and only add a new Document to uniqueDocuments, if seenDocumentTitles does not contain the title of the current Document (if seenDocumentTitles.add returns true, meaning the set didn't contain the documents title)
Set<String> seenDocumentTitles = Set<String>();
List<Document> uniqueDocuments = MyDocumentList.where((document) => seenDocumentTitles.add(document.title)).toList();
For an example check this out:
https://dartpad.dartlang.org/20a889b174d2e45f7255d1b110be0627
You can add a function that returns an earlier date time object from two given dates like so:
DateTime longer(String dateString, String dateString2) {
DateTime dateTime = DateFormat("yyyy-MMMM-dd").parse(dateString);
DateTime dateTime2 = DateFormat("yyyy-MMMM-dd").parse(dateString2);
return dateTime.isBefore(dateTime2) ? dateTime : dateTime2;
}
Then, to check which date amongst the three is earlier, you add a method to your class like:
DateTime longest(String dateTime, String dateTime2) {
DateTime thisDate = DateFormat("yyyy-MMMM-dd").parse(this.date);
return thisDate.isBefore(longer(dateTime, dateTime2)) ? thisDate : longer(dateTime, dateTime2);
}
If you want to decouple that method from your Document class you can do so:
DateTime longest(String dateTime, String dateTime2, String dateTime3) {
DateTime firstDate = DateFormat("yyyy-MMMM-dd").parse(dateTime);
return firstDate.isBefore(longer(dateTime2, dateTime3)) ? firstDate : longer(dateTime2, dateTime3);
}

ML.NET Dynamic InputModel

I'm using ML.NET to do Multiclass Classification. I have 3 use cases with different input models(different number of columns and data types) and there will be more to come so it doesn't make sense to have to create a physical file for each input models for every new use cases. I'd like to have preferably just ONE physical file that can adapt to any models if possible and if not, dynamically create the input model at runtime based on the column definitions defined out of a json string retrieved from a table in a Sql Server DB. Is this even possible? If so, can you share the sample codes?
Here are some snippets of the prediction codes that I'd like to make generic :-
public class DynamicInputModel
{
[ColumnName("ColumnA"), LoadColumn(0)]
public string ColumnA { get; set; }
[ColumnName("ColumnB"), LoadColumn(1)]
public string ColumnB { get; set; }
}
PredictionEngine<DynamicInputModel, MulticlassClassificationPrediction> predEngine = _predEnginePool.GetPredictionEngine(modelName: modelName);
IDataView dataView = _mlContext.Data.LoadFromTextFile<DynamicInputModel>(
path: testDataPath,
hasHeader: true,
separatorChar: ',',
allowQuoting: true,
allowSparse: false);
var testDataList = _mlContext.Data.CreateEnumerable<DynamicInputModel>(dataView, false).ToList();
I don't think you can do DynamicInput, however you can create pipelines from one input schema and create multiple different models based on the labels/features. I have an example below that does that...two label columns and you can pass in an array of what feature columns to use for the model. The one downside to this approach is that the input schema (CSV/Database) has to be static (not change on load):
https://github.com/bartczernicki/MLDotNet-BaseballClassification

How to format the id column with SHA1 digests in Rails application?

Without saving SHA1 digest string in table directly. Is it possible to format the column in select statement ?
For example (Hope you know what i mean):
#item = Item.where(Digest::SHA1.hexdigest id.to_s:'356a192b7913b04c54574d18c28d46e6395428ab')
No, not the way you want it. The hexdigest method you're using won't be available at the database level. You could use database-specific functions though.
For example:
Item.where("LOWER(name) = ?", entered_name.downcase)
The LOWER() function will be available to the database so it can pass the name column to it.
For your case, I can suggest two solutions:
Obviously, store the encrypted field in the table. And then match.
key = '356a192b7913b04c54574d18c28d46e6395428ab'
Item.where(encrypted_id: key)
Iterate over all column values (ID, in your case) and find the one that matches:
all_item_ids = Item.pluck("CAST(id AS TEXT)")
item_id = all_item_ids.find{ |val| Digest::SHA1.hexdigest(val) == key }
Then you could use Item.find(item_id) to get the item or Item.where(id: item_id) to get an ActiveRecord::Relation object.

Use DataColumn values as source for ValueList on UltraGridColumn

Is there a way I can use a DataColumn from a DataTable as the ValueList for an UltraGrid column?
I have the column style set to DropDownList. The question is, what do I use for ValueList?
ultraGrid1.DisplayLayout.Bands[0].Columns["col"].Style = ColumnStyle.DropDownList;
ultraGrid1.DisplayLayout.Bands[0].Columns["col"].ValueList = ???
Create a BindableValueList. One of the Constructors takes the object for the datasource, the dataMember, the displayMember, valueMember, and bindingContextControl.
The line of code to set the ValueList:
this.ultraGrid1.DisplayLayout.Bands[0].Columns["col"].ValueList = new BindableValueList(dt, "", "displayColumn","valueColumn", this.ultraGrid1);
In the above example, dt is the DataTable you are binding to, displayColumn and valueColumn would be the keys for the columns that you want to use as the display and value portions of the drop down. If you are binding to a DataSet and want to bind to a table other than the first table you would use the second parameter to pass in the name of the table to bind to.
This could be of help. It a simple procedure that receives a DataTable, the value for the Key property of the ValueList and the name of the field to use as list for the the ValueList and returns a Value list to set in one of your UltraGridColums
public ValueList DataTableToValueList(DataTable dt, string vlKey, string fieldName)
{
ValueList vl = new ValueList();
if (!string.IsNullOrEmpty(vlKey)) vl.Key = vlKey;
foreach (DataRowView r in dt.DefaultView)
vl.ValueListItems.Add(r[fieldName]);
return vl;
}
use in this way (probably in the InitializeLayout event)
grd.DisplayLayout.Bands[0].Columns["col"].ValueList =
DataTableToValueList(dtCustomers,"vlCustomer", "CustomerName");