I am having difficulty understanding the Gremlin data load format (for use with Amazon Neptune).
Say I have a CSV with the following columns:
date_order_created
customer_no
order_no
zip_code
item_id
item_short_description
The requirements for the Gremlin load format are that the data is in an edge file and a vertex file.
The edge file must have the following columns: id, label, from and to.
The vertex file must have: id and label columns.
I have been referring to this page for guidance: https://docs.aws.amazon.com/neptune/latest/userguide/bulk-load-tutorial-format-gremlin.html
It states that in the edge file, the from column must equate to "the vertex ID of the from vertex."
And that (in the edge file) the to column must equate to "the vertex ID of the to vertex."
My questions:
Which columns need to be renamed to id, label, from and to? Or, should I add new columns?
Do I only need one vertex file or multiple?
You can have one or more of each CSV file (nodes, edges) but it is recommended to use fewer large files rather than many smaller ones. This allows the bulk loader to split the file up and load it in a parallel fashion.
As to the column headers, let's say you had a node (vertex) file of the form:
~id,~label,name,breed,age:Int
dog-1,Dog,Toby,Retriever,11
dog-2,Dog,Scamp,Spaniel,12
The edge file (for dogs that are friends), might look like this
~id,~label,~from,~to
e-1,FRIENDS_WITH,dog-1,dog-2
In Amazon Neptune, so long as they are unique, any user provided string can be used as a node or edge ID. So in your example, if customer_no is guaranteed to be unique, rather than store it as a property called customer_no you could instead make it the ~id. This can help later with efficient lookups. You can think of the ID as being a bit like a Primary Key in a relational database.
So in summary, you need to always provide the required fields like ~id and ~label. They are accessed differently using Gremlin steps such as hasLabel and hasId once the data is loaded. Columns with names from your domain like order_no will become properties on the node or edge they are defined with, and will be accessed using Gremlin steps such as has('order_no', 'ABC-123')
To follow on from Kelvin's response and provide some further detail around data modeling...
Before getting to the point of loading the data into a graph database, you need to determine what the graph data model will look like. This is done by first deriving a "naive" approach of how you think the entities in the data are connected and then validating this approach by asking the relevant questions (which will turn into queries) that you want to ask of the data.
By way of example, I notice that your dataset has information related to customers, orders, and items. It also has some relevant attributes related to each. Knowing nothing about your use case, I may derive a "naive" model that looks like:
What you have with your original dataset appears similar to what you might see in a relational database as a Join Table. This is a table that contains multiple foreign keys (the ids/no's fields) and maybe some related properties for those relationships. In a graph, relationships are materialized through the use of edges. So in this case, you are expanding this join table into the original set of entities and the relationships between each.
To validate that we have the correct model, we then want to look at the model and see if we can answer relevant questions that we would want to ask of this data. By example, if we wanted to know all items purchased by a customer, we could trace our finger from a customer vertex to the item vertex. Being able to see how to get from point A to point B ensures that we will be able to easily write graph queries for these questions later on.
After you derive this model, you can then determine how best to transform the original source data into the CSV bulk load format. So in this case, you would take each row in your original dataset and convert that to:
For your vertices:
~id, ~label, zip_code, date_order_created, item_short_description
customer001, Customer, 90210, ,
order001, Order, , 2023-01-10,
item001, Item, , , "A small, non-descript black box"
Note that I'm reusing the no's/ids for the customer, item, and order as the ID for their related vertices. This is always good practice as you can then easily lookup a customer, order, or item by that ID. Also note that the CSV becomes a sparse 2-dimensional array of related entities and their properties. I'm only providing the properties related to each type of vertex. By leaving the others blank, they will not be created.
For your edges, you then need to materialize the relationships between each entity based on the fact that they are related by being in the same row of your source "join table". These relationships did not previously have a unique identifier, so we can create one (it can be arbitrary or based on other parts of the data; it just needs to be unique). I like using the vertex IDs of the two related vertices and the label of the relationship when possible. For the ~from and ~to fields, we are including the vertices from which the relationship is deriving and what it is applying to, respectively:
~id, ~label, ~from, ~to
customer001-has_ordered-order001, has_ordered, customer001, order001
order001-contains-item001, contains, order001, item001
I hope that adds some further color and reasoning around how to get from your source data and into the format that Kelvin shows above.
Using jqGrid 4.15.6-pre - free jqGrid
I am wondering why the following code will not show the specified column.
var cm = $('#nrtslist').jqGrid('getColProp','override');
cm.hidden = false;
This can't be done this way.
All the concept of the grid is that you can read the properties of colModel or any other grid option, but changing it does not mean that it will change something. With other words something must happen in order to change the property.
These properties describe the current status (in most cases) or this is a consequence, not a cause.
To change something in grid you will need to use the appropriate method or do your own one.
In your case you will need to use the showCol or hideCol methods
$('#nrtslist').jqGrid('showCol','override'); // this will show the column
$('#nrtslist').jqGrid('hideCol','override'); // this will hide it.
To add an attribute or feature to a product by webservice, I need the id
To get the id for "red", I must first make a call to
get product_attribute_values?filter[id_category]=9 // 9 is colours
Then from that list I have to
get product_attribute_values/2
get product_attribute_values/4
get product_attribute_values/17
...
to get the names of each attribute_value (green,blue,red) until I find the value I want.
Is there no way to get attribute id by name?
I could do it easily in a join or two in mysql, but I am trying to do a clean import module to use only the webservice.
Because attributes like dimensions and weight are to be stored as an attribute or feature, even if the product schema contains dimensions and weight, the dimension and weight lists are rather large.
A work-around could be to just once scan the attribute/feature/options, and build a translate table in my import software.
Then when a product needs an attribute value, look it up in the cached table, perhaps verify if the tranlation text->id is still correct, and if not, rescan the attributes.
But can it be avoided?
I am writing a Maya plugin in c++, and have a custom MPxLocatorNode.I have added numerous custom attributes to this node with:
MFnNumericAttribute nAttr;
Attrib1 = nAttr.create( "Attribute1", "att1", MFnNumericData::kInt,1.0 );
nAttr.setKeyable(false);
nAttr.setStorable(true);
nAttr.setReadable(true);
nAttr.setWritable(true);
nAttr.setHidden(false);
addAttribute( unitId );
Attrib2 = nAttr.create( "Attribute2", "att2", MFnNumericData::kInt,1.0 );
nAttr.setKeyable(false);
...
However I need to add plenty of these attributes and would therefore like to group them when displayed in the attribute editor. Is there a way to dynamically create groups or separators ?
I am a bit confused by the word grouping, but I'll answer the 2 grouping options.
The first one is to group attributes as a compound. A good example of a compound is a point compound. In short you got a point attribute which owns 3 double attributes. So the x, y, z coordinates are grouped under the Point attribute. If now you go in the Channel Editor, you'll see the grouping effect. In the Attribute Editor and for known / predefined types, controls will be assigned automatically, doing some grouping as well.
Now, and because you mentioned the Attribute Panel, I think you are more interested on how you teach the Attribute Editor to have a different layout than the default one. If you create an AE<my custom node name>Template.mel file, with a function of the same name in it, then you can take control on how the layout is done and group, show, hide, change controls of each individual attributes.
For example I got a custom node which you can download from here. Since it is Python and MEL, you can get the full source code from there.
The node name is adskMathNode.
So I created a AEadskMathNodeTemplate.mel file.
This file contains a function declare like this:
global proc AEasdkMathNodeTemplate (string $nodeName) {
editorTemplate -beginScrollLayout ;
...
}
This is where you describe the new layout.
You'll get plenty of bigger example in the Maya folder at
/scripts/AETemplates
I am trying to figure out how to calculate, then store the layout delta for a rendering programatically. The situation I'm in is that I have a rendering defined on my standard value. It's datasource is empty. I then have a process that creates an item based on that template, but I need to set the datasource on the rendering.
By default, the __Renderings field on the new item is blank (as is expected). So far, I've been able to get a RenderingReference to my rendering, detect that the datasource is blank, but I cannot for the life of me figure out how to set the datasource then store the correct delta in the __Renderings field on my item.
So far I have:
foreach (var device in new DeviceRecords(database).GetAll())
{
foreach (var rendering in myItem.Visualization.GetRenderings(device, false).Where(r => r.RenderingID == renderingId)
{
if (rendering.Settings.DataSource.IsNullOrEmpty())
{
var dataSourceItem = datasourceFolder.Add("Datasource name", dataSourceTemplate);
rendering.Settings.DataSource = dataSourceItem.ID.ToString();
using (new EditingContext(myItem)){
myItem[FieldIDs.LayoutField] == //????
}
}
}
}
My guess is I need to somehow invoke something in XmlDelta, but it looks like all of those methods want some Xml to work with, when all I have is the rendering item.
I wrote some code a while back that tried to extract data source information from Sitecore's XML deltas. I never tried updating it though, but this may work for you.
The class I used was Sitecore.Layouts.LayoutDefinition which is able to parse the XML and if I remember correctly it deals with the business of working out what the correct set of page controls is by combining the delta with the underlying template data. You can construct it like so:
string xml = LayoutField.GetFieldValue(item.Fields["__Renderings"]);
LayoutDefinition ld = LayoutDefinition.Parse(xml);
DeviceDefinition deviceDef = ld.GetDevice(deviceID);
foreach(RenderingDefinition renderingDef in deviceDef.GetRenderings(renderingID))
{
// do stuff with renderingDef.Datasource
}
So I think you can then use the API that LayoutDefinition, DeviceDefinition and RenderingDefinition provides to access the data. There's a bit more info on how I used this in the processImages() function in this blog post: https://jermdavis.wordpress.com/2014/05/19/custom-sitemap-filespart-three/
I think the missing step you're after is that you can modify the data this object stores (eg to set a data source for a particular rendering) and then use the ToXml() method to get back the revised data to store into your Renderings field?
You may be able to find more information by using something like Reflector or DotPeek to look inside the code for how something like the Layout Details dialog box modifies this data in the Sitecore UI.
-- Edited to add --
I did a bit more digging on this topic as I was interested in how to save the data again correctly. I wrote up what I discovered here: https://jermdavis.wordpress.com/2015/07/20/editing-layout-details/