All row data is not getting reflected in Power BI from Marklogic - powerbi

I have a csv file with 3 rows of data which is stored in Marklogic and available in collections.
I am trying to access the content of the csv file from Power BI.
But from Power BI I can only access 1 row of data at one time whereas requirement is to see all rows of data.
Here are the details of the process I followed to achieve the functionality:
Used Marklogic ODBC driver(64bit) to connect Marklogic SQL from Power BI
Created template(used xquery) to implement the table structure in Marklogic
Used mlcp to import the csv file in the marklogic database.
Here is the template I used:
xquery version "1.0-ml";
import module namespace tde = "http://marklogic.com/xdmp/tde" at "/MarkLogic/tde.xqy";
if (xdmp:database-name(xdmp:database()) = "Documents") then
let $shipment-CBE:=
<template xmlns="http://marklogic.com/xdmp/tde">
<context>/shipment</context>
<collections>
<collections-and>
<collection>PowerBI</collection>
<collection>Shipment</collection>
</collections-and>
</collections>
<rows>
<row>
<schema-name>Shipment</schema-name>
<view-name>CBE2</view-name>
<columns>
<column>
<name>DocType</name>
<scalar-type>string</scalar-type>
<val>DocType</val>
</column>
<column>
<name>User</name>
<scalar-type>string</scalar-type>
<val>User</val>
</column>
<column>
<name>ShipmentRef</name>
<scalar-type>string</scalar-type>
<val>ShipmentRef</val>
</column>
<column>
<name>Transmode</name>
<scalar-type>string</scalar-type>
<val>Trans_mode</val>
</column>
<column>
<name>Packagetype</name>
<scalar-type>string</scalar-type>
<val>Package_type</val>
</column>
<column>
<name>Customer_Party_Name</name>
<scalar-type>string</scalar-type>
<val>Customer_Party_Name</val>
</column>
</columns>
</row>
</rows>
</template>
return (
tde:template-insert("/powerbi/shipment-CBE2.xml",$shipment-CBE),
"shipment-CBE.xml OK"
)
else ("Please select the 'Documents' database.")
Here is the import command(saved as shipment.txt) used to import the csv file:
IMPORT
-input_file_path
../data/Shipment-CBE.csv
-input_file_type
delimited_text
-delimited_root_name
shipment
-username
admin
-password
*****
-host
owc-db01.owc.com
-port
8000
-output_uri_prefix
/powerbi/shipment/
-output_collections
PowerBI,Shipment
-uri_id
"DocType"
Running this MLCP command to execute the import command -
mlcp.bat -options_file shipment.txt

The issue appears to be that you are importing each of the 3 CSV rows and inserting with the same URI of /powerbi/shipment/Shipment
This is because you have configured the -uri_id to be the "DocType" column, and each of your rows have the same value: Shipment.
https://docs.marklogic.com/guide/mlcp/import#id_65814
Optionally, override the default document URI by setting -uri_id to the name of the element from which to derive the document URI.
You could instead configure it to use the User column, so that the docs would be inserted with unique URIs:
/powerbi/shipment/j_henderson
/powerbi/shipment/c_saunders
/powerbi/shipment/a_gatfield

Related

Retrive event logs contains specific string in DATA tag

I have developed a MFC application which reads windows event logs from event log file (EVTX) file and parse it to render in application
For reading log file, I am using XPATH query to retrieve specific event logs from event log file file consist of 40000 records
Sample log records look like below code
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Error_Log"/>
<EventID Qualifiers="20225">6002</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2018-05-31T10:37:16.000000000Z"/>
<EventRecordID>11679958</EventRecordID>
<Channel>Application</Channel>
<Security/>
</System>
<EventData>
<Data>16:07:16.339:(A)[app.exe] [scan] m_id = [1254]</Data>
<Binary>31363A30373A31362E3333393A2841295B7275706170702E6578655D205B5363616E5D206D5F6964203D205B313235345D</Binary>
</EventData>
</Event>
Here I want to retrieve only those log records where <DATA> tag contains sub-string value m_id. To achieve this I tried below query
LPWSTR Query = _T("Event/EventData[Data(Data='m_id')]");
EVT_HANDLE Results = EvtQuery(NULL, Path, Query, EvtQueryFilePath | EvtQueryForwardDirection);
But I am not able to retrieve any logs even if string m_id is present in input log file as shown in above code
You should be able to do this with XPath 2 by using contains()
Full Events:
/Event[EventData/Data[contains(text(),'m_id')]]
/Event[EventData/Data[contains(string(),'m_id')]]
Data Only:
/Event/EventData/Data[contains(string(),'m_id')]
/Event/EventData/Data[contains(text(),'m_id')]
Test Xpaths here
Advanced Xpath Filtering
string vs text

XQuery template in MarkLogic doesn't show any values, only attributes (TDE)

I've created a .xml-file and a template to extract some data, but only the attributes show up.
This is my .xml-testfile:
<user id="1234" email="test.user#live.com" password="1234">
<type>Human</type>
<notes>
<note reference="5432" id="753" xmlns="http://testnamespace.de/note">
<text>example</text>
<username>John Doe</username>
<groups>
<group id="42">Avengers</group>
<group id="55">JLA</group>
</groups>
<distinctiveTitle>title</distinctiveTitle>
<personNameInverted>Doe John</personNameInverted>
</note>
</notes>
and here the corresponding template:
import module namespace tde = "http://marklogic.com/xdmp/tde" at "/MarkLogic/tde.xqy";
declare namespace testns = "http://testnamespace.de/note";
let $userNoteTDE:=
<template xmlns="http://marklogic.com/xdmp/tde" xmlns:testns="http://testnamespace.de/note">
<context>/user/notes/testns:note</context>
<rows>
<row>
<schema-name>user</schema-name>
<view-name>notes</view-name>
<columns>
<column>
<name>reference</name>
<scalar-type>string</scalar-type>
<val>#reference</val>
<nullable>true</nullable>
<default>""</default>
</column>
<column>
<name>id</name>
<scalar-type>string</scalar-type>
<val>#id</val>
<nullable>true</nullable>
<default>""</default>
</column>
<column>
<name>text</name>
<scalar-type>string</scalar-type>
<val>text</val>
<nullable>true</nullable>
<default>""</default>
</column>
<column>
<name>username</name>
<scalar-type>string</scalar-type>
<val>username</val>
<nullable>true</nullable>
<default>""</default>
</column>
<column>
<name>distinctiveTitle</name>
<scalar-type>string</scalar-type>
<val>distinctiveTitle</val>
<nullable>true</nullable>
<default>""</default>
</column>
<column>
<name>personNameInverted</name>
<scalar-type>string</scalar-type>
<val>personNameInverted</val>
<nullable>true</nullable>
<default>""</default>
</column>
</columns>
</row>
</rows>
</template>
I changed the context to use the correct (?) path and the namespace (because this part should be nested into another template):
<context>/user/notes/testns:note</context>
If I check the template with tde:node-data-extract(fn:doc (TESTFILE PATH), $userNoteTDE)
I get the following output:
{
"TESTFILE PATH": [
{
"row": {
"schema": "user",
"view": "notes",
"data": {
"rownum": "1",
"reference": "5432",
"id": "753",
"text": "",
"username": "",
"distinctiveTitle": "",
"personNameInverted": ""
}
}
}
]
}
This shows, that the attributes are displayed correctly, but somehow the values (text, username, distinctiveTitle, personNameInverted) of the elements do not work.
My guess is, that the values need a more refined path or expression, but I can't find any information.
If I change the text value for example to <val>testns:text</val> in my template, I get the error: XDMP-UNBPRFX: (err:XPST0081) Prefix testns has no namespace binding
So somehow the elements can't use the declared namespace, but the attributes can.
Also I skipped the <groups> section in my template, because they would need a context on their own, that shouldn't matter, should it?
Thanks in advance for any helpful insight!
The MarkLogic Support gave me the answer for this problem, so I want to share it here!
(Thanks Chris Hamlin!)
It was indeed a namespace problem. The MarkLogic Documentation shows that for multiple namespaces "path-namespaces" should be used.
After declaring
...
<path-namespaces>
<path-namespace>
<prefix>testns</prefix>
<namespace-uri>http://testnamespace.de/note</namespace-uri>
</path-namespace>
</path-namespaces>
...
between template and context, and using the testns prefix on my elements like testns:text, the elements are being correctly displayed!

Location data for filling the report in JasperServer

I have developed a template design in JasperSoft Studio and upload jrxml file to JasperServer. I want to send data (JSON or XML) to filling report template from my python application and take back report in some popular formats like PDF, XLS using REST API. I do not want to store the data on the server. How can I do this? Or data must be stored on the server and there is no alternative way of their transmission by WEB-Service?
The data does not necessarily have to reside on the server.
You could design your template in such way that you can pass the data via input control parameters as #tobi6 suggested.
Then you could use either the reports service or the reportExecutions service to get the desired output.
In your case, the data could be the actual data(XML or JSON) or the source of the data(a URL to the data file).
Here are some basic samples for working with XML data(for JSON is quite similar):
With actual data as parameter
The JasperReports template:
<?xml version="1.0" encoding="UTF-8"?>
<!-- Created with Jaspersoft Studio version 6.1.1.final using JasperReports Library version 6.1.1 -->
<!-- 2016-05-25T14:18:00 -->
<jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="XmlDSReport_with_data" pageWidth="595" pageHeight="842" columnWidth="555" leftMargin="20" rightMargin="20" topMargin="20" bottomMargin="20" uuid="85d7b9ad-6feb-43dc-84cc-5175bf629546">
<parameter name="xmlString" class="java.lang.String">
<defaultValueExpression><![CDATA["<?xml version=\"1.0\" encoding=\"UTF-8\"?><a><b><val>val1</val></b><b><val>val2</val></b></a>"]]></defaultValueExpression>
</parameter>
<parameter name="XML_INPUT_STREAM" class="java.io.InputStream">
<defaultValueExpression><![CDATA[new java.io.ByteArrayInputStream($P{xmlString}.getBytes("UTF-8"))]]></defaultValueExpression>
</parameter>
<queryString language="xPath">
<![CDATA[/a/b]]>
</queryString>
<field name="value" class="java.lang.String">
<fieldDescription><![CDATA[val]]></fieldDescription>
</field>
<columnHeader>
<band height="31" splitType="Stretch">
<staticText>
<reportElement x="150" y="0" width="100" height="30" uuid="b33a123d-8987-4da4-b21b-1f9ccc50e92d"/>
<text><![CDATA[value]]></text>
</staticText>
</band>
</columnHeader>
<detail>
<band height="30" splitType="Stretch">
<textField>
<reportElement x="150" y="0" width="100" height="30" uuid="14c51219-5ce2-47ce-abb9-71bc11a6f28c"/>
<textFieldExpression><![CDATA[$F{value}]]></textFieldExpression>
</textField>
</band>
</detail>
</jasperReport>
After you deploy the report and create the input control for the xmlString parameter you can test it. Let's say you want to pass this XML instead of leaving the default in place:
<?xml version="1.0" encoding="UTF-8"?>
<a>
<b><val>new_val1</val></b>
<b><val>new_val2</val></b>
</a>
To test the reports service you run something similar to this in a terminal(I URL-encoded the XML string) and check the result:
curl -u user:password \
http://localhost:8080/jasperserver/rest_v2/reports/reports/XmlDSReport_with_data.pdf?xmlString=%3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22UTF-8%22%3F%3E%0A%20%20%20%20%3Ca%3E%0A%20%20%20%20%20%20%3Cb%3E%3Cval%3Enew_val1%3C%2Fval%3E%3C%2Fb%3E%0A%20%20%20%20%20%20%3Cb%3E%3Cval%3Enew_val2%3C%2Fval%3E%3C%2Fb%3E%0A%20%20%20%20%3C%2Fa%3E > report.pdf
To test the reportExecutions service, the main steps are:
1.Create an XML file with the request(name it reportExecutionRequest.xml)
<reportExecutionRequest>
<reportUnitUri>/reports/XmlDSReport_with_data</reportUnitUri>
<outputFormat>pdf</outputFormat>
<freshData>true</freshData>
<saveDataSnapshot>false</saveDataSnapshot>
<interactive>true</interactive>
<allowInlineScripts>true</allowInlineScripts>
<async>false</async>
<parameters>
<reportParameter name="xmlString">
<value><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<a><b><val>new_val1</val></b><b><val>new_val2</val></b></a>]]></value>
</reportParameter>
</parameters>
</reportExecutionRequest>
2.Make the request(you need to save the session cookie to retrieve the output):
curl -u user:password \
-H "Content-Type: application/xml" \
-d #reportExecutionRequest.xml \
-c cookies.txt \
http://localhost:8080/jasperserver/rest_v2/reportExecutions
3.Get the output with the requestID and exportID from the result of the previous request:
curl -b cookies.txt \
http://localhost:8080/jasperserver/rest_v2/reportExecutions/cc57b351-cfb6-429e-8c92-d0aebebbed66/exports/b71d6353-1eec-4304-8713-5d0f3105680e/outputResource > report.pdf
With data as source URL
It is the same report template, but with the two parameters replaced with:
<parameter name="xmlSource" class="java.lang.String">
<defaultValueExpression><![CDATA["http://serverwithdata/xmlData.xml"]]></defaultValueExpression>
</parameter>
<parameter name="net.sf.jasperreports.xml.source" class="java.lang.String">
<defaultValueExpression><![CDATA[$P{xmlSource}]]></defaultValueExpression>
</parameter>
Note: I created two parameters here just because I wanted to keep a shorter name for the parameter when passing it through the reports service. I also created an input control just for the xmlSource parameter.
The tests in this case are similar.
EDIT: To use JSON instead of XML, the original JasperReports template needs to be adjusted in this way:
For data as parameter, just change the xmlString parameter, the XML_INPUT_STREAM parameter and the queryString to this:
<parameter name="jsonString" class="java.lang.String">
<defaultValueExpression><![CDATA["{\"a\": [ {\"b\": { \"val\": \"val1\"}}, {\"b\": { \"val\": \"val2\" }}]}"]]></defaultValueExpression>
</parameter>
<parameter name="JSON_INPUT_STREAM" class="java.io.InputStream">
<defaultValueExpression><![CDATA[new java.io.ByteArrayInputStream($P{jsonString}.getBytes("UTF-8"))]]></defaultValueExpression>
</parameter>
<queryString language="json">
<![CDATA[a.b]]>
</queryString>
For data as source URL, change the xmlString parameter, the XML_INPUT_STREAM parameter and the queryString to this:
<parameter name="jsonSource" class="java.lang.String">
<defaultValueExpression><![CDATA["http://serverwithdata/jsonData.json"]]></defaultValueExpression>
</parameter>
<parameter name="net.sf.jasperreports.json.source" class="java.lang.String">
<defaultValueExpression><![CDATA[$P{jsonSource}]]></defaultValueExpression>
</parameter>
<queryString language="json">
<![CDATA[a.b]]>
</queryString>
The cURL tests for the REST services are basically the same with the main difference that you will be passing JSON instead of XML and use the JSON specific parameters jsonString or jsonSource.
The API is separated in three domains:
Administration services
Repository services
Report services
To generate reports, they first have to be deployed to the server via the repository service. If later on the report needs to be generated, it is called via the report services. Here, it is also possible to start synchronous or asynchronous report generation.
Therefore the report needs to be in the repository.
EDIT
Since you also need to deliver parameters, referring to this example it would work like this:
Add a report parameter
Add a input control for a report parameter
Call the API with your input parameter: http://<host>:<port>/jasperserver[-pro]/rest_v2/reports/reports/samples/EmployeeAccounts.html?EmployeeID=sarah_id
Do as needed for other parameters

Unable to load populator class "LifecycleListPopulator"

I create a WSO2 Governance artifact and i tried to add it an input to display a list of the existing lifecycles:
<field type="options">
<name label="Lifecycle Name">Lifecycle Name</name>
<values
class="org.wso2.carbon.governance.services.ui.utils.LifecycleListPopulator">
</values>
</field>
i also tried whit the class:"org.wso2.carbon.governance.generic.ui.LifecycleListPopulator"
The system logs show:
ERROR {org.wso2.carbon.governance.generic.ui.utils.GenericUIGenerator} - Unable to
load populator class
Does the class who populate the lifecycles list has changed ??
Thanks
Yes there have been some changes in the locations please try to use the following
"org.wso2.carbon.governance.generic.ui.utils.LifecycleListPopulator"
Regards

How to insert a new line or enter in Microsoft Word references style (XSL file)

I have prepared a modified references style for Microsoft word 2010 document. Following part of XSL file indicates how online resources should be displayed in the references list at the end of my word document.
<source type="DocumentFromInternetSite">
<column id="1">
<halign>right</halign>
<valign>top</valign>
<format>{[%RefOrder%]}</format>
</column>
<column id="2">
<halign>left</halign>
<valign>top</valign>
<format>{%Author:1%, }{%InternetSiteTitle|Title%,}{ %URL:l%}{ Published %Year%.}{ Retreived %Month%.}</format>
</column>
<sortkey></sortkey>
</source>
My aim is to have a new line or enter character in the format tag just before { Published %Year%.}{ Retreived %Month%.} in order to have the mentioned information in the next line. I tried to use
or
, but unfortunately did not work.
You may want to try the combination CR+LF,
for a line break.
Add <br></br> in desired location