I am unable to find any use case or any documentation where it's mentioned that how we can provide input data to inputHandler without hardocding it while using siddhi as a library.
All i found on internet is inputHandler with static data.
ex :- inputHandler.send(new Object[]{"WSO2", 50f, 30L}); etc..,
I want to know how we can pass JSON data to inputHandler or streams while using siddhi as a library ?
In siddhi a simple data flow can be depicted as below.
source(input) -> sourceMapper -> stream > {queries -> stream} -> .... stream -> sinkMapper -> sink(output)
When using siddhi as a library, InputHandler is used to provide data to a stream.
But when you provide data directly to a stream, mapping won't be happening. Therefore, you have to provide exact values for each attributes of the relevant stream.
If you need to programatically provide data to the source level, you can use a source of type "inMemory" [1] and then provide the data using an InMemoryBroker [2]
Please refer to the code [3] for an example.
[1] https://github.com/wso2-extensions/siddhi-map-json/blob/master/component/src/test/java/org/wso2/extension/siddhi/map/json/sourcemapper/JsonSourceMapperTestCase.java#L52
[2] https://github.com/wso2-extensions/siddhi-map-json/blob/master/component/src/test/java/org/wso2/extension/siddhi/map/json/sourcemapper/JsonSourceMapperTestCase.java#L87
[3] https://github.com/wso2-extensions/siddhi-map-json/blob/master/component/src/test/java/org/wso2/extension/siddhi/map/json/sourcemapper/JsonSourceMapperTestCase.java
Related
In the OneTBB user guide, it says: "You can define your own node types by inheriting from class graph_node, class sender and class receiver ..." (enter link description here).
Now I want create a multi-input and single-output function node type to solve a problem about choose one to output from muilt input data. Is there any involved tutorials or examples to guide the work?
In the problem, some input data to input port may be empty, so join_node + function_node is not very appropriate.
I am attempting to use the C++ StreamWriter class provided by Apache Arrow.
The only example of using StreamWriter uses the low-level Parquet API i.e.
parquet::schema::NodeVector fields;
fields.push_back(parquet::schema::PrimitiveNode::Make(
"string_field", parquet::Repetition::OPTIONAL, parquet::Type::BYTE_ARRAY,
parquet::ConvertedType::UTF8));
fields.push_back(parquet::schema::PrimitiveNode::Make(
"char_field", parquet::Repetition::REQUIRED, parquet::Type::FIXED_LEN_BYTE_ARRAY,
parquet::ConvertedType::NONE, 1));
auto node = std::static_pointer_cast<parquet::schema::GroupNode>(
parquet::schema::GroupNode::Make("schema", parquet::Repetition::REQUIRED, fields));
The end result is a std::shared_ptr<parquet::schema::GroupNode> that can then be passed into StreamWriter.
Is it possible to construct and use "high-level" Arrow schemas with StreamWriter? They are supported when using the WriteTable function (non-streaming) but I've found no examples of its use with the streaming API.
I, of course, can resort to using the low-level API but it is extremely verbose when creating large and complicated schema and I would prefer (but not need) to use the high-level Arrow schema mechanisms.
For example,
std::shared_ptr<arrow::io::FileOutputStream> outfile_;
PARQUET_ASSIGN_OR_THROW(outfile_, arrow::io::FileOutputStream::Open("test.parquet"));
// construct an arrow schema
auto schema = arrow::schema({arrow::field("field1", arrow::int64()),
arrow::field("field2", arrow::float64()),
arrow::field("field3", arrow::float64())});
// build the writer properties
parquet::WriterProperties::Builder builder;
auto properties = builder.build()
// my current best attempt at converting the Arrow schema to a Parquet schema
std::shared_ptr<parquet::SchemaDescriptor> parquet_schema;
parquet::arrow::ToParquetSchema(schema.get(), *properties, &parquet_schema); // parquet_schema is now populated
// and now I try and build the writer - this fails
auto writer = parquet::ParquetFileWriter::Open(outfile_, parquet_schema->group_node(), properties);
The last line fails because parquet_schema->group_node() (which is the only way I'm aware of to get access to the GroupNode for the schema) returns a const GroupNode* whereas ParquetFileWriter::Open) needs a std::shared_ptr<GroupNode>.
I'm not sure casting-away the constness of the returned group node and forcing it into the ::Open() call is an officially supported (or correct) usage of StreamWriter.
Is what I want to do possible?
It seems that you need to use low-level api for StreamWriter.
a very tricky way:
auto writer = parquet::ParquetFileWriter::Open(outfile_, std::shared_ptr<parquet::schema::GroupNode>(const_cast<parquet::schema::GroupNode *>(parquet_schema->group_node())), properties);
you may need to converter schema manually.
source code cpp/src/parquet/arrow/schema.cc may help you.
PARQUET_EXPORT
::arrow::Status ToParquetSchema(const ::arrow::Schema* arrow_schema,
const WriterProperties& properties,
std::shared_ptr<SchemaDescriptor>* out);
I have a nice format and a console log:
auto fmtStream = expressions::stream
<< "LineID: " << expressions::attr<unsigned int>("LineID") << " "
<< "Message: " << expressions::message:
boost::log::add_console_log(std::cout, keywords::format = fmtStream);
The format stream is of course somewhat longer..
Now I want to give a user to configure the logging:
std::ifstream file("log.conf");
init_from_stream(file);
A lot of formatting used in fmtStream is not possible using the format string in the config file.
How can I give the user the possibility to modify the console sink, e.g. add a filter? But I want to keep the format string as a default.
Possibilities I see for that:
1) Give the console log that I define in my code a name. The user could now modify it with a sink of the same name.
2) Set a default format that is taken for all sinks. But according to Boost Log changing the default logging::core formatter? this is not possible.
3) Any other ideas?
Thanks!
init_from_stream and init_from_settings functions will initialize the library as specified in the settings. These functions are designed to configure the library from scratch, so they will add new sinks with the specified settings, including filters and formatters. If all you need is to customize formatter for your existing sink and not allow full logging configuration then you should interpret the settings file yourself.
You can parse the settings file with the parse_settings function. From it you will receive a settings (or wsettings) object which you can analyze and modify as described here and here (sorry for the bad formatting in the reference docs). Since you're probably not planning to support all sinks and parameters supported by Boost.Log, you are not bound to the semantics Boost.Log puts in the parameters and can interpret the settings in any way you want. For instance, you may choose to only read the sink formatter:
boost::log::settings setts = boost::log::parse_settings(file);
if (boost::optional<std::string> fmt = setts["MySink"]["Format"])
{
// Sink format is specified in the setting file
}
Now, to turn this format string into a formatter, you will need the parse_formatter function (the string format is described here). This function returns a formatter object that you can install into your sink, provided that you saved a pointer to it.
auto sink = boost::log::add_console_log(std::cout, keywords::format = fmtStream);
boost::log::settings setts = boost::log::parse_settings(file);
if (boost::optional<std::string> fmt = setts["MySink"]["Format"])
{
sink->set_formatter(boost::log::parse_formatter(fmt.get()));
}
There's one more thing to remember though. If you're using attribute values of custom types in your formatter, like enums for severity, for example, you will have to register those types in the library before parsing the formatter. This way the parser will be able to create a formatter that knows your types and uses appropriate formatting operators. There's a tutorial describing how to do that.
This questions is addressed to developers using C++ and the NDK of Nuke.
Context: Assume a custom Op which implements the interfaces of DD::Image::NoIop and
DD::Image::Executable. The node iterates of a range of frames extracting information at
each frame, which is stored in a custom data structure. An custom knob, which is a member
variable of the above Op (but invisible in the UI), handles the loading and saving
(serialization) of the data structure.
Now I want to exchange that data structure between Ops.
So far I have come up with the following ideas:
Expression linking
Knobs can share information (matrices, etc.) using expression linking.
Can this feature be exploited for custom data as well?
Serialization to image data
The custom data would be serialized and written into a (new) channel. A
node further down the processing tree could grab that and de-serialize
again. Of course, the channel must not be altered between serialization
and de-serialization or else ... this is a hack, I know, but, hey, any port
in a storm!
GeoOp + renderer
In cases where the custom data is purely point-based (which, unfortunately,
it isn't in my case), I could turn the above node into a 3D node and pass
point data to other 3D nodes. At some point a render node would be required
to come back to 2D.
I am going into the correct direction with this? If not, what is a sensible
approach to make this data structure available to other nodes, which rely on the
information contained in it?
This question has been answered on the Nuke-dev mailing list:
If you know the actual class of your Op's input, it's possible to cast the
input to that class type and access it directly. A simple example could be
this snippet below:
//! #file DownstreamOp.cpp
#include "UpstreamOp.h" // The Op that contains your custom data.
// ...
UpstreamOp * upstreamOp = dynamic_cast< UpstreamOp * >( input( 0 ) );
if ( upstreamOp )
{
YourCustomData * data = yourOp->getData();
// ...
}
// ...
UPDATE
Update with reference to a question that I received via email:
I am trying to do this exact same thing, pass custom data from one Iop
plugin to another.
But these two plugins are defined in different dso/dll files.
How did you get this to work ?
Short answer:
Compile your Ops into a single shared object.
Long answer:
Say
UpstreamOp.cpp
DownstreamOp.cpp
define the depending Ops.
In a first attempt I compiled the first plugin using only UpstreamOp.cpp,
as usual. For the second plugin I compiled both DownstreamOp.cpp and
UpstreamOp.cpp into that plugin.
Strangely enough that worked (on Linux; didn't test Windows).
However, by overriding
bool Op::test_input( int input, Op * op ) const;
things will break. Creating and saving a Comp using the above plugins still
works. But loading that same Comp again breaks the connection in the node graph
between UpstreamOp and DownstreamOp and it is no longer possible to connect
them again.
My hypothesis is this: since both plugins contain symbols for UpstreamOp it
depends on the load order of the plugins if a node uses instances of UpstreamOp
from the first or from the second plugin. So, if UpstreamOp from the first plugin
is used then any dynamic_cast in Op::test_input() will fail and the two Op cannot
be connected anymore.
It is still surprising that Nuke would even bother to start at all with the above
configuration, since it can be rather picky about symbols from plugins, e.g if they
are missing.
Anyway, to get around this problem I did the following:
compile both Ops into a single shared object, e.g. myplugins.so, and
add TCL script or Python script (init.py/menu.py)which instructs Nuke how to load
the Ops correctly.
An example for a TCL scripts can be found in the dev guide and the instructions
for your menu.py could be something like this
menu = nuke.menu( 'Nodes' ).addMenu( 'my-plugins' )
menu.addCommand('UpstreamOp', lambda: nuke.createNode('UpstreamOp'))
menu.addCommand('DownstreamOp', lambda: nuke.createNode('DownstreamOp'))
nuke.load('myplugins')
So far, this works reliably for us (on Linux & Windows, haven't tested Mac).
I am trying to use transform a vtkPolyData object by using vtkTransform.
However, the tutorials I found are using pipeline, for example: http://www.vtk.org/Wiki/VTK/Examples/Cxx/Filters/TransformPolyData
However, I am using VTK 6.1 which has removed thge GetOutputPort method for stand-alone data object as mentioned here:
http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput
I have tried to replace the line:
transformFilter->SetInputConnection()
with
transformFilter->SetInputData(polydata_object);
Unfortunately, the data was not read properly (as the pipeline was not set correctly?)
Do you know how to correctly transform a stand-alone vtkPolyData without using pipeline in VTK6?
Thank you!
GetOutputPort was never a method on a data-object. It was always a method on vtkAlgorithm and it still is present on vtkAlgorithm (and subclasses). Where is the polydata_object coming from? If it's an output of a reader, you have two options:
// update the reader to ensure it executes and reads data.
reader->UpdatePipeline()
// now you can get access to the data object.
vtkSmartPointer<vtkPolyData> data = vtkPolyData::SafeDownCast(reader->GetOutputDataObject(0));
// pass that to the transform filter.
transformFilter->SetInputData(data.GetPointer());
transformFilter->Update();
Second option is to simply connect the pipeline:
transformFilter->SetInputConnection(reader->GetOutputPort());
The key is to ensure that the data is updated/reader before passing it to the transform filter, when not using the pipeline.