I am running into this issue, and I don't know how to get ride of this error:
Some possible reasons: 1) Malformed events 2) Input source configured...
I have develop a c# Console App to write a csv like this into an Azure event Hub:
datacol1;datacol2;datacol3
It's working fine, as I develop a reader to check that data is written correctly.
I try to configure an Azure Stream Analytics Job to read data from the previous Event Hub, but nothing arrived in the input. Logs from the Stream Analytics Job said that:
Some possible reasons: 1) Malformed events 2) Input source configured...
Event Hub is working, Stream Analytics is not working... why?
thx a lot!
Looking at your input - it doesn't look CSV but rather ";" separated values.
Please ensure the console app generated correct input - according to the job input specifications.
You can try your job with a pre-defined input through the portal - click on the input -> Test and see how it reacts.
You can also sample some data from the source in a similar way.
Ho men, I just find the way out!!!
Every CSV data in input to stream analytics must have the header!!!
Ho men!
Related
I need to configure live streaming using OBS studio software, live streaming has already been setup on AWS using ElementalMediaLive(auto wizard). but I am unable to figure out the way to find the server address and stream key which are required the configure the OBS studio.
can some one please guid me where can I find the above required information into AWS panel?
Thanks
That is the "stream name" in the rtmp ingest point URL. The format would be something like
rtmp://domain:1935/live/streamname
stream key in OBS is streamname
For detail, please refer to page 7 and 8 of this pdf: https://d2908q01vomqb2.cloudfront.net/fb644351560d8296fe6da332236b1f8d61b2828a/2020/04/14/Connecting-OBS-Studio-to-AWS-Media-Services-in-the-Cloud-v2.pdf
Have a look here:
https://aws.amazon.com/de/blogs/media/connecting-obs-studio-to-aws-media-services-in-the-cloud/
Especially this document:
https://d2908q01vomqb2.cloudfront.net/fb644351560d8296fe6da332236b1f8d61b2828a/2020/04/14/Connecting-OBS-Studio-to-AWS-Media-Services-in-the-Cloud-v2.pdf
Where it says:
STEP C: CONFIGURE THE APPLIANCE
Launch OBS Studio on the source system. Choose Settings to open the settings window. Choose Stream to access the streaming settings.
Complete the fields:
For Stream Type, choose Custom Streaming Server.
For URL, copy one of the endpoint URLs from the input you created in Step B. Remove the /<stream_name> at the end of the URL.
For Stream key, type the stream name.
Leave the Use authentication box unchecked.
Choose Apply to save your changes.
What OBS refers to as "stream key" is the App Instance.
Regards,
Setup
I use Azure stream analytics to stream data into Azure warehouse staging table.
The input source of the job is a EventHub stream.
I notice when I'm updating the job, the job input event backlog goes up massively after the start.
It looks like the job starting to process the complete EventHub queue again from the beginning.
Questions
how is the stream position management organised in stream analytics
is it possible to define a stream position where the job starts (event after queued after a specific point in time for example)
So far done
I notice a similar question here on StackOverflow.
There is mentioned a variable name "eventStartTime".
But since I use an "asaproj" project within visual studio to create, update and deploy the job I don't know where to place this before deploying.
For updating job without stop, it will use previous setting of "Joboutputstarttime", so it is possible for job starting to process the data from the beginning.
you can stop the job first, then choose "Joboutputstarttime" before you will start the job.
You can reference this document https://learn.microsoft.com/en-us/azure/stream-analytics/start-job to see detailed information for each mode. for your scenario, "When last stopped" mode maybe the one you need and it will not process data from beginning of the eventhub queue.
I am trying to create a template in NiFi just like a data ingest template which provide by kylo.
Basically I want to allow user to select input data source it can be database or a file. If he selects file and then database processor should automatically gets disabled.
I have create a template in NiFi and imported it kylo, but while creating feed It does not show the feed input option.
How I can do this.
While registering the template, in the "Input Properties" section, you have to select which properties have to be shown in feed creation UI for user input i.e. Enable "Allow user input?"
Attached the screenshot for reference:
I think the best approach here would be to use the RouteOnAttribute as the step after the Data Source/ Data type is chosen.
This way you don't have to overthink it.
I have been working on Kylo for around 3 months now and surely know a thing or two about it.
While starting a Feed, Kylo asks you for which source you want to start a feed no matter you have a single processor or multiple which can act as a data producer or fetcher. Once you select one and start a feed, rest of the source processors get disabled automatically by Kylo in the resultant deployment of feed.
I have a Lambda function and its logs in Cloudwatch (Log group and Log Stream). Is it possible to filter (in Cloudwatch Management Console) all logs that contain "error"? For example logs containing "Process exited before completing request".
In Log Groups there is a button "Search Events". You must click on it first.
Then it "changes" to "Filter Streams":
Now you should just type your filter and select the beginning date-time.
So this is kind of a side issue, but it was relevant for us. (I posted this to another answer on StackOverflow but thought it would be relevant to this conversation too)
We've noticed that tailing and searching logs gets really slow after a log group has a lot of Log Streams in it, like when an AWS Lambda Function has had a lot of invocations. This is because "tail" type utilities and searching need to connect to each log stream to run. Log Events get expired and deleted due to the policy you set on the Log Group itself, but the Log Streams never get cleaned up. I made a few little utility scripts to help with that:
https://github.com/four43/aws-cloudwatch-log-clean
Hopefully that save you some agony over waiting for those logs to get searched.
You can also use CloudWatch Insights (https://aws.amazon.com/about-aws/whats-new/2018/11/announcing-amazon-cloudwatch-logs-insights-fast-interactive-log-analytics/) which is an AWS extension to CloudWatch logs that gives a pretty powerful query and analytics tool. However it can be slow. Some of my queries take up to a minute. Okay, if you really need that data.
You could also use a tool I created called SenseLogs. It downloads CloudWatch data to your browser where you can do queries like you ask about. You can use either full text and search for "error" or if your log data is structured (JSON), you can use a Javascript like expression language to filter by field, eg:
error == 'critical'
Posting an update as CloudWatch has changed since 2016:
In the Log Groups there is a Search all button for a full-text search
Then just type your search:
I am tired of one problem so please make things clear to me.
Please read these following three points and help me out.
(1)
I have simply followed this https://www.wowza.com/docs/how-to-start-and-stop-live-stream-recordings-programmatically-livestreamrecordautorecord-example#documentation
I have attached my Application.xml. Now when I publish live stream name "test1" via FMLE it get recorded on server but when I run different instance of FMLE on different PC and publish live stream name "test2" it does not get record and I think it goes to previously recorded file "test1" (means no separate file being record, however there should be two files recorded test1 and test2).
Why this happenning ?
Is this com.wowza.wms.plugin.livestreamrecord.module.ModuleAutoRecordAdvancedExample for single stream recording ? means If I publish stream A B C D , it will record them in one single file ? (probably the output file will be A.mp4 as A was first published stream ?)
(2) What is this https://www.wowza.com/docs/how-to-start-and-stop-live-stream-recordings-programmatically-imediastreamactionnotify3#comments module for ?
I have implement this code in Eclipse and successfully put jar in lib folder and configured everything. Now again I am not able to record different streams with their corresponding name. Means If I publish stream1 and stream2 then desired output should be two different files (in content folder) but again I see one single file being record ?
(3) Can I use ModuleLiveStreamRecord.java ? This was in older version of WOWZA but I have properly imported required jar and tested it.
My requirement is very simple:
As soon as users start publishing, WOWZA should start live recording. If 10 users publishing live, 10 files should be generate.
Don't make things more difficult than necessary (assuming you have Wowza 4.x; if you still have 3.x then I highly recommend to upgrade for free)
Open the Engine Manager (http://your.server.com:8088)
Go to "Applications" from the top menu
Select your application from the left menu (e.g. "live")
In the setup window for this application, click the blue Edit button
Enable "Record all incoming streams"
Click "Save"
Click the orange "Restart now" button at the top
Done
Every stream that is published via this application will now automatically be recorded. The default folder for recordings is the /content folder in your Wowza installation. You can change this on the same page under "Streaming File Directory" (make sure it's a directory on your local system, unless you really well understand how Wowza works)
The filename is always the streamname + ".mp4", but when you start a new recording while the file already exists, the old file will be renamed first.
Want to control recording manually? Start publishing first, then select "Incoming streams" from the left menu and use the big red dot button behind a stream name to start recording.
If your server produces any different behavior with regards to the file (re)naming or recording, then you may need to review your Wowza setup.
I appreciate your response KBoek.
I sorted out issue but there were really debugging need if one doing custom module. I had to write custom module for live auto recording because I wanted HTTP authentication and then custom name of live recording.
thanks again