Why doesn't Dask dashboard update when I run some code? - python-2.7

I'm trying to recreate the behaviour of the Dask dashboard as illustrated in this Youtube video https://www.youtube.com/watch?time_continue=1086&v=N_GqzcuGLCY. I can see my dashboard, but it doesn't update when I run a computation.
I'm running everything on my local machine (Kubuntu 18.04).
I used anaconda to set up my environment, including
python 2.7.14
dask 0.17.4
dask-core 0.17.4
bokeh 1.0.4
tornado 4.5.1
I set up my scheduler from the command line
dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://192.168.1.204:8786
distributed.scheduler - INFO - bokeh at: :8787
distributed.scheduler - INFO - Local Directory: /tmp/scheduler-bYQe2p
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Register tcp://127.0.0.1:35007
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35007
...and a worker too.
dask-worker localhost:8786
distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:36345'
distributed.worker - INFO - Start worker at: tcp://127.0.0.1:44033
distributed.worker - INFO - Listening to: tcp://127.0.0.1:44033
distributed.worker - INFO - bokeh at: 127.0.0.1:8789
distributed.worker - INFO - nanny at: 127.0.0.1:36345
distributed.worker - INFO - Waiting to connect to: tcp://localhost:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 4
distributed.worker - INFO - Memory: 16.70 GB
distributed.worker - INFO - Local Directory: /home/fergal/orbital/repos/projects/safegraph/dask/dask-worker-space/worker-QjJ1ke
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://localhost:8786
distributed.worker - INFO - -------------------------------------------------
Then my code, borrowed from the video, is
from dask.distributed import Client
import dask.array as da
client = Client(processes=False)
print(client)
x = da.random.random((10000, 10000, 10), chunks=(1000,1000,5))
y = da.random.random((10000, 10000, 10), chunks=(1000,1000,5))
z = (da.arcsin(x) + da.arcsin(y)).sum(axis=(1,2))
z.visualize('eg.svg')
z.compute()
The code runs, and produces a graph via graph-viz. The bokeh server is accessible at 127.0.0.1:8787/status, and displays a big blue block at the top right, as per the first few seconds of the video. But when I run my code, the webpage doesn't update to show a running computation, nor does it show any results when the computation is finished. Iwould expect to see something like what I see around time 1:20 on the video.
I'm undoubtedly neglecting to set something up properly, but I can't find any clues in either the documentation or on Stack Overflow. So what am I doing wrong?

I found a solution.
Update dask to 1.1.5, shutdown the dask-scheduler (and dask-worker). I'm guessing my problem was that the version of dask from the default conda channel was out of date. I downloaded the newer version from conda-forge

Related

Unable to start Marklogic service on AWS

I have connected to an AWS instance which was set up for MarkLogic using the AWS Systems Manager. I am trying to start the MarkLogic Server, but I am receiving the following error response:
Set configuration: JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.252.b09-2.amzn2.0.1.x86_64"
Set configuration: MARKLOGIC_MDB_TYPE=""
Set configuration: AWS_REGION="ap-southeast-2"
Set configuration: AWS_DEFAULT_REGION="ap-southeast-2"
Set configuration: MARKLOGIC_ZONE="ap-southeast-2a"
Initialize Configuration.
AWS Region: ap-southeast-2, ZONE: ap-southeast-2a. INSTANCE: i-08c0992c858711a67
Instance is not managed
Waiting for device mounted to come online : /dev/nvme1n1
Volume /dev/sdf has failed to attach - aborting
Warning: ec2-startup did not complete successfully
Check the error logs for details
Starting MarkLogic: [FAILED]
This was the output on the log for mlcmd:
"2020-08-17 02:10:26,821 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:26,952 131 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:27,102 281 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:27,103 282 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:27,297 476 TRACE [main] mlcmd.trace - init-config: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:27,299 478 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
"2020-08-17 02:10:27,299 478 INFO [main] builtin.log - runing init-config.xsh
"2020-08-17 02:10:27,942 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:28,042 100 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:28,173 231 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:28,174 232 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:28,387 445 TRACE [main] mlcmd.trace - ec2-startup: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:28,389 447 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
How do I resolve this issue?
If there is more required information, do let me know and I will try to get it
It appears that you are attempting to start a self-managed instance/cluster, while the Managed Cluster feature has not been disabled.
The MarkLogic Managed Cluster feature is the recommended way to deploy a MarkLogic Cluster on AWS so it is enabled by default. Managed clusters are meant to be deployed using the MarkLogic CloudFormation Templates.
Deploying MarkLogic on EC2 Using CloudFormation
The Managed Cluster feature reduces the amount of work necessary to setup the initial cluster, and creates an Auto Scaling Group that will automatically re-launch an instance that gets terminated, and when MarkLogic starts on the new instance, it will remount the associated EBS data drive.
CloudFormation Template Overview
If you wish to have a self-managed cluster, then you will need to create an /etc/marklogic.conf file to disable the feature at startup.
AWS Configuration Variables
Best Practice Editing MarkLogic Server Environment Variables
I would recommend reviewing the following guide, as it details using both the Managed Cluster feature, as well as self-managed clusters.
MarkLogic Server on Amazon Web Services (AWS) Guide

NLP Flask app startup nodes timing out on Google Kubernetes GKE

I have a flask app that includes some NLP packages and takes a while to initially build some vectors before it starts the server. I've noticed this in the past with Google App Engine and I was able to set a max timeout in the app.yaml file to fix this.
The problem is that when I start my cluster on Kubernetes with this app, I notice that the workers keep timing out in the logs. Which makes sense because I'm sure the default amount of time is not enough. However, I can't figure out how to configure GKE to allow the workers enough time to do everything it needs to do before it starts serving.
How do I increase the time the workers can take before they timeout?
I deleted the old instances so I can't get the logs right now, but I can start it up if someone wants to see the logs.
It's something like this:
I 2020-06-26T01:16:04.603060653Z Computing vectors for all products
E 2020-06-26T01:16:05.660331982Z
95it [00:05, 17.84it/s][2020-06-26 01:16:05 +0000] [220] [INFO] Booting worker with pid: 220
E 2020-06-26T01:16:31.198002748Z [nltk_data] Downloading package stopwords to /root/nltk_data...
E 2020-06-26T01:16:31.198056691Z [nltk_data] Package stopwords is already up-to-date!
100it 2020-06-26T01:16:35.696015992Z [CRITICAL] WORKER TIMEOUT (pid:220)
E 2020-06-26T01:16:35.696015992Z [2020-06-26 01:16:35 +0000] [220] [INFO] Worker exiting (pid: 220)
I also see this:
The node was low on resource: memory. Container thoughtful-sha256-1 was using 1035416Ki, which exceeds its request of 0.
Obviously I don't exactly know what I'm doing. Why does it say I'm requesting 0 memory and can I set a timeout amount for the Kubernetes nodes?
Thanks for the help!
One thing you can do is add some sort of delay in a startup script for your GCP instances. You could try a simple:
#!/bin/bash
sleep <time-in-seconds>
Another thing you can try is adding some sort of delay to when your containers start in your Kubernetes nodes. For example, a delay in an initContainer
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: myapa:latest
initContainers:
- name: init-myservice
image: busybox:1.28
command: ['sh', '-c', "echo Waiting a bit && sleep 3600"]
Furthermore, you can try a StartupProbe combined with the Probe parameter initialDelaySeconds on your actual application container that way it actually waits for some time before saying: I'm going to see if the application has started.:
startupProbe:
exec:
command:
- touch
- /tmp/started
initialDelaySeconds: 3600

while using openapi Braid for generating api for my corda project its showing some error in the last steep

Downloads\bootcamp-openapi-master\bootcamp-openapi-master\build\nodes>openapi-generator generate -i http://localhost:10200/swagger.json -g javascript -o ./code-gen --
api-package io.generated.api --model-package io.generated.model
[main] ERROR io.swagger.v3.parser.util.RemoteUrl - unable to read
java.net.SocketException: Unexpected end of file from server
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
I am trying to work on the corda openapi generation project but its showing me above error in the last step. The project I am trying to work is
https://blog.b9lab.com/cordacon-2019-highlights-braid-server-and-openapi-generator-for-corda-flows-api-s-d24179ccb27c
and similar
https://github.com/corda/openapi-sample
But the problem is that I have done same as show in the link but my last step that is generating api is not executing and throwing the above mentioned errors. And one more thing is that my one step is showing different output compared to the one shown in the Git.
MY OUTPUT
10:03:39.328 [main] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - Starting Braid on port: 10200
10:03:39.718 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter starting...
10:03:39.781 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.DocsHandlerFactory - activating OpenAPI V3
10:03:39.874 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - swagger json bound to https://localhost:10200/swagger.json
10:03:39.890 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - Swagger UI bound to https://localhost:10200/
10:03:42.999 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.GetAllTokensFlow
10:03:43.062 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.TokenIssueFlowInitiator
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - REST end point bound to https://localhost:10200/api/rest
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter complete -- 3406 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupWebserver complete -- 781 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid server started on
10:03:43.905 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid service mounted on https://localhost:10200/api/
10:03:43.905 [vert.x-eventloop-thread-1] INFO io.bluebank.braid.corda.BraidServer - Braid server started successfully on 10200
Their OUTPUT
10:03:39.328 [main] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - Starting Braid on port: 10200
10:03:39.718 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter starting...
10:03:39.781 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.DocsHandlerFactory - activating OpenAPI V2
10:03:39.874 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - swagger json bound to http://localhost:10200/swagger.json
10:03:39.890 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - Swagger UI bound to http://localhost:10200/
10:03:42.999 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.GetAllTokensFlow
10:03:43.062 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.TokenIssueFlowInitiator
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - REST end point bound to http://localhost:10200/api/rest
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter complete -- 3406 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupWebserver complete -- 781 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid server started on
10:03:43.905 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid service mounted on https://localhost:10200/api/
10:03:43.905 [vert.x-eventloop-thread-1] INFO io.bluebank.braid.corda.BraidServer - Braid server started successfully on 10200
The difference is that their don't contains https but I don't think this could raise any error.
In your Braid server output it says activating OpenAPI V2 meaning your server is generating the API's with OpenAPI Specification version 2, while your openapi-generator is throwing error ERROR io.swagger.v3.parser.util.RemoteUrl meaning it's looking for v3 (not v2) that's why it's unable to parse the file.
In my Medium article (and R3 tutorial) we both use v3.
From my article (notice the 3 after 10200:
localhost:10004 user1 test 10200 3 “/home/your-user/path-to-project/bootcamp-openapi/build/nodes/PartyA/cordapps”
From R3's article (notice how we both use openapi version 3):
Now, find the down triangle to open up the run configuration. At the Program arguments, paste in:
localhost:10004 user1 test 10200 3
"/YOUR-PATH-TO-THIS-FOLDER/bootcamp-cordapp/build/nodes/PartyA/cordapps"
RPC connection address: localhost:10004
node login username: user1
node login password: test
Your desired expose port: 10200
openapi version: 3
Cordapp folder to pick up the jar: "YOUR-PATH-TO-THIS-FOLDER/bootcamp-cordapp/build/nodes/PartyA/cordapps"
So I'm not sure why you decided to use version 2.

WSO2 API Manager 2.1 Analytics - Fails to Start up while connecting to Oracle DB

I am deploying WSO2 API Manager 2.1 and Analytics using the Pattern as specified - https://github.com/wso2/docker-apim/tree/master/docker-compose/pattern-3
Here in all the components - nginx, Publisher, Store, Traffic Manager, Gateway Worker, Gateway Manager, Key Manager and Analytics are deployed as separate docker containers.
When I started these containers, it worked fine and by default it was using the mysql server for storing all the data.
But as per our requirement, we had to use the Oracle DB and hence, we created a user in there with all the required permissions and then run the oracle scripts and finally started all the containers one by one.
In addition for Analytics, we created two separate users for two data sources - WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB (Didnt run any scripts for these).
And now we have a problem with the Analytics COntainer not able to start and throwing the error -
[2017-07-11 12:53:54,017] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Starting WSO2 Carbon...
[2017-07-11 12:53:54,017] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Operating System : Linux 4.8.0-53-generic, amd64
[2017-07-11 12:53:54,017] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Home : /mnt/jdk-7u80/jre
[2017-07-11 12:53:54,017] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Version : 1.7.0_80
[2017-07-11 12:53:54,017] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java VM : Java HotSpot(TM) 64-Bit Server VM 24.80-b11,Oracle Corporation
[2017-07-11 12:53:54,018] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Carbon Home : /mnt/186.12.12.12/wso2am-analytics-2.1.0
[2017-07-11 12:53:54,018] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Temp Dir : /mnt/186.12.12.12/wso2am-analytics-2.1.0/tmp
[2017-07-11 12:53:54,018] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - User : root, en-US, GMT
[2017-07-11 12:54:34,104] INFO {org.wso2.carbon.core.internal.permission.update.PermissionUpdater} - Permission cache updated for tenant -1234
[2017-07-11 12:54:34,235] INFO {org.wso2.carbon.core.transports.http.HttpsTransportListener} - HTTPS port : 9444
[2017-07-11 12:54:34,235] INFO {org.wso2.carbon.core.transports.http.HttpTransportListener} - HTTP port : 9764
[2017-07-11 12:54:36,547] INFO {org.wso2.carbon.webapp.mgt.TomcatGenericWebappsDeployer} - Deployed webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/analytics].File[/mnt/186.12.12.12/wso2am-analytics-2.1.0/repository/deployment/server/webapps/analytics.war]
[2017-07-11 12:54:36,593] INFO {org.wso2.carbon.webapp.mgt.TomcatGenericWebappsDeployer} - Deployed webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/inputwebsocket].File[/mnt/186.12.12.12/wso2am-analytics-2.1.0/repository/deployment/server/webapps/inputwebsocket.war]
[2017-07-11 12:54:36,622] INFO {org.wso2.carbon.webapp.mgt.TomcatGenericWebappsDeployer} - Deployed webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/outputwebsocket].File[/mnt/186.12.12.12/wso2am-analytics-2.1.0/repository/deployment/server/webapps/outputwebsocket.war]
[
[2017-07-11 12:54:43,116] INFO {org.wso2.carbon.event.processor.core.EventProcessorDeployer} - Execution plan deployment held back and in inactive state : APIMAnalytics-RequestSummarizer-RequestSummarizer-realtime1.siddhiql, Dependency validation exception: Stream org.wso2.apimgt.statistics.requestsPerMinPerKeyStream:1.0.0 does not exist
[2017-07-11 12:54:43,186] INFO {org.wso2.carbon.event.processor.core.EventProcessorDeployer} - Execution plan deployment held back and in inactive state : APIMAnalytics-UnusualIPAccessTemplate-UnusualIPAccessAlert-realtime1.siddhiql, Dependency validation exception: Stream org.wso2.apimgt.statistics.perMinuteRequest:1.0.0 does not exist
[2017-07-11 12:54:43,218] INFO {org.wso2.carbon.databridge.receiver.thrift.ThriftDataReceiver} - Thrift Server started at 0.0.0.0
[2017-07-11 12:54:43,246] INFO {org.wso2.carbon.databridge.receiver.thrift.ThriftDataReceiver} - Thrift SSL port : 7712
[2017-07-11 12:54:43,253] INFO {org.wso2.carbon.databridge.receiver.thrift.ThriftDataReceiver} - Thrift port : 7612
[2017-07-11 12:54:43,277] INFO {org.apache.tomcat.util.net.NioSelectorPool} - Using a shared selector for servlet write/read
[2017-07-11 12:54:43,355] INFO {org.apache.tomcat.util.net.NioSelectorPool} - Using a shared selector for servlet write/read
[2017-07-11 12:54:43,408] INFO {org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl} - Task service starting in STANDALONE mode...
[2017-07-11 12:54:44,030] ERROR {org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent} - Error in activating analytics data service: null
java.lang.RuntimeException
at org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore$RDBMSResultSetIterator.next(RDBMSAnalyticsRecordStore.java:881)
at org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore$RDBMSResultSetIterator.hasNext(RDBMSAnalyticsRecordStore.java:843)
at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:848)
at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
at org.wso2.carbon.analytics.datasource.core.util.GenericUtils.listRecords(GenericUtils.java:284)
[2017-07-11 12:54:55,566] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2017-07-11 12:55:05,564] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent for -1234
org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting loganalyzer:1.0.0 of event bundle with events 1
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:181)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId loganalyzer:1.0.0 present in cache
Can someone please let me know how to resolve this issue.
You can get the newer version of jar from http://maven.wso2.org/nexus/content/groups/public/org/wso2/carbon/analytics/org.wso2.carbon.analytics.datasource.rdbms/

Elastic Beanstalk CloudWatch Log streaming stops working – How to debug

My Elastic Beanstalk environment is stopping streaming node.js events to CloudWatch Logs. Streaming works fine for a view minutes on a new instance. After a view minutes no more logs show up in CloudWatch.
I set up AWS Elastic Beanstalk to stream logs to CloudWatch under Configuration > Software Configuration > CloudWatch Logs > Log Streaming (true). I deactivated log streaming and reactivated it as a test. Taking a look at cloudwatch
Last eb-activity log is about 10 minutes old
Error log is not available (on neither of the instances)
nginx/access.log is a view seconds old
nodejs.log is about an hour old (short after relaunching instance)
Every health check writes an log entry every view seconds into nodejs.log though.
I did not find any logs on the ec2 instance regarding log streaming.
Has anyone similar issues?
How do I make Elastic Beanstalk stream nodejs logs to CloudWatch logs.
--- EDIT
[ec2-user#ip-###-##-##-## log]$ cat /var/log/awslogs.log
2017-03-07 11:01:05,928 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Detected file rotation, notifying reader
2017-03-07 11:01:05,928 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Reader is still alive.
2017-03-07 11:01:05,928 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'.
2017-03-07 11:01:05,928 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'.
2017-03-07 11:01:06,052 - cwlogs.push.reader - INFO - 31861 - Thread-8 - No data is left. Reader is leaving.
2017-03-07 11:01:10,929 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Removing dead reader [2177a5cce5ed29525de329bfdc292ff1, /var/log/nginx/access.log]
2017-03-07 11:01:10,929 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Starting reader for [92257964a10edeb586f084f4f2ba35de, /var/log/nginx/access.log]
2017-03-07 11:01:10,930 - cwlogs.push.reader - INFO - 31861 - Thread-11 - Start reading file from 0.
2017-03-07 11:01:10,930 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'.
2017-03-07 11:01:10,930 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'.
2017-03-07 11:01:15,931 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'.
2017-03-07 11:01:15,931 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'.
2017-03-07 11:01:16,788 - cwlogs.push.publisher - INFO - 31861 - Thread-7 - Log group: /aws/elasticbeanstalk/production/var/log/nginx/access.log, log stream: i-0bd24767864801e2c, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1488884470930, 'start_position': 0L, 'end_position': 114L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1488884472931, 'start_position': 341L, 'end_position': 454L}, 'source_id': '92257964a10edeb586f084f4f2ba35de', 'num_of_events': 4, 'batch_size_in_bytes': 554}
2017-03-07 11:01:20,932 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'.
2017-03-07 11:01:20,932 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'.
2017-03-07 11:01:25,933 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'.
2017-03-07 11:01:25,933 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'.
2017-03-07 11:01:27,881 - cwlogs.push.publisher - INFO - 31861 - Thread-7 - Log group: /aws/elasticbeanstalk/production/var/log/nginx/access.log, log stream: i-0bd24767864801e2c, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1488884481933, 'start_position': 454L, 'end_position': 568L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1488884482934, 'start_position': 568L, 'end_position': 681L}, 'source_id': '92257964a10edeb586f084f4f2ba35de', 'num_of_events': 2, 'batch_size_in_bytes': 277}
When Andrew (#andrew-ferk) and myself activated log streaming, it created all the log groups and streams in CloudWatch with the current log. After we deployed again, we noticed the logs stopped. This is because aws hashes the first line of the log. If it has seen that hash before it will treat that file like it's already been processed
If you are using npm start the first lines will be your application's name with version.
You can add a CMD date && npm start to your dockerfile to trigger a different first line each time or run npm in silent mode (as long as your first output is unique).
Also according to their docs you should add some policy to your elastic-beanstalk before enabling the feature AWS-Docs
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:GetLogEvents",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutRetentionPolicy"
],
"Resource": [
"*"
]
}
]
}
The following FAQs might be helpful:
CloudWatch Logs Agent FAQs
Why can’t I push log data to CloudWatch Logs with the awslogs agent?
Some things to check if you are streaming custom log files:
eb ssh into the instance and look at /var/log/awslogs.log. If that doesn't even mention "Loading additional configs from (your awslogs config file)", make sure you are installing your config file correct as well as restarting the awslogs service after installing it (presumably using .ebextensions. See "Custom Log File Streaming" in Using Elastic Beanstalk with Amazon CloudWatch Logs. See the commands section in logs-streamtocloudwatch-linux.config for how to restart the awslogs service.
The CloudWatch Logs Agent is stateful. If the first few lines of your log file are blank or never change, you may need to set file_fingerprint_lines. See CloudWatch Logs Agent Reference.