I am attempting to deploy a strongloop app to a Digitalocean remote box running Strongloop Process Manager. I have gotten as far as successfully running the deploy command as follows:
USER ~/projects/loopback/places-api $ slc deploy http://IPADDRESS deploy
Counting objects: 5215, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4781/4781), done.
Writing objects: 100% (5215/5215), 7.06 MiB | 4.27 MiB/s, done.
Total 5215 (delta 1130), reused 0 (delta 0)
To http://104.131.66.124:8701/api/services/1/deploy/default
* [new branch] deploy -> deploy
Deployed `deploy` as `placesAPI` to `http://IPADDRESS:8701/`
Next, I check the status of my Strongloop app by running the following command:
slc ctl -C http://IPADDRESS:8701
Service ID: 1
Service Name: placesAPI
Environment variables:
No environment variables defined
Instances:
Version Agent version Debugger version Cluster size Driver metadata
5.1.0 2.0.2 n/a 1 N/A
Processes:
ID PID WID Listening Ports Tracking objects? CPU profiling? Tracing? Debugging?
1.1.1050 1050 0
1.1.2065 2065 49
At this point, I am not able to access my app by visiting IPADDRESS:3001 as the Strongloop documentation would suggest and there are no processes listed in the above app status running on port 3001 as would be expected according to the Strongloop documentation.
Comparing my app status to the app status at this state of deployment shown in the Strongloop documentation, It appears I should have some processes listening to port 3001 which are not running in my app.
Here is the app status shown in the Strongloop documentation:
$ slc ctl -C http://prod.foo.com:7777
Service ID: 1
Service Name: appone
Environment variables:
No environment variables defined
Instances:
Version Agent version Cluster size
4.0.30 1.4.15 4
Processes:
ID PID WID Listening Ports Tracking objects? CPU profiling?
1.1.22555 22555 0
1.1.22741 22741 5 prod.foo.com:3001
1.1.22748 22748 6 prod.foo.com:3001
1.1.22773 22773 7 prod.foo.com:3001
1.1.22793 22793 8 prod.foo.com:3001
Notice the additional processes listening to port 3001.
My question is: how do I get my strongloop app to run and listen to these ports?
If it helps here are my package.json and config.json files:
:::::::::::::::::::::::package.json::::::::::::::::
{
"name": "placesAPI",
"version": "1.0.0",
"main": "server/server.js",
"scripts": {
"start": "node .",
"pretest": "jshint ."
},
"dependencies": {
"body-parser": "^1.9.0",
"compression": "^1.0.3",
"connect-ensure-login": "^0.1.1",
"cookie-parser": "^1.3.2",
"cors": "^2.5.2",
"errorhandler": "^1.1.1",
"express-flash": "0.0.2",
"express-session": "^1.7.6",
"jade": "^1.7.0",
"loopback": "^2.22.0",
"loopback-boot": "^2.6.5",
"loopback-component-explorer": "^2.1.0",
"loopback-component-passport": "^1.5.0",
"loopback-connector-postgresql": "^2.4.0",
"loopback-datasource-juggler": "^2.39.0",
"passport": "^0.3.2",
"passport-facebook": "^1.0.3",
"passport-google-oauth": "^0.2.0",
"passport-local": "^1.0.0",
"passport-oauth2": "^1.1.2",
"passport-twitter": "^1.0.3",
"serve-favicon": "^2.0.1"
},
"devDependencies": {
"jshint": "^2.5.6"
},
"repository": {
"type": "",
"url": ""
},
"description": "placesAPI",
"bundleDependencies": [
"body-parser",
"compression",
"connect-ensure-login",
"cookie-parser",
"cors",
"errorhandler",
"express-flash",
"express-session",
"jade",
"loopback",
"loopback-boot",
"loopback-component-explorer",
"loopback-component-passport",
"loopback-connector-postgresql",
"loopback-datasource-juggler",
"passport",
"passport-facebook",
"passport-oauth2",
"serve-favicon"
]
}
:::::::::::::::::::::::config.json::::::::::::::::
{
"restApiRoot": "/api",
"host": "0.0.0.0",
"port": 3000,
"cookieSecret": "REDACTED",
"remoting": {
"context": {
"enableHttpContext": false
},
"rest": {
"normalizeHttpPath": false,
"xml": false
},
"json": {
"strict": false,
"limit": "100kb"
},
"urlencoded": {
"extended": true,
"limit": "100kb"
},
"cors": false,
"errorHandler": {
"disableStackTrace": false
}
},
"legacyExplorer": false
}
There is also this error in the logs from log-dump:
2015-12-23T22:13:35.876Z pid:2720 worker:84 events.js:142
2015-12-23T22:13:35.882Z pid:2720 worker:84 throw er; // Unhandled 'error' event
2015-12-23T22:13:35.882Z pid:2720 worker:84 ^
2015-12-23T22:13:35.882Z pid:2720 worker:84 Error: connect ECONNREFUSED 127.0.0.1:5432
2015-12-23T22:13:35.882Z pid:2720 worker:84 at Object.exports._errnoException (util.js:856:11)
2015-12-23T22:13:35.882Z pid:2720 worker:84 at exports._exceptionWithHostPort (util.js:879:20)
2015-12-23T22:13:35.883Z pid:2720 worker:84 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1064:14)
2015-12-23T22:13:35.919Z pid:1106 worker:0 ERROR supervisor worker id 84 (pid 2720) accidental exit with 1
2015-12-23T22:13:38.253Z pid:1106 worker:0 INFO supervisor started worker 85 (pid 2738)
2015-12-23T22:13:38.253Z pid:1106 worker:0 INFO supervisor resized to 1
2015-12-23T22:13:39.858Z pid:2738 worker:85 INFO strong-agent native addon missing, install a compiler
2015-12-23T22:13:39.859Z pid:2738 worker:85 INFO strong-agent v2.0.2 profiling app 'placesAPI' pid '2738'
2015-12-23T22:13:39.890Z pid:2738 worker:85 INFO strong-agent[2738] started profiling agent
2015-12-23T22:13:44.943Z pid:2738 worker:85 INFO strong-agent not profiling, agent metrics requires a valid license.
2015-12-23T22:13:44.944Z pid:2738 worker:85 Please contact sales#strongloop.com for assistance.
2015-12-23T22:13:44.992Z pid:2738 worker:85 Web server listening at: http://0.0.0.0:3001
2015-12-23T22:13:44.997Z pid:2738 worker:85 Browse your REST API at http://0.0.0.0:3001/explorer
2015-12-23T22:13:45.103Z pid:2738 worker:85 Connection fails: { [Error: connect ECONNREFUSED 127.0.0.1:5432]
2015-12-23T22:13:45.104Z pid:2738 worker:85 code: 'ECONNREFUSED',
2015-12-23T22:13:45.104Z pid:2738 worker:85 errno: 'ECONNREFUSED',
2015-12-23T22:13:45.104Z pid:2738 worker:85 syscall: 'connect',
2015-12-23T22:13:45.104Z pid:2738 worker:85 address: '127.0.0.1',
2015-12-23T22:13:45.104Z pid:2738 worker:85 port: 5432 }
2015-12-23T22:13:45.104Z pid:2738 worker:85 It will be retried for the next request.
Just to put this in an answer, from the package.json we can see that you have included the loopback-connector-postgresql and in the log we see an attempted connection to port 5432 which is the default for that DBMS. It's trying to connect on the localhost (127.0.0.1) and my guess is that Postgres is either not installed on your Digital Ocean box, or not running. You'll need to update the config for your DB, or install (and run) the DB on your DO droplet.
If you have different configs for dev vs production then you can set up an environment-specific datasources config file: datasources.production.json for example. In that file you would put your prod config, and in datasources.json you would have your dev (local) config. When using this method, be sure to set the NODE_ENV variable on your DO droplet to production (to match the name of the prod datasources config file).
Related
I used the VS Code AWS Toolkit and SAM to create a basic SAM web app based on Go(I VS Code generate it).
It's some generic template code for the Lambda that reads IP from request and returns a Hello <IP> on the browser.
My end goal is to manage to see line by line debugging on my lambda function in VS Code.
There seem to be some default configurations in my launch.json:
{
"configurations": [
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "sunflowers:HelloWorldFunction (go1.x)",
"invokeTarget": {
"target": "template",
"templatePath": "${workspaceFolder}/sunflowers/template.yaml",
"logicalId": "HelloWorldFunction"
},
"lambda": {
"payload": {},
"environmentVariables": {},
"runtime": "go1.x"
}
},
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "API sunflowers:HelloWorldFunction (go1.x)",
"invokeTarget": {
"target": "api",
"templatePath": "${workspaceFolder}/sunflowers/template.yaml",
"logicalId": "HelloWorldFunction"
},
"api": {
"path": "/hello",
"httpMethod": "GET",
"payload": {
"json": {}
}
},
"lambda": {
"runtime": "go1.x"
}
}
]
}
However on clicking the "Start Debugging" button, I get that error that it can't install delve:
2022-08-03 21:20:46 [ERROR]: log level: info
2022-08-03 21:20:47 [INFO]: Retrieving AWS endpoint data
2022-08-03 21:20:47 [INFO]: OS: Darwin arm64 21.6.0
2022-08-03 21:20:47 [INFO]: Visual Studio Code extension host: 1.69.2
2022-08-03 21:20:47 [INFO]: AWS Toolkit: 1.46.0
2022-08-03 21:20:47 [INFO]: node: 16.13.2
2022-08-03 21:20:47 [INFO]: electron: 18.3.5
2022-08-03 21:21:49 [WARN]: SAM debug: missing AWS credentials (Toolkit is not connected)
2022-08-03 21:21:50 [INFO]: autoconnect: connected: 'profile:default'
2022-08-03 21:21:50 [INFO]: Command: (not started) [/Users/varungawande/Library/Application Support/Code/User/globalStorage/amazonwebservices.aws-toolkit-vscode/debuggers/delve/install-1.9.0.sh ]
2022-08-03 21:21:50 [ERROR]: Failed to cross-compile Delve debugger: Error: spawn Unknown system error -8
at ChildProcess.spawn (node:internal/child_process:412:11)
at Object.spawn (node:child_process:718:9)
at Function.l [as spawn] (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:27:9212)
at /Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:1734:1495
at new Promise (<anonymous>)
at cc.run (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:1734:1099)
at AH (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2076:252)
at _H (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2075:195)
at E6.invokeConfig (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:5965)
at E6.makeAndInvokeConfig (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:1314)
at E6.resolveDebugConfigurationWithSubstitutedVariables (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:1202) {
errno: -8,
code: 'Unknown system error -8',
syscall: 'spawn'
}
2022-08-03 21:22:27 [INFO]: Command: (not started) [/Users/varungawande/Library/Application Support/Code/User/globalStorage/amazonwebservices.aws-toolkit-vscode/debuggers/delve/install-1.9.0.sh ]
2022-08-03 21:22:27 [ERROR]: Failed to cross-compile Delve debugger: Error: spawn Unknown system error -8
at ChildProcess.spawn (node:internal/child_process:412:11)
at Object.spawn (node:child_process:718:9)
at Function.l [as spawn] (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:27:9212)
at /Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:1734:1495
at new Promise (<anonymous>)
at cc.run (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:1734:1099)
at AH (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2076:252)
at _H (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2075:195)
at E6.invokeConfig (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:5965)
at E6.makeAndInvokeConfig (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:1314)
at E6.resolveDebugConfigurationWithSubstitutedVariables (/Users/varungawande/.vscode/extensions/amazonwebservices.aws-toolkit-vscode-1.46.0/dist/src/main.js:2082:1202) {
errno: -8,
code: 'Unknown system error -8',
syscall: 'spawn'
}
2022-08-03 21:25:49 [INFO]: telemetry: sent batch (size=5)
My lambda requires the architecture amd64, while my Mac has the M1 chip, so it has an architecture of arm64. Is the above error due to this architecture mismatch?
I tried running the lambda locally using docker and that worked(using sam local start-api and sam local invoke), but am not sure the above with work as smoothly when using a debugger.
Turns out this is due to a bug in AWS Toolkit extension of VS Code. It was fixed in AWS Toolkit 1.47.
i am following this document to set up the distributed tracing : https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengistio-intro-topic.htm#exploring_istio_observability
My Cluster is on GKE GCP for testing purposes, installed istio top of it and followed document and setup services.
Services are up and running with Prometheus, Grafana, Jeger & Zipkin.
It's failing from step : Performing Distributed Tracing with OCI Application Performance Monitoring.
Tried udpating configmap for sidecar injector so that i can push tracing details to zipkin domain.
Configured Zipkin domain and using public-span use of now in configmap.
apiVersion: v1
kind: ConfigMap
metadata:
name: istio-custom-bootstrap-config
namespace: default
data:
custom_bootstrap.json: |
{
"tracing": {
"http": {
"name": "envoy.tracers.zipkin",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com", // [Replace this with data upload endpoint of your apm domain]
"collector_endpoint": "/20200101/observations/private-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=2C6YOLQSUZ5Q7IGN", // [Replace with the private datakey of your apm domain. You can also use public datakey but change the observation type to public-span]
"collectorEndpointVersion": "HTTP_JSON",
"trace_id_128bit": true,
"shared_span_context": false
}
}
},
"static_resources": {
"clusters": [{
"name": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com", // [Replace this with data upload endpoint of your apm domain:443]
"type": "STRICT_DNS",
"lb_policy": "ROUND_ROBIN",
"load_assignment": {
"cluster_name": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com", // [Replace this with data upload endpoint of your apm domain]
"endpoints": [{
"lb_endpoints": [{
"endpoint": {
"address": {
"socket_address": {
"address": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com", // [Replace this with data upload endpoint of your apm domain]
"port_value": 443
}
}
}
}]
}]
},
"transport_socket": {
"name": "envoy.transport_sockets.tls",
"typed_config": {
"#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
"sni": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com" // [Replace this with data upload endpoint of your apm domain]
}
}
}]
}
}
above configmap not working as expected, the sidecar is crashing due to missing key connection_timeout although after adding in configmap sidecar not showing error.
There is no error found in Zipkin or Istiod containers, not sure how to debug further.
Error log :
2022-03-30T05:59:33.146580Z info FLAG: --concurrency="2"
2022-03-30T05:59:33.146632Z info FLAG: --domain="default.svc.cluster.local"
2022-03-30T05:59:33.146642Z info FLAG: --help="false"
2022-03-30T05:59:33.146648Z info FLAG: --log_as_json="false"
2022-03-30T05:59:33.146672Z info FLAG: --log_caller=""
2022-03-30T05:59:33.146678Z info FLAG: --log_output_level="default:info"
2022-03-30T05:59:33.146682Z info FLAG: --log_rotate=""
2022-03-30T05:59:33.146687Z info FLAG: --log_rotate_max_age="30"
2022-03-30T05:59:33.146693Z info FLAG: --log_rotate_max_backups="1000"
2022-03-30T05:59:33.146699Z info FLAG: --log_rotate_max_size="104857600"
2022-03-30T05:59:33.146704Z info FLAG: --log_stacktrace_level="default:none"
2022-03-30T05:59:33.146715Z info FLAG: --log_target="[stdout]"
2022-03-30T05:59:33.146725Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2022-03-30T05:59:33.146730Z info FLAG: --outlierLogPath=""
2022-03-30T05:59:33.146736Z info FLAG: --proxyComponentLogLevel="misc:error"
2022-03-30T05:59:33.146741Z info FLAG: --proxyLogLevel="warning"
2022-03-30T05:59:33.146747Z info FLAG: --serviceCluster="reviews.default"
2022-03-30T05:59:33.146753Z info FLAG: --stsPort="0"
2022-03-30T05:59:33.146760Z info FLAG: --templateFile=""
2022-03-30T05:59:33.146767Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-03-30T05:59:33.146784Z info Version 1.8.0-c87a4c874df27e37a3e6c25fa3d1ef6279685d23-Clean
2022-03-30T05:59:33.146991Z info Obtained private IP [10.4.1.6]
2022-03-30T05:59:33.147107Z info Apply proxy config from env {"tracing":{"zipkin":{"address":"caadc76wvdp7edddddddccclii.apm-agt.ap-mumbai-1.oci.oraclecloud.com:443"}},"proxyMetadata":{"DNS_AGENT":""}}
2022-03-30T05:59:33.148650Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
DNS_AGENT: ""
serviceCluster: reviews.default
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: caadc76wvdp7edddddddccclii.apm-agt.ap-mumbai-1.oci.oraclecloud.com:443
2022-03-30T05:59:33.148721Z info Proxy role: &model.Proxy{RWMutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, Type:"sidecar", IPAddresses:[]string{"10.4.1.6"}, ID:"reviews-v1-5d6559df86-qbg6b.default", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"default.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), VerifiedIdentity:(*spiffe.Identity)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), WatchedResources:map[string]*model.WatchedResource(nil)}
2022-03-30T05:59:33.148732Z info JWT policy is third-party-jwt
2022-03-30T05:59:33.148777Z info PilotSAN []string{"istiod.istio-system.svc"}
2022-03-30T05:59:33.148827Z info sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012 Citadel
2022-03-30T05:59:33.148916Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2022-03-30T05:59:33.149082Z info citadelclient Citadel client using custom root: istiod.istio-system.svc:15012 -----BEGIN CERTIFICATE-----
MIIC/DCCAeSgAwIBAgIQOzOVPb98v+UHCpf80MI1pTANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIyMDMyOTE3MzIyOFoXDTMyMDMy
NjE3MzIyOFowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAO4j6Sa5VoFCUctY/ehMsFfXjejVHE05PzgaTt0x
zGK6WDLd4bQHVxiEERs2bQcPYP55T+AqBo4cyU5BFi7gEvrVdfHDMGdl4f3rhojB
RNdPLw9axyBNulOYBGIOIthpYY45fPLqvADQmU6GIUqcpg83zuwiyufbaCuElVuJ
h3eMebBQL6zsm+4BFZOTECvjMMpH/HSjOKdW/XsUU71FSVPo9q6devzLgCquZemO
kWHGjTtibwPcyRTZiL9FgBMnFF5gXe5K8FauIQlgkTDTWPj99n2FPGrfgEEC+z3q
O12NYi41zdY9RTk7f6kFHTzLRcGQ8ItG9MRebfZSfDqudCsCAwEAAaNCMEAwDgYD
VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFKAN5Ltn7oIN
l+9yoTfvOIvhBdTCMA0GCSqGSIb3DQEBCwUAA4IBAQCOKu1XEvJKXwRR/VNaL19L
iTIsC5csW4Dg1Z8aFQk+1UwroBbsdjCkPiwK0FJKHMoobIOtSbjn9k+OaUfv4pZo
D8dsDznqGJpkkiZ7zviwmpS3+B2YHoKFRs0ZXHu4hC081AUFjfFvFcwjtfPYKSGU
KqtxKPuvXCVGqaPdmkg5J4gG5q+Yutxno4m3VxGVocuHzXI9/Kox2Lz0C3royfF7
XoTxNy08TzkjDPuPCLqYy85zFOM7PzuuuK7ZIkdXpKbStIWLbjkciqLPzwi18JaH
eyS1/hORUC7AKMj8a3fKWrFsRiMu4Mdv+knnQ1ntLqb5Vy85VTvNFAvAB7mwD/NN
-----END CERTIFICATE-----
2022-03-30T05:59:33.219251Z info sds SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"
2022-03-30T05:59:33.219548Z info xdsproxy Initializing with upstream address istiod.istio-system.svc:15012 and cluster Kubernetes
2022-03-30T05:59:33.219346Z info sds Start SDS grpc server
2022-03-30T05:59:33.220303Z info xdsproxy adding watcher for certificate var/run/secrets/istio/root-cert.pem
2022-03-30T05:59:33.220894Z info Starting proxy agent
2022-03-30T05:59:33.222017Z info Opening status port 15020
2022-03-30T05:59:33.222278Z info Received new config, creating new Envoy epoch 0
2022-03-30T05:59:33.222328Z info Epoch 0 starting
2022-03-30T05:59:33.239683Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster reviews.default --service-node sidecar~10.4.1.6~reviews-v1-5d6559df86-qbg6b.default~default.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --config-yaml {
"tracing": {
"http": {
"name": "envoy.tracers.zipkin",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com",
"collector_endpoint": "/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=MAYH36IJELZRXTEETKL7QEA7NPA5UNEI",
"collectorEndpointVersion": "HTTP_JSON",
"trace_id_128bit": true,
"shared_span_context": false
}
}
},
"static_resources": {
"clusters": [{
"name": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com:443",
"connect_timeout": "5s",
"type": "STRICT_DNS",
"lb_policy": "ROUND_ROBIN",
"load_assignment": {
"cluster_name": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com",
"endpoints": [{
"lb_endpoints": [{
"endpoint": {
"address": {
"socket_address": {
"address": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com",
"port_value": 443
}
}
}
}]
}]
},
"transport_socket": {
"name": "envoy.transport_sockets.tls",
"typed_config": {
"#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
"sni": "aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com"
}
}
}]
}
}
--concurrency 2]
2022-03-30T05:59:33.315619Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2022-03-30T05:59:33.315693Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2022-03-30T05:59:33.316469Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2022-03-30T05:59:33.316542Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2022-03-30T05:59:33.390651Z info xdsproxy Envoy ADS stream established
2022-03-30T05:59:33.391110Z info xdsproxy connecting to upstream XDS server: istiod.istio-system.svc:15012
2022-03-30T05:59:33.396461Z warning envoy main there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2022-03-30T05:59:33.478768Z info sds resource:ROOTCA new connection
2022-03-30T05:59:33.479543Z info sds Skipping waiting for gateway secret
2022-03-30T05:59:33.479419Z info sds resource:default new connection
2022-03-30T05:59:33.479917Z info sds Skipping waiting for gateway secret
2022-03-30T05:59:33.682346Z info cache Root cert has changed, start rotating root cert for SDS clients
2022-03-30T05:59:33.682714Z info cache GenerateSecret default
2022-03-30T05:59:33.683386Z info sds resource:default pushed key/cert pair to proxy
2022-03-30T05:59:34.079948Z info cache Loaded root cert from certificate ROOTCA
2022-03-30T05:59:34.080300Z info sds resource:ROOTCA pushed root cert to proxy
2022-03-30T05:59:34.154971Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 10.8.14.87_14250: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_20001: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
10.8.1.76_3000: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_9411: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
10.8.1.191_15021: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_9080: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
10.8.5.43_443: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_15010: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_15014: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
10.8.14.87_14268: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_80: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
0.0.0.0_9090: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
virtualInbound: envoy.tracers.zipkin: unknown cluster 'aaaabbbb.apm-agt.us-ashburn-1.oci.oraclecloud.com'
2022-03-30T05:59:35.844720Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 reje
After 2-3 days of debugging was able to resolve distributed tracing issue with istio, Zipkin and OCI APM.
Note : With root user it was not working, so I created one compartment in OCI created IAM policy, group and give full access of compartment to the group.
Added root user to group and weirdly it started working while with direct root user and default policy it was not working.
Ref doc for policy : https://docs-uat.us.oracle.com/en/cloud/paas/application-performance-monitoring/apmgn/perform-oracle-cloud-infrastructure-prerequisite-tasks.html
Working configmap sidecar
connect_timeout key is required otherwise sidecar is failing and due to that PODs won't come in Ready state. Port 443 mentioned in the official documentation is not required.
apiVersion: v1
kind: ConfigMap
metadata:
name: istio-custom-bootstrap-config
data:
custom_bootstrap.json: |
{
"tracing": {
"http": {
"name": "envoy.tracers.zipkin",
"typed_config": {
"#type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
"collector_cluster": "aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com",
"collector_endpoint": "/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=M7SOSHXXXXXXXXXXXXXXXXXXXUZEHOGRSA",
"collector_endpoint_version": "HTTP_JSON",
"trace_id_128bit": true,
"shared_span_context": false
}
}
},
"static_resources": {
"clusters": [{
"name": "aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com",
"type": "STRICT_DNS",
"connect_timeout": "5s",
"lb_policy": "ROUND_ROBIN",
"load_assignment": {
"cluster_name": "aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com",
"endpoints": [{
"lb_endpoints": [{
"endpoint": {
"address": {
"socket_address": {
"address": "aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com",
"port_value": 443
}
}
}
}]
}]
},
"transport_socket": {
"name": "envoy.transport_sockets.tls",
"typed_config": {
"#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
"sni": "aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com"
}
}
}]
}
}
Istio config
sampling: 100 will push mostly all traces to Zipkin and OCI APM domain. Also i enabled enableTracing: true
Read more at : https://istio.io/latest/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: aacytncaaaaaaaal2a.apm-agt.ap-mumbai-1.oci.oraclecloud.com:443
enablePrometheusMerge: true
rootNamespace: istio-system
outboundTrafficPolicy:
mode: ALLOW_ANY
trustDomain: cluster.local
meshNetworks: 'networks: {}'
OCI console
I am trying to launch Loopback on Digital Ocean following this tutorial:
https://loopback.io/doc/en/lb4/deploying-with-pm2-and-nginx.html
The problem is that when I launch it with "pm2 start" command, it seems that pm2 starts, but it doesn't start the Loopback app. The logs don't show anything (screenshot - https://i.stack.imgur.com/MvaV2.png).
I double checked - the Loopback app is fine and it starts with "npm run start:local" successfully.
Here's my package.json commands:
"start:local": "node -r source-map-support/register .",
"start": "pm2 start ecosystem.config.js --env production",
"stop": "pm2 stop ecosystem.config.js --env production",
ecosystem.config.js:
module.exports = {
apps: [
{
name: 'BFF',
script: './dist/index.js',
instances: 1,
interpreter : 'node#10.20.1',
autorestart: true,
watch: true,
max_memory_restart: '1G',
env: {
NODE_ENV: 'development',
},
env_production: {
NODE_ENV: 'production',
},
},
],
};
Here is the mup.js:
module.exports = {
servers: {
one: {
host: '52.41.186.122',
username: 'ubuntu',
pem: 'aws-key/xanthelabs.pem'
// password:
// or leave blank for authenticate from ssh-agent
}
},
meteor: {
name: 'deep-app',
path: '/home/cortana/Desktop/deep-app',
servers: {
one: {}
},
buildOptions: {
serverOnly: true,
},
env: {
ROOT_URL: 'http://52.41.186.122:8888',
MONGO_URL: 'mongodb://localhost/meteor'
},
//dockerImage: 'kadirahq/meteord'
deployCheckWaitTime: 60
},
mongo: {
oplog: true,
port: 27017,
servers: {
one: {},
},
},
};
The error is:
Started TaskList: Start Meteor
[52.41.186.122] - Start Meteor
[52.41.186.122] - Start Meteor: SUCCESS
[52.41.186.122] - Verifying Deployment
[52.41.186.122] x Verifying Deployment: FAILED
-----------------------------------STDERR-----------------------------------
npm WARN deprecated semver behavior.
npm WARN package.json meteor-dev-bundle#0.0.0 No description
npm WARN package.json meteor-dev-bundle#0.0.0 No repository field.
npm WARN package.json meteor-dev-bundle#0.0.0 No README data
npm WARN cannot run in wd meteor-dev-bundle#0.0.0 node npm-rebuild.js (wd=/bundle/bundle/programs/server)
=> Starting meteor app on port:80
assert.js:93
throw new assert.AssertionError({
^
AssertionError: "undefined" === "function"
at wrapPathFunction (/bundle/bundle/programs/server/mini-files.js:77:10)
at Object.<anonymous> (/bundle/bundle/programs/server/mini-files.js:108:24)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/bundle/bundle/programs/server/boot.js:9:13)
at Module._compile (module.js:456:26)
-----------------------------------STDOUT-----------------------------------
To see more logs type 'mup logs --tail=50'
----------------------------------------------------------------------------
(node:8968) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error:
-----------------------------------STDERR-----------------------------------
npm WARN deprecated semver behavior.
npm WARN package.json meteor-dev-bundle#0.0.0 No description
npm WARN package.json meteor-dev-bundle#0.0.0 No repository field.
npm WARN package.json meteor-dev-bundle#0.0.0 No README data
npm WARN cannot run in wd meteor-dev-bundle#0.0.0 node npm-rebuild.js (wd=/bundle/bundle/programs/server)
=> Starting meteor app on port:80
assert.js:93
throw new assert.AssertionError({
^
AssertionError: "undefined" === "function"
at wrapPathFunction (/bundle/bundle/programs/server/mini-files.js:77:10)
at Object.<anonymous> (/bundle/bundle/programs/server/mini-files.js:108:24)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/bundle/bundle/programs/server/boot.js:9:13)
at Module._compile (module.js:456:26)
-----------------------------------STDOUT-----------------------------------
To see more logs type 'mup logs --tail=50'
----------------------------------------------------------------------------
Everything has been successfully setup. I dont understand what is throwing the error. Please explain the error and its reasons along with the solution. Thanks
About the system:
OS: ubuntu 16.04
deployment OS: Ubuntu 14.04 (on aws)
meteor : v1.4.1.2
Edit:
On using meteor-up fork suggested, I am getting this error:
[52.41.186.122] - Installing Node.js
[52.41.186.122] - Installing Node.js: SUCCESS
[52.41.186.122] - Installing PhantomJS
[52.41.186.122] - Installing PhantomJS: SUCCESS
[52.41.186.122] - Setting up Environment
[52.41.186.122] - Setting up Environment: SUCCESS
[52.41.186.122] - Copying MongoDB configuration
[52.41.186.122] x Copying MongoDB configuration: FAILED
Received exit code 0 while establishing SFTP session
The mup.js file I have is:
{
// Server authentication info
"servers": [
{
"host": "52.41.186.122",
"username": "ubuntu",
// "password": "ubuntu"
// or pem file (ssh based authentication)
"pem": "aws-key/xanthelabs.pem"
}
],
// Install MongoDB in the server, does not destroy local MongoDB on future setup
"setupMongo": true,
// WARNING: Node.js is required! Only skip if you already have Node.js installed on server.
"setupNode": true,
// WARNING: If nodeVersion omitted will setup 0.10.43 by default. Do not use v, only version number.
"nodeVersion": "4.4.7",
// Install PhantomJS in the server
"setupPhantom": true,
// Show a progress bar during the upload of the bundle to the server.
// Might cause an error in some rare cases if set to true, for instance in Shippable CI
"enableUploadProgressBar": true,
// Application name (No spaces)
"appName": "deep-app",
// Location of app (local directory)
"app": "/home/cortana/Desktop/deep-app",
// Configure environment
"env": {
"PORT": 8888,
"ROOT_URL": "http://52.41.186.122",
"MONGO_URL": "mongodb://localhost/meteor"
},
// Meteor Up checks if the app comes online just after the deployment
// before mup checks that, it will wait for no. of seconds configured below
"deployCheckWaitTime": 15
}
what is wrong here?
i had the same issues. add dockerImage: "abernix/meteord:base", to you meteor part of MUP.json.
If it's still not working. replace Flow Router with React Router
I'm trying to push a very simple rails app on a DigitalOcean droplet. Unfortunatly, i'm enable to continue the deployment : i get stuck with a very elusive error message :
info: [agent] get agent status
info: [agent] agent is running: true
info: [agent] get agent status
info: [agent] agent is running: true
info: [agent] get agent status
info: [agent] agent is running: true
info: [agent] get agent status
info: [agent] agent is running: true
info: Connecting to http://192.168.50.4:2375
PLAY [all] *********************************************************************
TASK [setup] *******************************************************************
fatal: [default]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "/bin/sh: 1: /usr/bin/python: not found\r\n", "msg": "MODULE FAILURE", "parsed": false}
NO MORE HOSTS LEFT *************************************************************
to retry, use: --limit #playbooks/setup.retry
PLAY RECAP *********************************************************************
default : ok=0 changed=0 unreachable=0 failed=1
Am i the only one who ever encounter this problem ?
Here is my Azkfile too :
/**
* Documentation: http://docs.azk.io/Azkfile.js
*/
// Adds the systems that shape your system
systems({
'apptelier-website': {
// Dependent systems
depends: [],
// More images: http://images.azk.io
image: {docker: 'azukiapp/ruby:2.3.0'},
// Steps to execute before running instances
provision: [
"bundle install --path /azk/bundler"
],
workdir: "/azk/#{manifest.dir}",
shell: "/bin/bash",
command: ["bundle", "exec", "rackup", "config.ru", "--pid", "/tmp/ruby.pid", "--port", "$HTTP_PORT", "--host", "0.0.0.0"],
wait: 20,
mounts: {
'/azk/#{manifest.dir}': sync("."),
'/azk/bundler': persistent("./bundler"),
'/azk/#{manifest.dir}/tmp': persistent("./tmp"),
'/azk/#{manifest.dir}/log': path("./log"),
'/azk/#{manifest.dir}/.bundle': path("./.bundle")
},
scalable: {"default": 1},
http: {
domains: [
'#{env.HOST_DOMAIN}', // used if deployed
'#{env.HOST_IP}', // used if deployed
'#{system.name}.#{azk.default_domain}' // default azk domain
]
},
ports: {
// exports global variables
http: "3000/tcp"
},
envs: {
// Make sure that the PORT value is the same as the one
// in ports/http below, and that it's also the same
// if you're setting it in a .env file
RUBY_ENV: "production",
RAILS_ENV: "production",
RACK_ENV: 'production',
WORKER_RETRY: 1,
BUNDLE_APP_CONFIG: '/azk/bundler',
APP_URL: '#{system.name}.#{azk.default_domain}'
}
},
deploy: {
image: {docker: 'azukiapp/deploy-digitalocean'},
mounts: {
'/azk/deploy/src': path('.'),
'/azk/deploy/.ssh': path('#{env.HOME}/.ssh'), // Required to connect with the remote server
'/azk/deploy/.config': persistent('deploy-config')
},
// This is not a server. Just run it with `azk deploy`
scalable: {default: 0, limit: 0},
envs: {
GIT_REF: 'master',
AZK_RESTART_COMMAND: 'azk restart -Rvv',
BOX_SIZE: '512mb'
}
}
});
Thanks for the help !
Edouard, nice to meet you. I'm from azk core team and I'm not sure if you're using the latest deployment image.
Please follow these steps to update it:
adocker pull azukiapp/deploy-digitalocean:0.0.7;
Edit the deploy system in your Azkfile and add the tag 0.0.7 to the used deployment image to ensure we're using the latest one. It should be like: image: {docker: 'azukiapp/deploy-digitalocean:0.0.7'};
Next, be sure you have the env DEPLOY_API_TOKEN set in your .env file. If you don't have it set yet, take a look on the Step 7 of the article we've published on DigitalOcean Community Tutorials: https://www.digitalocean.com/community/tutorials/how-to-deploy-a-rails-app-with-azk#step-7-%E2%80%94-obtaining-a-digitalocean-api-token
Finally, re-run the deploy command:
azk deploy clear-cache;
azk deploy
Please let me know if this is enough to solve your problem.