We are building a web-app using Micronaut (v1.2.0) which will be deployed in a Kubernetes cluster (we are using Istio as the service-mesh).
We would like to instrument the critical method calls so that they can generate their own spans within a HTTP request span context. For this we are using the Micronaut OpenTracing support and Jaeger integration.
The following dependencies are included in the pom.xml
...
<dependency>
<groupId>io.micronaut</groupId>
<artifactId>micronaut-tracing</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-thrift</artifactId>
<scope>runtime</scope>
</dependency>
...
Have implemented Filter method with #ContinueSpan (also tried the same with #NewSpan) as shown below
#Filter("/**")
public class TraceTestFilter implements HttpServerFilter {
#Override
public Publisher<MutableHttpResponse<?>> doFilter(
HttpRequest<?> request, ServerFilterChain chain) {
return testMethodTracing(request, chain);
}
#ContinueSpan
public Publisher<MutableHttpResponse<?>> testMethodTracing(
HttpRequest<?> request, ServerFilterChain chain) {
// Details ommitted here
}
}
The following is maintained in the application-k8s.yml (also have an application.yml with the same settings)
---
tracing:
jaeger:
enabled: true
sampler:
probability: 1
sender:
agentHost: jaeger-agent.istio-system
agentPort: 5775
However we only see the trace entries that are generated by Istio (Envoy proxies) but we don't see the details of the method calls itself.
Any ideas as to what could be going wrong here?
Istio have this feature called Distributed Tracing, which enables users to track requests in mesh that is distributed across multiple services. This can be used to visualize request latency, serialization and parallelism.
For this to work Istio uses Envoy Proxy - Tracing feature.
You can deploy Bookinfo Application and see how Trace context propagation works.
If you have the same issue explained in this ticket, you need to wait for the next release of micronaut or use the workaround mentioned by micronaut guys there.
https://github.com/micronaut-projects/micronaut-core/issues/2209
Related
Kubernetes and Istio already installed in the cluster. Three micro services deployed as PODs. The flow is
Micro service A to Micro Service B calls => HTTP
Micro service B to Micro service C calls => via Kafka
Micro service A expose a HTTP API to outside
I guess when client hit the Ingres, Istio generate traceId and spanId in HTTP header and enter to Service A.
Are these spanId and traceId propagate to Micro service B and C without using separate API like Spring Cloud sleuth?
No, Istio does not provide tracing headers propagation. However it can be configured on application side without use of 3rd party APIs.
According to Istio documentation:
Istio leverages Envoy’s distributed tracing feature to provide tracing integration out of the box. Specifically, Istio provides options to install various tracing backend and configure proxies to send trace spans to them automatically. See Zipkin, Jaeger and LightStep task docs about how Istio works with those tracing systems.
Istio documentation also has an example of application side header propagation for the bookinfo demo application:
Trace context propagation
Although Istio proxies are able to automatically send spans, they need some hints to tie together the entire trace. Applications need to propagate the appropriate HTTP headers so that when the proxies send span information, the spans can be correlated correctly into a single trace.
To do this, an application needs to collect and propagate the following headers from the incoming request to any outgoing requests:
x-request-id
x-b3-traceid
x-b3-spanid
x-b3-parentspanid
x-b3-sampled
x-b3-flags
x-ot-span-context
Additionally, tracing integrations based on OpenCensus (e.g. Stackdriver) propagate the following headers:
x-cloud-trace-context
traceparent
grpc-trace-bin
If you look at the sample Python productpage service, for example, you see that the application extracts the required headers from an HTTP request using OpenTracing libraries:
def getForwardHeaders(request):
headers = {}
# x-b3-*** headers can be populated using the opentracing span
span = get_current_span()
carrier = {}
tracer.inject(
span_context=span.context,
format=Format.HTTP_HEADERS,
carrier=carrier)
headers.update(carrier)
# ...
incoming_headers = ['x-request-id']
# ...
for ihdr in incoming_headers:
val = request.headers.get(ihdr)
if val is not None:
headers[ihdr] = val
return headers
The reviews application (Java) does something similar:
#GET
#Path("/reviews/{productId}")
public Response bookReviewsById(#PathParam("productId") int productId,
#HeaderParam("end-user") String user,
#HeaderParam("x-request-id") String xreq,
#HeaderParam("x-b3-traceid") String xtraceid,
#HeaderParam("x-b3-spanid") String xspanid,
#HeaderParam("x-b3-parentspanid") String xparentspanid,
#HeaderParam("x-b3-sampled") String xsampled,
#HeaderParam("x-b3-flags") String xflags,
#HeaderParam("x-ot-span-context") String xotspan) {
if (ratings_enabled) {
JsonObject ratingsResponse = getRatings(Integer.toString(productId), user, xreq, xtraceid, xspanid, xparentspanid, xsampled, xflags, xotspan);
When you make downstream calls in your applications, make sure to include these headers.
I am using istio with version 1.3.5. Is there any configuration to be set to allow istio-proxy to log traceId? I am using jaeger tracing (wit zipkin protocol) being enabled. There is one thing I want to accomplish by having traceId logging:
- log correlation in multiple services upstream. Basically I can filter all logs by certain traceId.
According to envoy proxy documentation for envoy v1.12.0 used by istio 1.3:
Trace context propagation
Envoy provides the capability for reporting tracing information regarding communications between services in the mesh. However, to be able to correlate the pieces of tracing information generated by the various proxies within a call flow, the services must propagate certain trace context between the inbound and outbound requests.
Whichever tracing provider is being used, the service should propagate the x-request-id to enable logging across the invoked services to be correlated.
The tracing providers also require additional context, to enable the parent/child relationships between the spans (logical units of work) to be understood. This can be achieved by using the LightStep (via OpenTracing API) or Zipkin tracer directly within the service itself, to extract the trace context from the inbound request and inject it into any subsequent outbound requests. This approach would also enable the service to create additional spans, describing work being done internally within the service, that may be useful when examining the end-to-end trace.
Alternatively the trace context can be manually propagated by the service:
When using the LightStep tracer, Envoy relies on the service to propagate the
x-ot-span-context
HTTP header while sending HTTP requests to other services.
When using the Zipkin tracer, Envoy relies on the service to propagate the B3 HTTP headers (
x-b3-traceid,
x-b3-spanid,
x-b3-parentspanid,
x-b3-sampled,
and
x-b3-flags).
The
x-b3-sampled
header can also be supplied by an external client to either enable or
disable tracing for a particular request. In addition, the single
b3
header propagation format is supported, which is a more compressed
format.
When using the Datadog tracer, Envoy relies on the service to propagate the Datadog-specific HTTP headers (
x-datadog-trace-id,
x-datadog-parent-id,
x-datadog-sampling-priority).
TLDR: traceId headers need to be manually added to B3 HTTP headers.
Additional information: https://github.com/openzipkin/b3-propagation
Currently I'm using Redis that is provided by PCF. I'm connecting to it using JedisConnectionFactory from spring-data-redis providing needed configs like this:
#Configuration
public class RedisConfig {
#Bean
public JedisConnectionFactory jedisConnectionFactory() {
final JedisConnectionFactory jedisConFactory = new JedisConnectionFactory();
jedisConFactory.setHostName("pivotal-redis-host");
jedisConFactory.setPort(1234);
jedisConFactory.setPassword("mySecretPassword");
return jedisConFactory;
}
}
spring-cloud-config provides AbstractCloudConfig class that can be used to configure various connections. Is there any noticeable benefits one must use it instead of JedisConnectionFactory? Looks like less configs is needed to be provided, but is there any other reason?
public class RedisCloudConfig extends AbstractCloudConfig {
#Bean
public RedisConnectionFactory redisConnection() {
return connectionFactory().redisConnectionFactory();
}
}
Thanks in advance.
The main difference with Spring Cloud Connectors is that it's reading the service information from the Redis service that you bound to your application on Cloud Foundry. It then automatically configures the Redis connection based on that dynamically bound information.
Your example of using JedisConnectionFactory as well as #avhi's solution are placing the configuration information directly into either your source code or application configuration files. In this case, if your service changes then you'd need to reconfigure your app and run cf push again.
With Spring Cloud Connectors, you can change services by simply unbinding and binding a new Redis service through CF, and running cf restart.
In my opinion even you don't need to define #Bean configuration specifically.
You can simply use auto configuration by providing Redis server details in application.yml or application.properties simply.
spring:
redis:
host: pivotal-redis-host
port: 1234
password: mySecretPassword
By following the tutorial, I could bring up the spring-boot with Jetty running using the following dependencies.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
However, how could I configure the Jetty server such as:
Server threads (Queue thread pool)
Server connectors
Https configurations.
all those configuration available in Jetty...?
Is there an easy way to do in
application.yml?
Configuration class?
Any example would be greatly appreciated.
Many thanks!!
There are some general extension points for servlet containers and also options for plugging Jetty API calls into those, so I assume everything you would want is in reach. General advice can be found in the docs. Jetty hasn't received as much attention yet so there may not be the same options available for declarative configuration as with Tomcat, and for sure it won't have been used much yet. If you would like to help change that, then help is welcome.
Possibility to configure Jetty (in parts) programatically from http://howtodoinjava.com/spring/spring-boot/configure-jetty-server/
#Bean
public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory() {
JettyEmbeddedServletContainerFactory jettyContainer =
new JettyEmbeddedServletContainerFactory();
jettyContainer.setPort(9000);
jettyContainer.setContextPath("/home");
return jettyContainer;
}
If anyone is using Spring Boot - you can easily configure this in you application.properties thusly:
server.max-http-post-size=n
where n is the maximum size to which you wish to set this property. For example I use:
server.max-http-post-size=5000000
As of the year 2020, while working on newer versions, this is what you need to do, to configure Jetty port, context path and thread pool properties. I tested this on Spring Boot version 2.1.6 while the document I referred to is for version 2.3.3
Create a server factory bean in a configuration file.
#Bean
public ConfigurableServletWebServerFactory webServerFactory() {
JettyServletWebServerFactory factory = new JettyServletWebServerFactory();
factory.setPort(8080);
factory.setContextPath("/my-app");
QueuedThreadPool threadPool = new QueuedThreadPool();
threadPool.setMinThreads(10);
threadPool.setMaxThreads(100);
threadPool.setIdleTimeout(60000);
factory.setThreadPool(threadPool);
return factory;
}
Following is the link to Spring Docs:
customizing-embedded-containers
Spring Boot provides following Jetty specific configuration through property file:-
server:
jetty:
connection-idle-timeout: # Time that the connection can be idle before it is closed.
max-http-form-post-size: # Maximum size of the form content in any HTTP post request e.g. 200000B
accesslog:
enabled: # Enable access log e.g. true
append: # Enable append to log e.g. true
custom-format: # Custom log format
file-date-format: # Date format to place in log file name
filename: # Log file name, if not specified, logs redirect to "System.err"
format: # Log format e.g ncsa
ignore-paths: # Request paths that should not be logged
retention-period: # Number of days before rotated log files are deleted e.g. 31
threads:
acceptors: # Number of acceptor threads to use. When the value is -1, the default, the number of acceptors is derived from the operating environment.
selectors: # Number of selector threads to use. When the value is -1, the default, the number of selectors is derived from the operating environment.
min: # Minimum number of threads e.g. 8
max: # Maximum number of threads e.g. 200
max-queue-capacity: # Maximum capacity of the thread pool's backing queue. A default is computed based on the threading configuration.
idle-timeout: # Maximum thread idle time in millisecond e.g. 60000ms
Please refer official Spring Boot documentation for more configuration details.
Is there an embedded zookeeper so that we could use it in unit testing? It can be shipped with the test and run out of the box. Maybe we could mock some service and register to the embedded zookeeper
The Curator framework has TestingServer and TestingCluster classes (see https://github.com/Netflix/curator/wiki/Utilities) that are in a separate maven artifact (curator-test - see the Maven/Artifacts section of https://github.com/Netflix/curator/wiki).
They're pretty self explanatory, or you can download the curator code base and see how they're used internally in their own test cases.
We've used both successfully within unit tests at $DAY_JOB.
You could use Apache Curator Utilities provided in-process ZooKeeper server TestingServer that can be used for testing.
With maven you can dependency as follows
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-test</artifactId>
<version>3.2.1</version>
</dependency>
And you can create in process zookeeper server as folows
TestingServer zkServer;
#Before
public void setUp() throws Exception
{
zkServer = new TestingServer(2181, true);
}
#After
public void tearDown() throws Exception
{
zkServer.stop();
}
For testing Cluster use can use TestingCluster, which creates an internally running ensemble of ZooKeeper servers
You could use the zookeeper-maven-plugin, which is documented here.
The zookeeper project produces a "fat-jar" that it uses itself for system test.
There is a written up README, showing how easy it is to launch, but unfortunately it is not being made as an artifact, so cannot be linked to maven.