Directly connecting to websocket using Streambuilder works seamlessly but I tried to make the stream part of the provider so that I can access the stream data in multiple widgets without encountering the "Bad State: Stream has already been listened to".
Is this a best way of handling multistreaming of data, if not what are my options?
Websocket server is part of Django
Code for provider is mentioned below
late final WebSocketChannel _fpdSockets;
Map _webSocketMessages = {};
Map get webSocketMessages {
return _webSocketMessages;
}
WebSocketStreamProvider()
: _fpdSockets = IOWebSocketChannel.connect(
Uri.parse('ws://127.0.0.1:8000/ws/socket-server/'),
);
Stream<Map<String, dynamic>> get dataStream => _fpdSockets.stream
.asBroadcastStream()
.map<Map<String, dynamic>>((value) => (jsonDecode(value)));
void sendDataToServer(dataToServer) {
print("Sending Data");
_fpdSockets.sink.add(
jsonEncode(dataToServer),
);
}
void closeConnection() {
_fpdSockets.sink.close();
}
handleMessages(data) {
print(data);
_webSocketMessages = data;
// notifyListeners();
}
}
Related
I am creating a service on top of a Ktor client. My payload is XML, and as such a simplified version of my client looks like this :
class MavenClient(private val client : HttpClient) {
private suspend fun getRemotePom(url : String) =
try{ MavenClientSuccess(client.get<POMProject>(url)) }catch (e: Exception) { MavenClientFailure(e)
}
companion object {
fun getDefaultClient(): HttpClient {
return HttpClient(Apache) {
install(JsonFeature) {
serializer = JacksonSerializer(jackson = kotlinXmlMapper)
accept(ContentType.Text.Xml)
accept(ContentType.Application.Xml)
accept(ContentType.Text.Plain)
}
}
}
}
}
Note the use of a custom XMLMapper, attached to a custom data class.
I want to test this class, and follow the documentation.
I end up with the following code for my test client :
private val mockClient = HttpClient(MockEngine) {
engine {
addHandler { request ->
when (request.url.fullUrl) {
"https://lengrand.me/minimal/1.2/minimal-1.2.pom" -> {
respond(minimalResourceStreamPom.readBytes()
, headers = headersOf("Content-Type" to listOf(ContentType.Application.Xml.toString())))
}
"https://lengrand.me/unknown/1.2/unknown-1.2.pom" -> {
respond("", HttpStatusCode.NotFound)
}
else -> error("Unhandled ${request.url.fullUrl}")
}
}
}
// TODO : How do I avoid repeating this again ? That's my implementation?!
install(JsonFeature) {
serializer = JacksonSerializer(jackson = PomParser.kotlinXmlMapper)
accept(ContentType.Text.Xml)
accept(ContentType.Application.Xml)
accept(ContentType.Text.Plain)
}
}
private val Url.hostWithPortIfRequired: String get() = if (port == protocol.defaultPort) host else hostWithPort
private val Url.fullUrl: String get() = "${protocol.name}://$hostWithPortIfRequired$fullPath"
private val mavenClient = MavenClient(mockClient)
Now, I am not worried about the Mapper itself, because I test it directly.
However what bothers me is that I essentially have to duplicate the complete logic of my client to test behaviour?
This seems very brittle, because for example it will cause my tests to fail and have to be updated if I move to Json tomorrow. Same if I start using Response Validation for example.
This is even more true for another client where I am using a defaultRequest, which I have to completely copy over as well:
private val mockClient = HttpClient(MockEngine) {
install(JsonFeature) {
serializer = JacksonSerializer(mapper)
accept(ContentType.Application.Json)
}
defaultRequest {
method = HttpMethod.Get
host = "api.github.com"
header("Accept", "application/vnd.github.v3+json")
if (GithubLogin().hasToken()) header("Authorization", GithubLogin().authToken)
}
Am I doing things wrong? Am I testing too much ? I am curious as to how I can improve this.
Thanks a lot for your input!
P.S : Unrelated but the page about testing on Ktor mentions adding the dependency to the implementation. Sounds like I should use testImplementation instead to avoid shipping the lib with my application ?
The MockEngine is designed for stubbing real HTTP client implementation to test objects that use it. The duplication problem, you encounter, lies in the fact that transforming response body responsibility belongs to the client. So I suggest either use Jackson directly to transform a response body (in this case you don't need to use JsonFeature) or extract common configuration in a extension function and call it for both engines.
I am processing the bulk json payload from s3.
code as follows:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.JsonParser;
import com.amazonaws.services.s3.model.S3Object;
import static com.fasterxml.jackson.core.JsonToken;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
public boolean sync(Job job)
throws IOException
//validating the json payload from s3.
try(InputStream s3Stream = readStreamFromS3())
{
validationService.validate(s3Stream);
}
catch (S3SdkInteractionException e) {
{
logger.error(e.getLocalizedMessage();
}
//process the json payload from s3.
try (InputStream s3Stream = readStreamFromS3())
{
syncService.process(s3Stream);
}
catch (S3SdkInteractionException e) {
{
logger.error(e.getLocalizedMessage();
}
}
public InputSteam readStreamFromS3()
{
return S3Object.getObjectContent();
}
// Process will sync the user data in the s3 stream.
// I am not closing the stream till the entire stream is processed. I
// need to handle as a stream processing.
// I dont want keep the contents in memory for processing, not
feasible for my use case.
public boolean process(InputStream s3Stream)
{
jsonFactory = objectMapper.getFactory();
try(JsonParser jsonParser = jsonFactory.createParser(s3Stream) {
JsonToken jsonToken = jsonParser.nextToken();
List<HttpResponseFuture<UserResponse> userFutures = new ArrayLsit<>(20);
while(true) {
for(int i = 0; i < 20; i++)
{
try {
// stream is processed fully
if (jsonToken == null || jsonToken == JSONTOKEN.END_OBJECT) { break; }
while (!jsonToken.isStructStart()) {
jsonToken = jsonParser.nextToken();
}
// Fetch the user record from the stream
if (jsonTokenn.isStructStart()) {
Map<String,Object> userNode = jsonParser.readValueAs(Map.class);
// calling an external service and adding future response
userFutures.add(executeAsync(httpClient, userNode);
//Move to the next user record
if (jsonToken == JSONTOKEN.START_OBJECT) {
jsonToken = jsonParser.nextToken();
}
}
}
catch (JsonParseException jpe) {
logger.error(jpe.getLocalizedMessage());
break;
}
}
for(ListenableFuture<UserResponse> responseFuture : Futures.inCompletionOrder(userFutures)) {
JsonResponse response = responseFuture.get();
}
}
}
return false;
}
There is serviceA through which we are ingesting data (json payload) to S3.
Another serviceB (the pseudocode shown above) will process the s3 data and call another serviceC to sync the data (json payload) in underlying store.
Problem:
I am seeing repeated s3 warning in our code. com.amazonaws.services.s3.internal.S3AbortableInputStream Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use
The validation phase is executing as expected without any issues.
However on syncing the data(ie. syncService.process()), the s3Stream is getting closed before the entire payload is processed.
Since the stream is getting the closed before i process the entire stream, i am in inconsistent state.
Dependency information as follows
aws-java-sdk-s3:1.11.411
guava:guava-25.0-jre
jackson-core:2.9.6
Json payload could vary between few MB's to 2 GB.
Any help would be appreciated.
Why is AWS SQS not a default connector for Apache Flink? Is there some technical limitation to doing this? Or was it just something that didn't get done? I want to implement this, any pointers would be appreciated
Probably too late for an answer to the original question... I wrote a SQS consumer as a SourceFunction, using the Java Messaging Service library for SQS:
SQSConsumer extends RichParallelSourceFunction<String> {
private volatile boolean isRunning;
private transient AmazonSQS sqs;
private transient SQSConnectionFactory connectionFactory;
private transient ExecutorService consumerExecutor;
#Override
public void open(Configuration parameters) throws Exception {
String region = ...
AWSCredentialsProvider credsProvider = ...
// may be use a blocking array backed thread pool to handle surges?
consumerExecutor = Executors.newCachedThreadPool();
ClientConfiguration clientConfig = PredefinedClientConfigurations.defaultConfig();
this.sqs = AmazonSQSAsyncClientBuilder.standard().withRegion(region).withCredentials(credsProvider)
.withClientConfiguration(clientConfig)
.withExecutorFactory(()->consumerExecutor).build();
this.connectionFactory = new SQSConnectionFactory(new ProviderConfiguration(), sqs);
this.isRunning = true;
}
#Override
public void run(SourceContext<String> ctx) throws Exception {
SQSConnection connection = connectionFactory.createConnection();
// ack each msg explicitly
Session session = connection.createSession(false, SQSSession.UNORDERED_ACKNOWLEDGE);
Queue queue = session.createQueue(<queueName>);
MessageConsumer msgConsumer = session.createConsumer(queue);
msgConsumer.setMessageListener(msg -> {
try {
String msgId = msg.getJMSMessageID();
String evt = ((TextMessage) msg).getText();
ctx.collect(evt);
msg.acknowledge();
} catch (JSMException e) {
// log and move on the next msg or bail with an exception
// have a dead letter queue is configured so this message is not lost
// msg is not acknowledged so it may be picked up again by another consumer instance
}
};
// check if we were canceled
if (!isRunning) {
return;
}
connection.start();
while (!consumerExecutor.awaitTermination(1, TimeUnit.MINUTES)) {
// keep waiting
}
}
#Override
public void cancel() {
isRunning = false;
// this method might be called before the task actually starts running
if (sqs != null) {
sqs.shutdown();
}
if(consumerExecutor != null) {
consumerExecutor.shutdown();
try {
consumerExecutor.awaitTermination(1, TimeUnit.MINUTES);
} catch (Exception e) {
//log e
}
}
}
#Override
public void close() throws Exception {
cancel();
super.close();
}
}
Note if you are using a standard SQS queue you may have to de-dup the messages depending on whether exactly-once guarantees are required.
Reference:
Working with JMS and Amazon SQS
At the moment, there is no connector for AWS SQS in Apache Flink. Have a look at the already existing connectors. I assume you already know about this, and would like to give some pointers. I was also looking for an SQS connector recently and found this mail thread.
Apache Kinesis Connector is somewhat similar to what you can implement on this. See whether you can get a start on this using this connector.
I have a method I need to refactor, as F.Promise has been deprecated in Play 2.5. It's pretty readable actually. It sends a request and authenticates via a custom security token and returns true if the response is 200.
public boolean verify(final String xSassToken){
WSRequest request = WS.url(mdVerifyXSassTokenURL)
.setHeader("X-SASS", xSassToken)
.setMethod("GET");
final F.Promise<WSResponse> responsePromise = request.execute();
try {
final WSResponse response = responsePromise.get(10000);
int status = response.getStatus();
if(status == 200 ) { //ok
return true;
}
} catch (Exception e) {
return false;
}
return false;
}
First thing I had to do was change this line:
final F.Promise<WSResponse> responsePromise = request.execute();
To this:
final CompletionStage<WSResponse> responsePromise = request.execute();
However, CompletionStage(T) doesn't have an equivalent get() method so I'm not sure the quickest and easiest way to get a WSResponse that I can verify the status of.
Yes, it does not. At least not directly.
What you are doing is "wrong" in the context of PlayFramework. get is a blocking call and you should avoid blocking as much as possible. That is why WS offers a non blocking API and a way to handle asynchronous results. So, first, you should probably rewrite your verify code to be async:
public CompletionStage<Boolean> verify(final String xSassToken) {
return WS.url(mdVerifyXSassTokenURL)
.setHeader("X-SASS", xSassToken)
.setMethod("GET")
.execute()
.thenApply(response -> response.getStatus() == Http.Status.OK);
}
Notice how I'm using thenApply to return a new a java.util.concurrent.CompletionStage instead of a plain boolean. That means that the code calling verify can also do the same. Per instance, an action at your controller can do something like this:
public class MyController extends Controller {
public CompletionStage<Result> action() {
return verify("whatever").thenApply(success -> {
if (success) return ok("successful request");
else return badRequest("xSassToken was not valid");
});
}
public CompletionStage<Boolean> verify(final String xSassToken) { ... }
}
This way your application will be able to handle a bigger workload without hanging.
Edit:
Since you have to maintain compatibility, this is what I would do to both evolve the design and also to keep code compatible while migrating:
/**
* #param xSassToken the token to be validated
* #return if the token is valid or not
*
* #deprecated Will be removed. Use {#link #verifyToken(String)} instead since it is non blocking.
*/
#Deprecated
public boolean verify(final String xSassToken) {
try {
return verifyToken(xSassToken).toCompletableFuture().get(10, TimeUnit.SECONDS);
} catch (Exception e) {
return false;
}
}
public CompletionStage<Boolean> verifyToken(final String xSassToken) {
return WS.url(mdVerifyXSassTokenURL)
.setHeader("X-SASS", xSassToken)
.setMethod("GET")
.execute()
.thenApply(response -> response.getStatus() == Http.Status.OK);
}
Basically, deprecate the old verify method and suggest users to migrate to new one.
I am logging RequestXML for a webservice client using SoapHandler as follows
public boolean handleMessage(SOAPMessageContext smc) {
logToSystemOut(smc);
return true;
}
private void logToSystemOut(SOAPMessageContext smc) {
Boolean outboundProperty = (Boolean)
smc.get (MessageContext.MESSAGE_OUTBOUND_PROPERTY);
if (outboundProperty.booleanValue()) {
out.println("\nOutbound message:");
} else {
out.println("\nInbound message:");
}
SOAPMessage message = smc.getMessage();
try {
message.writeTo(out);
out.println("");
} catch (Exception e) {
out.println("Exception in handler: " + e);
}
}
Got a new requirenment to add this xml to DB along with some extra values(which are not present in the xml). Is there any way I can pass few additional fields to above soap handler (in handleMessage method)?
Please note that changing the xml/WSDL or adding this to SOAP message header is not an option for me as it is owned by other interface. Any other solution?
Thanks!
You can cast your service class to a class of type "BindingProvider". In this form you can use it to assign it objects which you can access later from your SOAPHandler. Another useful usage is that you also can change the endPoint URL this way.
Before calling the service you do:
MySoapServicePortType service = new MySoapService().getMySoapServicePort();
BindingProvider bp = (BindingProvider)service;
MyTransferObject t = new MyTransferObject();
bp.getRequestContext().put("myTransferObject", t);
TypeResponse response = service.doRequest();
SOAPMessage message = t.getRequestMessage(message);
From your logging function you do:
private void logToSystemOut(SOAPMessageContext smc) {
...
MyTransferObject t = (MyTransferObject) messageContext.get("myTransferObject");
if (outboundProperty.booleanValue())
t.setRequestMessage(message);
else
t.setResponseMessage(message);
...
}