Using WSO2 DSS we'd like to create a scheduled task as a file based configuration (so we can use an automation tool to deploy the severs). Is it possible?
When I create a DSS scheduled task manually, I don't see any new file created.
Indeed there's a registry resource created:
/_system/governance/repository/components/org.wso2.carbon.tasks/definitions/-1234/DATA_SERVICE_TASK/api_agenda
-
<taskInfo>
<locationResolverClass>org.wso2.carbon.ntask.core.impl.RoundRobinTaskLocationResolver</locationResolverClass>
<locationResolverProperties/>
<name>api_agenda</name>
<properties>
<entry>
<key>DATA_SERVICE_TASK_NAME</key>
<value>api_agenda</value>
</entry>
<entry>
<key>DATA_TASK_CLASS_NAME</key>
<value>com.company.AgendaApiSynchronize</value>
</entry>
</properties>
<taskClass>org.wso2.carbon.dataservices.task.DSTaskExt</taskClass>
<triggerInfo>
<disallowConcurrentExecution>true</disallowConcurrentExecution>
<intervalMillis>14400000</intervalMillis>
<misfirePolicy>DEFAULT</misfirePolicy>
<repeatCount>-1</repeatCount>
</triggerInfo>
</taskInfo>
When I create the registry resource and restart the server, the task is scheduled. But - is there a file-based configuration variant?
Thank you for any hint
Related
We are currently in the process of setting up an Artifactory Pro instance on GCP and want to use GCS as its Filestore. The connection to the bucket is successful, uploads and downloads to and from the bucket via Artifactory are successful (using a generic repo).
However: Artifactory does not delete an artifact if we tell it so, via the GUI. The Artifact gets deleted and disappears in the GUI, (Trash Can is disabled in the System Settings) but continues to exist in the bucket in GCS.
This is our binarystore.xml:
<?xml version="1.0" encoding="UTF-8"?>
<config version="v1">
<chain>
<provider id="cache-fs" type="cache-fs">
<provider id="eventual" type="eventual">
<provider id="retry" type="retry">
<provider id="google-storage" type="google-storage"/>
</provider>
</provider>
</provider>
</chain>
<provider id="google-storage" type="google-storage">
<endpoint>commondatastorage.googleapis.com</endpoint>
<bucketName>rtfdev</bucketName>
<identity>xxx</identity>
<credential>xxx</credential>
<bucketExists>false</bucketExists>
<httpsOnly>true</httpsOnly>
<httpsPort>443</httpsPort>
</provider>
</config>
Our setup:
Artifactory 7.12.6
OS: Debian 10 (buster)
Machine Type: e2-highcpu-4 (4 vCPUs, 4 GB memory)
Disk: 200 GB SSD
The questions are:.
Is this working as intended? Does Artifactory never ever delete artifacts in a bucket?
On a related note: How can we convince Artifactory to be more verbose with its interactions with GCS? (the artifactory-binarystore.log is suspiciously empty, console.log is quiet as well...)
The reason you are not seeing the artifact being deleted immediately from the storage is the fact that Artifactory is using a checksum based storage.
TL;DR - you will see the artifact deleted from storage once the garbage collection process will delete it.
Artifactory stores any binary file only once. This is what we call "once and once only storage". First time a file is uploaded, Artifactory runs the required checksum calculations when storing the file, however, if the file is uploaded again (to a different location, for example), the upload is implemented as a simple database transaction that creates another record mapping the file's checksum to its new location. There is no need to actually store the file again in storage. No matter how many times a file is uploaded, the filestore only hosts a single copy of the file.
Deleting a file is also a simple database transaction in which the corresponding database record is deleted. The file itself is not directly deleted, even if the last database entry pointing to it is removed. So-called "orphaned" files are removed in the background by Artifactory's garbage collection processes.
I'm running a Spring Boot application within the Google Cloud Platform and viewing the log files viewing the Google Platform Logs Viewer. Before using Spring Boot and just using simple servlets, the logging entries would be displayed as:
Each request would be grouped and all the logging information for that request could be seen by expanding the row. However, when using Spring Boot the requests are no longer grouped and the log entries are just shown line by line. When there are multiple requests the log entries get very confusing as a result because it isn't possible to view them in a grouped way. I have my logging.properties setup in the same way:
.level = INFO
handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=FINEST
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.SimpleFormatter.format = [%1$tc] %4$s: %2$s - %5$s %6$s%n
The Logger is initialised in each class as:
private static final java.util.logging.Logger LOG = java.util.logging.Logger.getLogger(MyClass.class.getName());
And then the logging API is used as:
LOG.info("My Message");
I don't understand why the statements are being logged differently and no longer grouped but it must have something with the way Spring Boot handles logging?
Since recent runtimes, AppEngine is evolving with a behaviour that is more and more converging with a container based approach, more "opened" as new other products (like Cloud Run for example).
This is changing a little the way we're developing with GAE, specific legacy libraries aren't available (SearchAPI...), and it is changing also how logs are managed.
We can reproduce this "nested log feature" with new java11 runtime, but we need to manage it ourself.
As official docs mentioned:
In the Logs Viewer, log entries correlated by the same trace can be
viewed in a "parent-child" format.
It means, if we retrieve the trace identifier received inside X-Cloud-Trace-Context HTTP header of our request, we can then use it to add a new LogEntry by passing it as the trace identifier attribute.
This can be done by using Stackdriver Logging Client libraries
With Spring GCP
Fortunately, Spring Cloud GCP is there to make our lives easier.
You can find a sample project which implements it. Be careful, it's a AppEngine Flexible example, but it will work fine with Standard runtime.
It uses Logback.
From a working Spring Boot project on GAE Java11, steps to follow are :
Add spring-cloud-gcp-starter-logging dependency :
<!-- Starter for Stackriver Logging -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-logging</artifactId>
<version>1.2.1.RELEASE</version>
</dependency>
Add a logback-spring.xml inside src/main/resources folder :
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/cloud/gcp/autoconfigure/logging/logback-appender.xml" />
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<root level="INFO">
<!-- If running in GCP, remove the CONSOLE appender otherwise logs will be duplicated. -->
<appender-ref ref="CONSOLE"/>
<appender-ref ref="STACKDRIVER" />
</root>
</configuration>
Enable Spring GCP logging feature, inside src/main/resources/application.properties :
spring.cloud.gcp.logging.enabled=true
And use LOGGER inside your code:
#SpringBootApplication
#RestController
public class DemoApplication {
private static final Log LOGGER = LogFactory.getLog(DemoApplication.class);
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#GetMapping()
public SomeData get() {
LOGGER.info("My info message");
LOGGER.warn("My warning message");
LOGGER.error("My error message");
return new SomeData("Hello from Spring boot !");
}
}
Result will be in Stackdriver Logging viewer, for appengine.googleapis.com/request_log :
I want to access Google Cloud Storage via Hadoop client. I want to use it on machine outside of Google Cloud.
I followed instructions from here.
I created service account and generated key file. I also created core-site.xml file and downloaded the necessary library.
However, when I am trying to run simple hdfs dfs -ls gs://bucket-name command, all I get is this:
Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
When I am doing this inside the Google Cloud it works, but trying to connect to GCS from outside, it shows error above.
How to connect to GCS with Hadoop Client in this way? Is it even possible? I have no route to 169.254.169.254 address.
Here is my core-site.xml(I changed the key path and email in this example):
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>spark.hadoop.google.cloud.auth.service.account.enable</name>
<value>true</value>
</property>
<property>
<name>spark.hadoop.google.cloud.auth.service.account.json.keyfile</name>
<value>path/to/key.json</value>
</property>
<property>
<name>fs.gs.project.id</name>
<value>ringgit-research</value>
<description>
Optional. Google Cloud Project ID with access to GCS buckets.
Required only for list buckets and create bucket operations.
</description>
</property>
<property>
<name>fs.AbstractFileSystem.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
<description>The AbstractFileSystem for gs: uris.</description>
</property>
<property>
<name>fs.gs.auth.service.account.email</name>
<value>myserviceaccountaddress#google</value>
<description>
The email address is associated with the service account used for GCS
access when fs.gs.auth.service.account.enable is true. Required
when authentication key specified in the Configuration file (Method 1)
or a PKCS12 certificate (Method 3) is being used.
</description>
</property>
</configuration>
could be that the hadoop services haven’t taken the updates made in your core-site.xml file yet, so my suggestion is restart the hadoop’s services,another action that you can take is check the Access control options[1].
If You still having the same issue after having taken the action suggested, please post the complete error message.
[1]https://cloud.google.com/storage/docs/access-control/
The problem is with the fact that I've tried wrong authentication method. Used method assumes that it's running inside google cloud and it's trying to connect to google metadata servers. When running outside of google it doesn't work for obvious reasons.
The answer to this is here: Migrating 50TB data from local Hadoop cluster to Google Cloud Storage with the proper core-site.xml in the selected answer.
Property fs.gs.auth.service.account.keyfile should be used instead of spark.hadoop.google.cloud.auth.service.account.json.keyfile. The only difference is that this property needs p12 key file instead of json.
My tech req is the following :
Poll CSV Files
Read Data line by line
Transform data to desired format
convert to JSON/XML
Publish data thru REST/JMS
Deploy in WSO2 EI6.1.1
How is this possible in DS Tooling 3.8.0 of WSO2 ?
I know inbound endpoints,mediators,sequences proxy services etc can be used, but cant find a single document/article that helps in doing this.
Where do I start? How do I sequentially execute these steps? Artifacts are independently created, but dont how to automate them to an integration flow.
Appreciate if someone can shed some light.
Solution :
Create empty ESB solution project.
Create Proxy-Service.
Use Smooks-config for tranformation of CSV flat data to XML format
Create endpoint, for producing JMS messages to JMS queue of ActiveMQ.
Use datamapper mediator, if transformation is required
Use log mediator , for logging.
Use property mediator for setting endpoint related properties.
Config - axis2.xml,axis2Client.xml for enabling transport settings on E1611.
Export to CAR file, deploy on EI611 management console.
Happy Testing!!
Two questions on WSO2 BAM 2.5 Output Event Adaptor - 1) Why is there no "email" option in the output event adaptor type? As per the documentation, it should be there. Even if I create my own XML file for the Email event adaptor and drop it in the required folder, then type "email" is not recognized and the BAM is showing that as "inactive". 2) Which directory and file does the default logger output event adaptor write the logs to? I have configured that and I can see that the messages have got generated through Hive scripts and written to BAMNotifications column family but I am not able to see the logs in the repository/logs directory log files? Please help.
1) This issue occurred because of an OSGI loading issue in soap output adapter (It causes failure to some other output adapters). We have fixed that in next BAM version. For the moment to overcome this issue, please remove the soap output adapter jar (from plugins directory), restart and continue.
2) It needs to go to wso2carbon.log file. Can you please verify the log4j properties.