Is there any way of viewing cache data in kubernetes cluster ? Does it get stored in a folder somewhere or can It be shown via a monitoring tool (Prometheus) ?
And how to clean the cache data in k8s cluster ?
There is cache directory in Kubernetes; however, it is held within the Master Node. As shown in this github link, the issue was found when debugging the kube-controller which is handled by the Master Node.
I am not familiar with Amazon's cloud service; however, within GKE, the Master node is a managed service and customers do not have access to it. So to answer your question, yes there is a cache directory in Kubernetes, but you cannot view it within GKE.
with this command you can see the buff/cahe memory for each pod
kubectl exec -it my-cluster-kafka-1 -- bash
and then just top
It would help if you could specify which kind of cache data you would like to clear, and what you would like to achieve.
For kubectl there is a default cache directory$HOME/.kube/cache 1, and $HOME/.kube/http-cache for the http cache 2.
There’s also a cache server like Memcached that is used for caching web content in k8s reference 3.
Related
I'd occasionally like to run a processing-heavy job on demand. Knative serving seemed like a good fit since it can scale from/to zero pods.
Is it possible to specify a node pool or node selector for a Knative service? Or is there some other way to ensure pods are created with specific machine types? I couldn't find anything in the docs.
I'm using GKE.
EDIT: Open feature request (please upvote!) https://issuetracker.google.com/issues/114402172
It isn't currently possible to specify a node pool/selector while using knative, nor a way to ensure pods are created with a predefined machine type on knative.
I encourage you to file this as a feature request.
This has moved to this Knative issue, as the question doesn't seem to be GKE specific.
I'm trying to create DCOS services that download artifacts(custom config files etc.) from hdfs. I was using simple ftp server before for it but I wanted to use hdfs. It is allowed to use "hdfs://" in artifact uri but it doesn't work correctly.
Artifact fetch ends with error because there's no "hadoop" command. Weird. I read that I need to provide own hadoop for it.
So I downloaded hadoop, set up necessary variables in /etc/profile. I can run "hadoop" without any problem when ssh'ing to node but service still ends with the same error.
It seems that environment variables configured in service are used after the artifact fetch because they don't work at all. Also, it looks like services completely ignore /etc/profile file.
So my question is: how do I set up everything so my service can fetch artifacts stored on hdfs?
The Mesos fetcher supports local Hadoop clients, please check your agent configuration and in particular your --hadoop_home setting.
I am really hoping to use Presto in an ETL pipeline on AWS EMR, but I am having trouble configuring it to fully utilize the cluster's resources. This cluster would exist solely for this one query, and nothing more, then die. Thus, I would like to claim the maximum available memory for each node and the one query by increasing query.max-memory-per-node and query.max-memory. I can do this when I'm configuring the cluster by adding these settings in the "Edit software settings" box of the cluster creation view in the AWS console. But the Presto server doesn't start, reporting in the server.log file an IllegalArgumentException, saying that max-memory-per-node exceeds the useable heap space (which, by default, is far too small for my instance type and use case).
I have tried to use the session setting set session resource_overcommit=true, but that only seems to override query.max-memory, not query.max-memory-per-node, because in the Presto UI, I see that very little of the available memory on each node is being used for the query.
Through Google, I've been led to believe that I need to also increase the JVM heap size by changing the -Xmx and -Xms properties in /etc/presto/conf/jvm.config, but it says here (http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto.html) that it is not possible to alter the JVM settings in the cluster creation phase.
To change these properties after the EMR cluster is active and the Presto server has been started, do I really have to manually ssh into each node and alter jvm.config and config.properties, and restart the Presto server? While I realize it'd be possible to manually install Presto with a custom configuration on an EMR cluster through a bootstrap script or something, this would really be a deal-breaker.
Is there something I'm missing here? Is there not an easier way to make Presto allocate all of a cluster to one query?
As advertised, increasing query.max-memory-per-node, and also by necessity the -Xmx property, indeed cannot be achieved on EMR until after Presto has already started with the default options. To increase these, the jvm.config and config.properties found in /etc/presto/conf/ have to be changed, and the Presto server restarted on each node (core and coordinator).
One can do this with a bootstrap script using commands like
sudo sed -i "s/query.max-memory-per-node=.*GB/query.max-memory-per-node=20GB/g" /etc/presto/conf/config.properties
sudo restart presto-server
and similarly for /etc/presto/jvm.conf. The only caveats are that one needs to include the logic in the bootstrap action to execute only after Presto has been installed, and that the server on the coordinating node needs to be restarted last (and possibly with different settings if the master node's instance type is different than the core nodes).
You might also need to change resources.reserved-system-memory from the default by specifying a value for it in config.properties. By default, this value is .4*(Xmx value), which is how much memory is claimed by Presto for the system pool. In my case, I was able to safely decrease this value and give more memory to each node for executing the query.
As a matter of fact, there are configuration classifications available for Presto in EMR. However, please note that these may vary depending on the EMR release version. For a complete list of the available configuration classifications per release version, please visit 1 (make sure to switch between the different tabs according to your desired release version). Specifically regarding to jvm.config properties, you will see in 2 that these are not currently configurable via configuration classifications. That being said, you can always edit the jvm.config file manually per your needs.
Amazon EMR 5.x Release Versions
1
Considerations with Presto on Amazon EMR - Some Presto Deployment Properties not Configurable:
2
I have an Amazon EC2 instance that I'd like to use as a development server for client projects as well as run JIRA. I have a domain pointed to the EC2 server IP. I'm new to docker so unsure if my approach is correct.
I'd like to have a JIRA container installed (with another jiradb MYSQL container) running at jira.domain.com as well as the potential to host client staging websites at client.domain.com which point to the client's docker containers.
I've been trying to use This JIRA docker image using the provided command
docker run --detach --publish 8080:8080 cptactionhank/atlassian-jira:latest
but the container always stops running mid setup (set up takes a while in-between steps). When I run the container again it goes back to the start of setup.
Once I have JIRA set up how would I run it under a subdomain? And how could I then have client.domain.com point to a separate docker container?
Thanks in advance!
As you probably know there's two considerations for getting Jira setup, whether as server or container:
You need to enter a license key early in the setup process (and it requires an Internet connection for verification), even if it's an evaluation
By default Jira will use its built-in (H2, IIRC) database, unless you configure an external one
So, in the case of 2) you probably want to make sure you have your external database ready and set up.
See Connecting Jira applications to external databases for preparatory steps for a variety of databases.
You didn't mention at what stage your first setup run fails, however once you've gotten past step 1) or any further successful setup, one of the first things I did, so as not to lose all work I'd done, was to commit the container!
docker commit -a 'My Name' -m 'Jira configured and set up' <container ID> myrepo/myjira:mytag
That way you don't lose all your previous work and you save your container into a new image in one fell swoop.
Is there a way to review content/files on a flapping app instance?
I had today this problem with one go application and unfortunately since container didnt start, I couldnt check what files are there. So the only way to debug problem (which was btw related to wrong filename) was the log stream.
Thank you,
Leszek
PS.
I am using HPE Stackato, but I assume the approach will be similar to the approach in CF and PCF...
With Pivotal Cloud Foundry, you can cf ssh to SSH into the container, or to set up port-forwarding so that you can use plain ssh or even scp and sftp to access the container or view its file system. You can read some more about it in:
The diego-ssh repository's README
The documentation on Accessing Apps with SSH
I highly doubt this functionality exists with HPE Stackato because it uses an older Cloud Foundry architecture. Both PCF and HPE are based off of open source Cloud Foundry, but PCF is on the newer Diego architecture, HPE is still on the DEA architecture according to the nodes listed in the Stackato Cluster Setup docs.
With the DEA architecture, you should be able to use the cf files command, which has the following usage:
NAME:
files - Print out a list of files in a directory or the contents of a specific file
USAGE:
cf files APP_NAME [PATH] [-i INSTANCE]
ALIAS:
f
OPTIONS:
-i Instance
To deal with a container that is failing to start, there is currently no out-of-the-box solution with Diego, but it has been discussed. This blog post discusses some options, including:
For the app in question explicitly specify a start command by adding a ";sleep 1d" The push command would like this - cf push <app_name> -c "<original_command> ;sleep 1d". This will keep the container around for a day after process within the container has exited.