I am running my rails tests on 4Gb Digital ocean servers using gitlab runner. After I changed runner size to 3Gb and restarted coordinator it stopped to run the builds at all
# /etc/gitlab-runner/config.toml
[[runners]]
...
[runners.machine]
...
MachineOptions = [
"digitalocean-size=3gb" # before there was 4gb
...
]
No Digital ocean servers are created and jobs show the following information:
Running with gitlab-runner 11.5.1
on <Project name> build coordinator <ID>
...
Nothing happens and in the end job fails with timeout
Is it possible to reduce Digital ocean job runner size from 4Gb to 3Gb?
On January 16, 2018 Digital ocean introduced new droplet plans. Droplet slug names were changed accordingly. To use 3Gb servers to run your builds you must have the following line in your MachineOptions:
# /etc/gitlab-runner/config.toml
[[runners]]
...
[runners.machine]
...
MachineOptions = [
"digitalocean-size=s-1vcpu-3gb"
...
]
Related
I have installed informatica version 10.2,everything is running and working fine.
I have created two informatica powercenter repositories name: PRS_3, PRS_F2 from cliet side.
These repostories are present in domain as well in administrative console.
Snap:
I want to know where these two Repos are stored PRS_3, PRS_F2 in physical machine.
I checked in informatica installation in server side, dir but I didnot find these files.are repos stored in client side??
[ ~]$ cd informatica_installation/
[informatica_installation]$ ls
install.sh properties Server SilentInput_upgrade_NewConfig.properties silentinstall.sh upgrade_utils
logs sapsolutions SilentInput_DT.properties SilentInput_upgrade.properties source
Messages saptrans SilentInput.properties silentinstallDT.sh unjar_esd.sh
[informatica_installation]$ cd source/
[source]$ ls
connectors DataTransformation DiskSpaceInfo.properties externaljdbcjars isp java ODBC7.1 plugins server services thirdpartynotice tomcat tools
[ModelRepositoryService]$ ls
activation-1.1.jar lucene-queryparser-4.3.0.jar
avalon-framework-4.1.3.jar lucene-sandbox-4.3.0.jar
com.infa-com.infa.products.platform.modelutil.common-metamodel-10.2.0.9.490-SNAPSHOT.jar lucene-snowball-2.4.1.jar
com.infa-com.infa.products.repository.prs.deployer.isp.service-10.2.0.82.490-SNAPSHOT.jar lucene-spatial-4.3.0.jar
[ ModelRepositoryService]$ cd ..
[ services]$ ls
AdministratorConsole DataArchiveService HumanTaskService ISPPlugins PowerExchange SearchService WebAppApplicationService
AnalystService DataIntegrationService IDDService MetadataManagerService resourcemanager shared WebServiceHub
CatalogService DQContent InfaHadoopService ModelRepositoryService SAPBWService TDMService
ContentManagementService EmailService IntelligentDQService OAuthWebService SchedulerService Tutorial
OS: Linux for server side, Windows for client side.
Let me know the physical location of Repositories created by informatica powercenter Repositories services.
Informatica Power Center(10.x) uses a client-server architecture where metadata is stored in a database and executables exists in physical server(called as Node).
If you want to know where is PRS_3 repository service, go to Admin console > properties. You will see DB info.
Refer to below picture - in your case
Informatica services like Integration service and Repository service- runs in your Linux box where executables exists or wherever you installed infa.
Repo and Domain metadata - exists in a relational DB(Oracle/SQL Server etc.).
PC Clients - they can be installed in Windows machine. They talk to services and services talk to metadata base and completes a request.
Issue
So I have two AWS instances: a Puppet master and a Puppet client. When I run sudo puppet agent --test on my client, the tasks defined in my master's manifest didn't apply to the client instance.
Where I am right now
puppetmaster is installed on the master instance
puppet is installed on client instance
Master just finished signing my client's certificate. No errors were displayed
Master has a /etc/puppet/manifests/site.pp
Client's puppet.conf file has a server=dns_of_master line
My Puppet version is 5.4.0. I'm using the default manifest configuration.
Here's the guide that I'm following: https://www.digitalocean.com/community/tutorials/getting-started-with-puppet-code-manifests-and-modules. The only changes are the site.pp content and that I'm using AWS.
If it helps, here's my AWS instances' AMI: ami-06d51e91cea0dac8d
Details
Here's the content on my master's /etc/puppet/manifests/site.pp:
node default {
package { 'nginx':
ensure => installed
}
service { 'nginx':
ensure => running,
require => Package['nginx']
}
file { '/tmp/hello_world':
ensure => present,
content => 'Hello, World!'
}
}
The file has a permission of 777.
Here's the ouput when I run sudo puppet agent --test. This is after I ran sudo puppet agent --enable:
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for my_client_dns
Info: Applying configuration version '1578968015'
Notice: Applied catalog in 0.02 seconds
I have looked at other StackOverflow posts with this issue. I know that my catalog is not getting applied due to the lack of status messages and the quick time. Unfortunately, the solutions didn't apply to my case:
My site.pp is named correctly and in the correct file path /etc/puppet/manifests
I didn't touch my master's puppet.conf file
I tried restarting the server with sudo systemctl but nothing happens
So I have fixed the issue. The guide that I was following required an older version of Ubuntu (16.4, rather than 18.4 as I'm using). This needs a different AMI than the one that I used to create the instances.
i'm running through a simple and useless toy using PCF on azure, trying to create and run the stream 'time | log'
i successfully get SCDF started, and the stream created, but when i try to deploy the stream, SCDF creates two (cf) apps that won't run - they exist as far as cf-apps is concerned
○ → cf apps
Getting apps in org tess / space tess as admin...
OK
name requested state instances memory disk urls
yascdf-server started 1/1 2G 2G yascdf-server.apps.cf.tess.info
yascdf-server-LE7xs4r-tess-log stopped 0/1 512M 2G yascdf-server-LE7xs4r-tess-log.apps.cf.tess.info
yascdf-server-LE7xs4r-tess-time stopped 0/1 512M 2G yascdf-server-LE7xs4r-tess-time.apps.cf.tess.info
if i try to view the logs for either, nothing ever returns. but the logs in apps manager look like this:
2017-08-10T10:24:42.147-04:00 [API/0] [OUT] Created app with guid de8fee78-0902-4df7-a7ae-bba8a7710dca
2017-08-10T10:24:43.314-04:00 [API/0] [OUT] Updated app with guid de8fee78-0902-4df7-a7ae-bba8a7710dca ({"route"=>"97e1d26b-d950-479e-b9df-fe1f3b0c8a74", :verb=>"add", :relation=>"routes", :related_guid=>"97e1d26b-d950-479e-b9df-fe1f3b0c8a74"})
the routes don't work:
404 Not Found: Requested route ('yascdf-server-LE7xs4r-tess-log.apps.cf.tess.info') does not exist.
and trying to (re)start the route i get:
○ → cf start yascdf-server-LE7xs4r-tess-log
Starting app yascdf-server-LE7xs4r-tess-log in org tess / space tess as admin...
Staging app and tracing logs...
The app package is invalid: bits have not been uploaded
FAILED
here's the SCDF shell stuff i ran, if this helps:
server-unknown:>dataflow config server http://yascdf-server.apps.cf.tess.info/
Successfully targeted http://yascdf-server.apps.cf.cfpush.info/
dataflow:>app import --uri http://.../1-0-4-GA-stream-applications-rabbit-maven
Successfully registered applications: [<chop>]
dataflow:>stream create tess --definition "time | log"
Created new stream 'tess'
dataflow:>stream deploy tess
Deployment request has been sent for stream 'tess'
dataflow:>
anyone know what's going on here? i'd be grateful for a nudge...
Spring Cloud Data Flow: Server
1.2.3 (using built spring-cloud-dataflow-server-cloudfoundry-1.2.3.BUILD-SNAPSHOT.jar)
Spring Cloud Data Flow: Shell
1.2.3 (using downloaded spring-cloud-dataflow-shell-1.2.3.RELEASE.jar)
Deployment Environment
PCF v1.11.6 (on Azure)
pcf dev v0.26.0 (on mac)
App Starters
http://bit-dot-ly/1-0-4-GA-stream-applications-rabbit-maven
Logs
stream deploy log
It has been identified that java-buildpack 4.4 (JBP4) was used by OP and by running SCDF against this version, there is an issue with memory allocation in reactor-netty (used by JBP4 internally), which causes the out-of-memory error. The reactor team is addressing this issue in the upcoming 0.6.5 release. JBP4 will adapt to it eventually.
With all this said, SCDF is still not compatible with JPB4. It is recommended to downgrade to JPB 3.19 or latest in this release line instead.
I have the following docker containers that I have set up to test my web application:
Jenkins
Apache 1 (serving a laravel app)
Apache 2 (serving a legacy codeigniter app)
MySQL (accessed by both Apache 1 and Apache 2)
Selenium HUB
Selenium Node — ChromeDriver
The jenkins job runs a behat command on Apache 1 which in turn connects to Selenium Hub, which has a ChromeDriver node to actually hit the two apps: Apache 1 and Apache 2.
The whole system is running on an EC2 t2.small instance (1 core, 2GB RAM) with AWS linux.
The problem
The issue I am having is that if I run the pipeline multiple times, the first few times it runs just fine (the behat stage takes about 20s), but on the third and consecutive runs, the behat stage starts slowing down (taking 1m30s) and then failing after 3m or 10m or whenever I lose patience.
If I restart the docker containers, it works again, but only for another 2-4 runs.
Clues
Monitoring docker stats each time I run the jenkins pipeline, I noticed that the Block I/O, and specifically the 'I' was growing exponentially after the first few runs.
For example, after run 1
After run 2
After run 3
After run 4
The Block I/O for the chromedriver container is 21GB and the driver hangs. While I might expect the Block I/O to grow, I wouldn't expect it to grow exponentially as it seems to be doing. It's like something is... exploding.
The same docker configuration (using docker-compose) runs flawlessly every time on my personal MacBook Pro. Block I/O does not 'explode'. I constrain Docker to only use 1 core and 2GB of RAM.
What I've tried
This situation has sent me down the path of learning a lot more about docker, filesystems and memory management, but I'm still not resolving the issue. Some of the things I have tried:
Memory
I set mem_limit options on all containers and tuned them so that during any given run, the memory would not reach 100%. Memory usage now seems fairly stable, and never 'blows up'.
Storage Driver
The default for AWS Linux Docker is devicemapper in loop-lvm mode. After reading this doc
https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-docker-with-devicemapper
I switched to the suggested direct-lvm mode.
docker-compose restart
This does indeed 'reset' the issue, allowing me to get a few more runs in, but it doesn't last. After 2-4 runs, things seize up and the tests start failing.
iotop
Running iotop on the host shows that reads are going through the roof.
My Question...
What is happening that causes the block i/o to grow exponentially? I'm not clear if it's docker, jenkins, selenium or chromedriver that are causing the problem. My first guess is chromedriver, although the other containers are also showing signs of 'exploding'.
What is a good approach to tuning a system like this with multiple moving parts?
Additonal Info
My chromedriver container has the following environment set in docker-compose:
- SE_OPTS=-maxSession 6 -browser browserName=chrome,maxInstances=3
docker info:
$ docker info
Containers: 6
Running: 6
Paused: 0
Stopped: 0
Images: 5
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 4.862 GB
Data Space Total: 20.4 GB
Data Space Available: 15.53 GB
Metadata Space Used: 2.54 MB
Metadata Space Total: 213.9 MB
Metadata Space Available: 211.4 MB
Thin Pool Minimum Free Space: 2.039 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay null host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.4.51-40.60.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.03
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.956 GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
I am trying to write a Vagrantfile with multiple machines backed up by multiple providers. I specifically want to be able to spawn more than one of those machines in one go. Basically I want to run the command:
vagrant up vb_vm aws_vm
I am aware of the --provider flag, but this would apply to all machines being spawned, so not applicable in my case.
This is my (very trimmed down but still valid) Vagrantfile:
Vagrant.configure(2) do |config|
config.vm.define 'vb_vm' do |vb_vm|
vb_vm.vm.box='ubuntu/trusty64' # from hashicorp
vb_vm.vm.provider :virtualbox do |v|
end
end
config.vm.define 'aws_vm' do |aws_vm|
aws_vm.vm.box = "aws/dummy"
aws_vm.vm.box_url = 'https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box'
aws_vm.vm.provider :aws do |a, override|
a.access_key_id = 'something'
a.secret_access_key = 'something'
a.ami='something'
end
end
end
A vagrant box list shows that the boxes used for each definitions are of the right type:
aws/dummy (aws, 0)
ubuntu/trusty64 (virtualbox, 20150928.0.0)
But a vagrant status gives me (note that I do have the lxc plugin available, which became the default)
Current machine states:
aws_vm not created (aws)
vb_vm not created (lxc)
So this shows that spawning multiple machine with multiple provider is indeed possible, but the choice of provider is wrong.
I am aware of the tricks to set up the default provider, but this only makes things worse (virtualbox used everywhere, aws not used at all...)
I am aware of old stackoverflow questions as well, but they are related to a much older version of Vagrant.
So the question is: how do I make sure that each box defined uses its proper provider?
The trick will be to create the VM with their own provider.
example: I've defined a quick Vagrantfile (minimized) with boxes for each provider
Vagrant.configure(2) do |config|
config.vm.define "db" do |db|
db.vm.box = "..."
db.vm.hostname = "db"
end
config.vm.define "app", primary: true do |app|
app.vm.box = "..."
app.vm.hostname = "app"
app.ssh.forward_agent = true
app.ssh.forward_x11 = true
app.vm.provider "vmware_fusion" do |vm|
vm.vmx["memsize"] = "4096"
end
end
end
I create each VM separately
fhenri#machine:~/project/examples/vagrant/multimachine$ vagrant up db --provider=virtualbox
Bringing machine 'db' up with 'virtualbox' provider...
....
fhenri#machine:~/project/examples/vagrant/multimachine$ vagrant up app
Bringing machine 'app' up with 'vmware_fusion' provider...
....
then I halt everything and next time I do vagrant up
fhenri#machine:~/project/examples/vagrant/multimachine$ vagrant up
Bringing machine 'db' up with 'virtualbox' provider...
Bringing machine 'app' up with 'vmware_fusion' provider...
and status looks good
fhenri#machine:~/project/examples/vagrant/multimachine$ vagrant status
Current machine states:
db running (virtualbox)
app running (vmware_fusion)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.