I want to upload with a Django ImageField to a nfs storage but I get this error:
[Errno 37] No locks available
This is in /etc/fstab/:
173.203.221.112:/home/user/project/media/uploads/ /home/user/project/media/uploads nfs rw,bg,hard,lock,intr,tcp,vers=3,wsize=8192,rsize=8192 0 0
I also tried to patch django to use flock() instead of lockf() but still not working.
http://code.djangoproject.com/ticket/9400
Any idea whats wrong?
I have this messy issue once, and after losing a lot of time looking for an answer I found this solution: rpc.statd
I have to execute that command in both sides of the NFS folders, in my case was my Computer and a Virtual Machine
Some information about this command can be found here:
Linux command: rpc.statd - NSM status monitor
In case that is not enough, some times I faced this issue I have to execute the statd service manually because it wasn't running. This other way to fix the problem is execute in both sides of the NFS the command:
service statd start
After executing the command in both sides the locking problem should dissappear.
Some more information on NFS software can be found here:
Archlinux wiki: NFS
You could check if nfslock is running on both the nfs server and client machines. It is responsible for managing the locks.
Related
I am having an issue where Google Cloud Ops Agent logging gathers a lot of data and fills up my entire debian server hard drive in about 3 weeks due to the ever increasing size of the log file.
I do not want to increase the size of my server hard drive.
Does anyone know how to configure Google Cloud Ops Agent so that it only retains log data for the previous 7 days ?
EDIT: Google Cloud Ops Agent log file is stored in directory below
/var/log/google-cloud-ops-agent/subagents/logging-module.log
I faced the same issue recently while using agent 2.11.0. And it's not just an enormous log file, it's also a ridiculous CPU usage! Check it out in htop.
If you open the log file you'll see it spamming errors about buffer chunks. Apparently, they got broken smh, so the agent can't read them and send away. Thus, high IO and CPU usage.
The solution is to stop the service:
sudo service google-cloud-ops-agent stop
Then clear all buffer chunks:
sudo rm -rf /var/lib/google-cloud-ops-agent/fluent-bit/buffers/
And delete log file if you want:
sudo rm -f /var/log/google-cloud-ops-agent/subagents/logging-module.log
Then start the agent:
sudo service google-cloud-ops-agent start
This helped me out.
Btw this issue is described here and it seems that Google "fixed" it since 2.7.0-1. Whatever they mean by it since we still faced it...
After quite a bit of trial and error and a step by step attempt to find solutions I thought I share the problems here and answer them myself according to what I've found. There is not too much documentation on this anywhere except small bits and pieces and this will hopefully help others in the future.
Please note that this is specific to Django, Celery, Redis and the Digital Ocean App Platform.
This is mostly about the below errors and further resulting implications:
OSError: [Errno 38] Function not implemented
and
Cannot connect to redis://......
The first error happens when you try run the celery command celery -A your_app worker --beat -l info
or similar on the App Platform. It appears that this is currently not supported on digital ocean. The second error occurs when you make a number of potential mistakes.
PART 1:
While Digital Ocean might remedy this in the future here is an approach that will offer a workaround. The problem is the not supported execution pool. Google "celery execution pools" if you want to know more and how they work. The default one is prefork. But what you need is either gevent or eventlet. I went with the former for my purposes.
Whichever you pick you will have to install it as it doesn't come with celery by default. In my case it was: pip install gevent (and don't forget adding it to your requirements as well).
Once you have that you can re-run the celery command but note that gevent and beat are not supported within a single command (will result in an error). Instead do the following:
celery -A your_app worker --pool=gevent -l info
and then separately (if you want to run beat that is) in another terminal/console
celery -A your_app beat -l info
In the first line you can also specify the concurrency like so: --concurrency=100. This is not required but useful. Read up on it what it does as that goes beyond the solution here.
PART 2:
In my specific case I've tested the above locally (development) first to make sure they work. The next issue was getting this into production. I use Redis as the db/broker.
In my specific setup I have most of my celery configuration in the_main_app/celery/__init__.py file but sometimes people put it directly into the_main_app/celery.py. Whichever it is you do make sure that the REDIS_URL is set correctly. For development it usually looks something like this:
YOUR_VAR_NAME = os.environ.get('REDIS_URL', 'redis://localhost:6379') where YOUR_VAR_NAME is then set to the broker with everything as below:
YOUR_VAR_NAME = os.environ.get('REDIS_URL', 'redis://localhost:6379')
app = Celery('the_main_app')
app.conf.broker_url = YOUR_VAR_NAME
The remaining settings are all documented on the "celery first steps with django" help page but are not relevant for what I am showing here.
PART 3:
When you setup your Redis Database on the App Platform (which is very simple) you will see the connection details as 'public network' and 'VPC network'.
The celery documentation says to use the following URL format for production: redis://:password#hostname:port/db_number. This didn't work. If you are not using a yaml file then you can simply copy paste the entire connection string (select from the dropdown!) from the Redis DB connection details and then setup an App-Level environment variable in your Digital Ocean project named REDIS_URL and paste in that entire string (and also encrypt it!).
The string should look like something like this (redis with 2 s!)
rediss://USER:PASS#URL.db.ondigitialocean.com:PORT.
You are almost done. The last step is to setup the workers. It was fine for me to run the PART 1 commands as console commands on the App Platform to test them but eventually I've setup a small worker (+ Add Component) for each line pasted them into the Run Command.
That is basically the process step by step. Good luck!
I have deployed two rails apps to Digital Ocean, Ubuntu 18.04 with Passenger and Nginx.
Both apps were built on rails 5.2.2 with ruby 2.5.1, and the second app has all the same gems at the same versions. While the first app runs fine, the second will not launch.
The last useful line of the Passenger log says:
[ E 2020-08-06 22:41:56.6186 30885/T1i age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/html/AppName_Prod/current: The application encountered the following error: ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
I know this is somethign to do with the master.key file, but that is present and contains the correct key. I'm not using environment vars to store the master keys - they are in the master.key file inside each app's dir structure.
I've read every SO post I could find on this and none have solved my issue.
Any suggestions for getting these two apps (and more) to work on the same droplet?
I'm all out of ideas.
Thank you for any help you can offer.
For anyone who might have the same issue, it was a bit deceptive.
I had tried rails credentials:edit and it didn't fix the issue, but I found that the app's containing folder was owned by user:user, whereas my other app was owned by user:root.
When I changed this, everything started to work.
I hope it helps someone because I didn't find this info anywhere online and it was a lot of trial and error.
Use ls -l to list the current owner of folders in the current working directory, so you can compare them.
For me, this turned out to be somewhat complicated. I had provisioned my server using Ansible, which has a task to copy the Nginx conf. After provisioning the server, I changed RAILS_MASTER_KEY.
It turns out that my Ansible task does not re-write the Nginx conf if it already exists on the server (the file is not compared, I guess). So although I updated RAILS_MASTER_KEY in my Ansible playbook (and it was even getting copied across to the server's environment variables!), it was not accessible to Rails through passenger because it does not pass on the user's environment variables.
Whew!
To fix this (and create a snowflake server in the process...) I manually logged into the server and updated RAILS_MASTER_KEY to my new value in the Nginx passenger_env_var.
Please help I have trouble with Divio App trying to making it work.
When I press "set up project"
it gives me this
*
Creating workspace
cloning project repository
Cloning into '/c/Users/Ubisoft/Documents/iloveit'...
Bad owner or permissions on /home/divio/.ssh/config
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
There was an error trying to run a command. This is most likely
not an issue with divio-cli, but the called program itself.
Try checking the output of the command above.
The command was:
git clone git#git.divio.com:iloveit.git /c/Users/Ubisoft/Documents/iloveit
*
and in windows power shell it gives me this
Creating workspace
cloning project repository
Cloning into '/c/Users/Ubisoft/Documents/iloveit'...
Bad owner or permissions on /home/divio/.ssh/config
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
------------------------------------------------------------------------------------------------------------------------
There was an error trying to run a command. This is most likely
not an issue with divio-cli, but the called program itself.
Try checking the output of the command above.
The command was:
git clone git#git.divio.com:iloveit.git /c/Users/Ubisoft/Documents/iloveit
divio#app-1.0.0 /c/Users/Ubisoft/Documents
$
I also tried this from virtual MacOS and getting this message:
https://i.stack.imgur.com/QccvY.png
I also tried to mess arouond with creating SSH keys but didn't work out.
Can someone provide me step by step explanation how to make this wonderful app work?
In the Windows examples you show, I see:
Bad owner or permissions on /home/divio/.ssh/config
I am not sure how that has happened, but that is what is preventing your local environment from providing the expected key to the Divio Control Panel.
In the Macintosh example, the environment doesn't have a key that the Control Panel knows about.
You will need to add the key (probably from ~/.ssh/rsa_id.pub) to https://control.divio.com/account/ssh-keys/. If you don't already have a key in the Macintosh environment, you will need to set one up.
For anyone who may face this issue. On my MacOS virtual machine I managed to find solution. Looks like it is Internet Service Provider blocking port 22 something like that. Okay looks like problem resolved. I used VPN and without any hastle with SSH I got different result. Looks like it is working now not finished creating project yet but promising:
Creating workspace
cloning project repository
Cloning into '/Users/johnwick/Documents/best-project'...
Locking the website...
remote: Counting objects: 785, done.
remote: Compressing objects: 100% (739/739), done.
Unlocking the website...(385/785), 1.05 MiB | 524.00 KiB/s
remote: Total 785 (delta 112), reused 0 (delta 0)
Receiving objects: 100% (785/785), 1.77 MiB | 448.00 KiB/s, done.
Resolving deltas: 100% (112/112), done.
Checking out files: 100% (615/615), done.
downloading remote docker images
Pulling db ... done
Pulling web ... done
building local docker images
db uses an image, skipping
Building web
Step 1/7 : FROM divio/base:4.15-py3.6-slim-stretch
4.15-py3.6-slim-stretch: Pulling from divio/base
I have been trying to find out the best way to run background jobs using PHP on AWS Elastic beanstalk, and after many hours searching on Google and SO, I believe that one good solution is using SWF and activity workers.
I found this example buried in the aws-sdk-for-php: https://github.com/amazonwebservices/aws-sdk-for-php/tree/master/_samples/AmazonSimpleWorkflow/cron
The read-me file says:
To run this sample, you need to execute three scripts from the command line in separate terminal/console windows
and
Note that the start_cron_example_workflow.php script will exit quickly
while the decider and activity worker scripts keep running until you
manually terminate them.
the decider and activity worker will loop "forever", and trying to run these in EB is what I'm having trouble doing.
In my .ebextensions directory I have a file that executes these files:
container_commands:
01background_task:
command: "php -f start_cron_example_activity_workers.php"
02background_task:
command: "php -f start_cron_example_workflow_workers.php"
But I get the following error messages:
ERROR
Failed to deploy application version.
ERROR
Some instances have not responded to commands. Responses were not received from [i-a5417ed4].
Any way I can do this using config files? How can I make this work in AWS EB without introducing a single point of failure?
Thank you.
You might consider using a service like IronWorker — this is specifically designed for what you are trying to do and will probably work better than putting together your own solution on a micro instance.
I have not used Iron.io yet, but was evaluating it as I am looking to move my stuff over to AWS so I need to have cron jobs handled as well.
Have you taken a look at the Fat Controller ? It can daemonise anything. There's documentation and examples on the website: http://fat-controller.sourceforge.net