I have a WordPress site with gunicorn and varnish running on an AWS instance.
This morning, the website gave a "502 Bad Gateway nginx" error.
Upon investigation, it looks like the varnish.service port was:
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
According to some notes, the port needs to be 80 and not 6081. Changing the port to 80 fixed the nginx error.
This issue seems to happen about once a year where the varnish.service port suddenly changes by itself and someone has to manually change the port back to 80.
So my question is - why would varnish.service suddenly change its port? As far as I know, there were no updates or changes anywhere.
It depends on what file you're editing.
Make sure you're editing /etc/systemd/system/varnish.service. If that file isn't there, just run the following command:
sudo cp /lib/systemd/system/varnish.service /etc/systemd/system/
When you're done editing the port, just run the following 2 commands:
sudo systemctl daemon-reload
sudo systemctl restart
See https://www.varnish-software.com/developers/tutorials/installing-varnish-ubuntu/#systemd-configuration for a detailed tutorial.
When I run docker-compose up in my Docker project it fails with the following message:
Error starting userland proxy: listen tcp 0.0.0.0:3000: bind: address already in use
netstat -pna | grep 3000
shows this:
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN -
I've already tried docker-compose down, but it doesn't help.
In your case it was some other process that was using the port and as indicated in the comments, sudo netstat -pna | grep 3000 helped you in solving the problem.
While in other cases (I myself encountered it many times) it mostly is the same container running at some other instance. In that case docker ps was very helpful as often I left the same containers running in other directories and then tried running again at other places, where same container names were used.
How docker ps helped me:
docker rm -f $(docker ps -aq) is a short command which I use to remove all containers.
Edit: Added how docker ps helped me.
This helped me:
docker-compose down # Stop container on current dir if there is a docker-compose.yml
docker rm -fv $(docker ps -aq) # Remove all containers
sudo lsof -i -P -n | grep <port number> # List who's using the port
and then:
kill -9 <process id> (macOS) or sudo kill <process id> (Linux).
Source: comment by user Rub21.
I had the same problem. I fixed this by stopping the Apache2 service on my host.
You can kill the process listening on that port easily with one command below :
kill -9 $(lsof -t -i tcp:<port#>)
ex :
kill -9 $(lsof -t -i tcp:<port#>)
or for ubuntu:
sudo kill -9 `sudo lsof -t -i:8000`
Man page for lsof : https://man7.org/linux/man-pages/man8/lsof.8.html
-9 is for hard kill without checking any deps.
(Not related, but might be useful if its PORT 5000 mystery) - the culprit process is due to Mac OS monterery.
The port 5000 is commonly used to serve local development servers. When updating to the latest macOS operating system, I was unable the docker to bind to port 5000, because it was already in use. (You may find a message along the lines of Port 5000 already in use.)
By running lsof -i :5000, I found out the process using the port was named ControlCenter, which is a native macOS application. If this is happening to you, even if you use brute force (and kill) the application, it will restart itself. In my laptop, lsof -i :5000 returns that Control Center is being used by process id 433. I could do killall -p 433, but macOS keeps restarting the process.
The process running on this port turns out to be an AirPlay server. You can deactivate it in
System Preferences › Sharing, and unchecking AirPlay Receiver to release port 5000.
I had same problem,
docker-compose down --rmi all (in the same directory where you run docker-compose up)
helps
UPD: CAUTION - this will also delete the local docker images you've pulled (from comment)
For Linux/Unix:
Simple search for linux utility using following command
netstat -nlp | grep 8888
It'll show processing running at this port, then kill that process using PID (look for a PID in row) of that process.
kill PID
In some cases it is critical to perform a more in-depth debugging to the problem before stopping a container or killing a process.
Consider following the checklist below:
1) Check you current docker compose environment
Run docker-compose ps. If port is in use by another container, stop it with docker-compose stop <service-name-in-compose-file> or remove it by replacing stop with rm.
2) Check the containers running outside your current workspace
Run docker ps to see list of all containers running under your host.
If you find the port is in use by another container, you can stop it with docker stop <container-id>.
(*) Because you're not under the scope of the origin compose environment - it is a good practice first to use docker inspect to gather more information about the container that you're about to stop.
3) Check if port is used by other processes running on the host
For example if the port is 6379 run:
$ sudo netstat -ltnp | grep ':6379'
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 915/redis-server 12
tcp6 0 0 ::1:6379 :::* LISTEN 915/redis-server 12
(*) You can also use the lsof command which is mainly used to retrieve information about files that are opened by various processes (I suggest running netstat before that).
So, In case of the output above the PID is 915. Now you can run:
$ ps j 915
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 915 915 915 ? -1 Ssl 123 0:11 /usr/bin/redis-server 127.0.0.1:6379
And see the ID of the parent process (PPID) and the execution command.
You can also run: $ pstree -s <PID> to a visual display of the process and its related processes.
In our case we can see that the process probably is a daemon (PPID is 1) - In that case consider running: A) $ cat /proc/<PID>/status in order to get a more in-depth information about the process like the number of threads spawned by the process, its capabilities, etc'.
B) $ systemctl status <PID> in order to see the systemd unit that caused the creation of a specific process. If the service is not critical - you can stop and disable the service.
4) Restart Docker service
Run: sudo service docker restart.
5) You reached this point and..
Only if its not placing your system at risk - consider restarting the server.
In my case it was
Error starting userland proxy: listen tcp 0.0.0.0:9000: bind: address already in use
And all that I need is turn off debug listening in php storm
Most probably this is because you are already running a web server on your host OS, so it conflicts with the web server that Docker is attempting to start.
So try this one-liner before trying anything else:
sudo service apache2 stop; sudo service nginx stop; sudo nginx -s stop;
I had apache running on my ubuntu machine. I used this command to kill it!
sudo /etc/init.d/apache2 stop
I was getting the below error when i was trying to launch a new container -
listen tcp 0.0.0.0:8080: bind: address already in use.
To check which process is running on port 8080, run below command:
netstat -tulnp | grep 8080
i got the output below
[root#ip-112-x6x-2x-xxx.xxxxx.compute.internal (aws_main) ~]# netstat -tulnp | grep 8080 tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN **12749**/java [root#ip-112-x6x-2x-xxx.xxxxx.compute.internal (aws_main) ~]#
run
kill -9 12749
Then try to relaunch the container it should work
If redis server is started as a service, it will restart itself when you using kill -9 <process_id> or sudo kill -9 `sudo lsof -t -i:<port_number>` . In that case you will need to stop the redis service using following command.
sudo service redis-server stop
I upgraded my docker this afternoon and ran into the same problem. I tried restarting docker but no luck.
Finally, I had to restart my computer and it worked. Definitely a bug.
Check docker-compose.yml, it might be the case that the port is specified twice.
version: '3'
services:
registry:
image: mysql:5.7
ports:
- "3306:3306" <--- remove either this line or next
- "127.0.0.1:3306:3306"
Changing network_mode: "bridge" to "host" did it for me.
This with
version: '2.2'
services:
bind:
image: sameersbn/bind:latest
dns: 127.0.0.1
ports:
- 172.17.42.1:53:53/udp
- 172.17.42.1:10000:10000
volumes:
- "/srv/docker/bind:/data"
environment:
- 'ROOT_PASSWORD=secret'
network_mode: "host"
I ran into the same issue several times. Restarting docker seems to do the trick
A variation of #DmitrySandalov's answer: I had tomcat/java running on 8080, which needed to keep going. Looked at the docker-compose.yml file and altered the entry for 8080 to another of my choosing.
nginx:
build: nginx
ports:
#- '8080:80' <-- original entry
- '8880:80'
- '8443:443'
Worked perfectly. (The only wrinkle is the change will be wiped if I ever update the project, since it's coming from an external repo.)
At first, make sure which service you are running in your specific port. In your case, you are already using port number 3000.
netstat -aof | findstr :3000
now stop that process which is running on specific port
lsof -i tcp:3000
I resolve the issue by restarting Docker.
It makes more sense to change the port of the docker update instead of shutting down other services that use port 80.
Just a side note if you have the same issue and is with Windows:
In my case the process in my way is just grafana-server.exe. Because I first downloaded the binary version and double click the executable, and it now starts as a service by user SYSTEM which I cannot taskkill (no permission)
I have to go to "Service manager" of Windows and search for service "Grafana", and stop it. After that port 3000 is no longer occupied.
Hope that helps.
The one that was using the port 8888 was Jupiter and I had to change the configuration file of Jupiter notebook to run on another port.
to list who is using that specific port.
sudo lsof -i -P -n | grep 9
You can specify the port you want Jupyter to run uncommenting/editing the following line in ~/.jupyter/jupyter_notebook_config.py:
c.NotebookApp.port = 9999
In case you don't have a jupyter_notebook_config.py try running jupyter notebook --generate-config. See this for further details on Jupyter configuration.
Before it was running on :docker run -d --name oracle -p 1521:1521 -p 5500:5500 qa/oracle
I just changed the port to docker run -d --name oracle -p 1522:1522 -p 5500:5500 qa/oracle
it worked fine for me !
On my machine a PID was not being shown from this command netstat -tulpn for the in-use port (8080), so i could not kill it, killing the containers and restarting the computer did not work. So service docker restart command restarted docker for me (ubuntu) and the port was no longer in use and i am a happy chap and off to lunch.
maybe it is too rude, but works for me. restart docker service itself
sudo service docker restart
hope it works for you also!
I have run the container with another port, like... 8082 :-)
I came across this problem. My simple solution is to remove the mongodb from the system
Commands to remove mongodb in Ubuntu:
sudo apt-get purge mongodb mongodb-clients mongodb-server mongodb-dev
sudo apt-get purge mongodb-10gen
sudo apt-get autoremove
Let me add one more case, because I had the same error and none of the solutions listed so far works:
serv1:
...
networks:
privnet:
ipv4_address: 10.10.100.2
...
serv2:
...
# no IP assignment, no dependencies
networks:
privnet:
ipam:
driver: default
config:
- subnet: 10.10.100.0/24
depending on the init order, serv2 may get assigned the IP 10.10.100.2 before serv1 is started, so I just assign IPs manually for all containers to avoid the error. Maybe there are other more elegant ways.
I have the same problem and by stopping docker container it was resolved.
sudo docker container stop <container-name>
i solved with this sudo service redis-server stop
I am a beginner with aerospike with Python client. I have an aerospike [build - 3.9.1.1] installed in a google instance.
How can I connect to the server from another instance?
I tried the following in shell and wouldn't budge:
import aerospike
config = {'hosts': [('xx.mmm.nn.oo', 3000)]}
# the IP from ifconfig | grep "inet addr"
client = aerospike.client(config)
client.connect()
This yields exception as below:
ClientError: (-1L, 'Failed to connect', 'src/main/aerospike/as_cluster.c', 459)
In the remote I tried the following:
This works fine:
asinfo -v "namespaces"
Also this works when I do:
telnet xx.mmm.nn.oo 3003
....
namespaces
...
Going desperate, I have tried setting access-address in the local with the IP of the server [xx.mmm.nn.oo] and that didn't work either!
Please help folks!
Issue apparently resolved by allowing connections by editing the iptables:
/sbin/iptables -A INPUT -p tcp -s XXX.XXX.XXX.XXX -j ACCEPT /sbin/iptables -A OUTPUT -p tcp -d XXX.XXX.XXX.XXX -j ACCEPT –
I'm testing out using memcached to cache django views. How can I tell if memcached is actually caching anything from the Linux command line?
You could use the official perl script:
memcached-tool 127.0.0.1:11211 stats
Or just use telnet and the stats command e.g.:
# telnet localhost [memcacheport]
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stats
STAT pid 2239
STAT uptime 10228704
STAT time 1236714928
STAT version 1.2.3
STAT pointer_size 32
STAT rusage_user 2781.185813
STAT rusage_system 2187.764726
STAT curr_items 598669
STAT total_items 31363235
STAT bytes 37540884
STAT curr_connections 131
STAT total_connections 8666
STAT connection_structures 267
STAT cmd_get 27
STAT cmd_set 30694598
STAT get_hits 16
STAT get_misses 11
STAT evictions 0
STAT bytes_read 2346004016
STAT bytes_written 388732988
STAT limit_maxbytes 268435456
STAT threads 4
END
I know this question is old, but here is another useful approach for testing memcached with django:
As #Jacob mentioned, you can start memcached in very verbose mode (not as a daemon):
memcached -vv
To test your django cache config, you can use the low-level cache api.
First, start up the python interpreter and load your django project settings:
python manage.py shell
From the shell, you can use the low-level cache api to test your memcache server:
from django.core.cache import cache
cache.set('test', 'test value')
If your cache configuration is correct, you should see output in memcache similar to this:
<32 set :1:test 0 300 10
>32 STORED
Start memcache not as a daemon but normal, so just run memcached -vv for very verbose. You will see when get's and sets come in to the memcache server.
Simple way to test for memcache working was to sneak in a commented out timestamp on every page served up. If the timestamp stayed the same on multiple requests to a page, then the page was being cached by memcache.
In Django settings, I also setup the cache mechanism to use a file cache on the filesystem (really slow), but after hitting up the pages I could see that there were actual cache files being placed in the file path so I could confirm caching was active in Django.
I used both these steps to work out my caching problem. I actually did not have caching turned on correctly in Django. The newer method to activate caching is using the 'django.middleware.cache.CacheMiddleware' middleware (not the middleware with two middleware pieces that have to be the first/last middleware settings.)
From the command line, try the command below:
echo stats | nc 127.0.0.1 11211
If it doesn't return anything, memcache isn't running. Otherwise it should return a bunch of stats including uptime (and hit and miss counts)
The reference article is here,
https://www.percona.com/blog/2008/11/26/a-quick-way-to-get-memcached-status/
To see changes every 2 seconds:
watch "echo stats | nc 127.0.0.1 11211"
In Bash, you can check the statistics of memcache by this command:
exec 3<>/dev/tcp/localhost/11211; printf "stats\nquit\n" >&3; cat <&3
To flush the cache, use memflush command:
echo flush_all >/dev/tcp/localhost/11211
and check if the stats increased.
To dump all the cached objects, use memdump or memcdump command (part of memcached/libmemcached package):
memcdump --servers=localhost:11211
or:
memdump --servers=localhost:11211
If you're using PHP, to see whether is supported, check by: php -i | grep memcached.
Tracing
To check what memcached process is exactly processing, you can use network sniffers or debuggers (e.g. strace on Linux or dtrace/dtruss on Unix/OS X) for that. Check some examples below.
Strace
sudo strace -e read,write -fp $(pgrep memcached)
To format output in a better way, check: How to parse strace in shell into plain text?
Dtruss
Dtruss is a dtrace wrapper which is available on Unix systems. Run it as:
sudo dtruss -t read -fp $(pgrep memcached)
Tcpdump
sudo tcpdump -i lo0 -s1500 -w- -ln port 11211 | strings -10
Memcached can actually write to a logfile on its own, without having to resort to restarting it manually. The /etc/init.d/memcached init script (/usr/lib/systemd/system/memcached.service on EL7+; ugh) can call memcached with the options specified in /etc/memcached.conf (or /etc/sysconfig/memcached on EL5+). Among these options are verbosity and log file path.
In short, you just need to add (or uncomment) these two lines to that conf/sysconfig file...
-vv
logfile /path/to/log
...and restart the daemon with service memcached restart(EL3-7) or /etc/init.d/memcached restart(debuntus)
And then you can monitor this log in the traditional way, like tail -f /path/to/log, for example.
For extend Node's response, you can use socat UNIX-CONNECT:/var/run/memcached.sock STDIN to debug a unix socket.
Example:
$ socat UNIX-CONNECT:/var/run/memcached.sock STDIN
stats
STAT pid 931
STAT uptime 10
STAT time 1378574384
STAT version 1.4.13
STAT libevent 2.0.19-stable
STAT pointer_size 32
STAT rusage_user 0.000000
STAT rusage_system 0.015625
STAT curr_connections 1
STAT total_connections 2
STAT connection_structures 2
You can test memcached or any server by below script
lsof -i :11211 | grep 'LISTEN'>/dev/null 2>/dev/null;echo $?
if it returns 0 then the server is actually running or if 1 its not so if you want to know that the server is actually running on some port use the following script
lsof -i :11211 | grep 'LISTEN'>/dev/null 2>/dev/null;
if [ $? -eq 0]; then
echo "Your memcache server is running"
else
echo "No its not running"
fi
Can you use curl to fetch a page a few hundred times and time the results? You could also look at running a process on the server that simulates heavy CPU/disk load while doing this.
I wrote an expect script is-memcached-running that tests if memcached is running on a host/port combination (run as is-memcached-running localhost 11211):
#! /usr/bin/env expect
set timeout 1
set ip [lindex $argv 0]
set port [lindex $argv 1]
spawn telnet $ip $port
expect "Escape character is '^]'."
send stats\r
expect "END"
send quit\r
expect eof
If you run your system from a Makefile rule, you could make your startup depend on a make target that asserts it is up and running (or helps you get that state). It is verbose when the check fails to make it easy for us to debug failed ci runs, installs memcached when it's missing, and is brief and to the point otherwise:
#! /bin/bash
if [[ "$(type -P memcached)" ]]; then
echo 'memcached installed; checking if it is running'
memcached_debug=`mktemp memcache-check.XXXXX`
if is-memcached-running localhost 11211 >$memcached_debug 2>&1; then
echo 'Yep; memcached online'
else
cat $memcached_debug
echo
echo '****** Error: memcached is not running! ******'
if [[ "$OSTYPE" =~ ^darwin ]]; then
echo
echo 'Instructions to auto-spawn on login (or just start now) are shown'
echo 'at the end of a "brew install memcached" run (try now, if you did'
echo 'not do so already) or, if you did, after a "brew info memcached".'
echo
fi
exit 1
fi
rm -f $memcached_debug
else
echo memcached was not found on your system.
if [[ "$OSTYPE" =~ ^darwin ]]; then
brew install memcached
elif [[ "$OSTYPE" =~ ^linux ]]; then
sudo apt-get install memcached
else
exit 1
fi
fi
Following Aryashree post, this helped me to get an error if memcached not running locally:
import subprocess
port = 11211
res = subprocess.Popen(f"echo stats | nc 127.0.0.1 {port}",
shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if res.stdout:
lines = res.stdout.read()
lineArr = lines.split('\r\n')
pidlineArr = lineArr[0].split(' ')
pid = pidlineArr[-1]
print(f"[MemCached] pid {pid} Running on port {port}")
else:
raise RuntimeError(f"No Memcached is present on port {port}")
I'm using Mezzanine and the only answer that worked for me was Jacobs answer. So stopping the daemon and running memcached -vv
If you're using RHEL or Centos 8
To get memcached log stuff to /var/log/messages (quick without rotation)
https://serverfault.com/questions/208538/how-to-specify-the-log-file-for-memcached-on-rhel-centos/1054741#1054741