I've been trying to run Playwright on AWS Lambda with xvfb for days. I was able to build a few variants of Docker images that worked locally, but when pushed to AWS always encountered errors.
Dockerfile:
FROM mcr.microsoft.com/playwright/python:v1.24.0-focal
COPY index.py /
COPY start.sh /
RUN apt install xvfb
RUN chmod 755 /start.sh
CMD ["/bin/bash", "./start.sh"]
start.sh
#!/bin/bash
xvfb-run --auto-servernum --server-num 1 --server-args "-screen 0, 1920x1080x24" python index.py
index.py
from playwright.sync_api import sync_playwright
def lambda_handler(event, context):
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.set_default_timeout(0)
page.goto("http://example.com")
title = page.title()
browser.close()
return title
Looks like for some permission-related reason the browser can't be started - any ideas what might be causing it & how to solve it?
2022-08-30T09:50:28.053+02:00 Traceback (most recent call last):
2022-08-30T09:50:28.053+02:00 File "index.py", line 8, in lambda_handler
2022-08-30T09:50:28.053+02:00 page = browser.new_page()
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/sync_api/_generated.py", line 11188, in new_page
2022-08-30T09:50:28.053+02:00 self._sync(
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_sync_base.py", line 89, in _sync
2022-08-30T09:50:28.053+02:00 return task.result()
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_browser.py", line 174, in new_page
2022-08-30T09:50:28.053+02:00 page = await context.new_page()
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_browser_context.py", line 230, in new_page
2022-08-30T09:50:28.053+02:00 return from_channel(await self._channel.send("newPage"))
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 43, in send
2022-08-30T09:50:28.053+02:00 return await self._connection.wrap_api_call(
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 369, in wrap_api_call
2022-08-30T09:50:28.053+02:00 return await cb()
2022-08-30T09:50:28.053+02:00 File "/usr/local/lib/python3.8/dist-packages/playwright/_impl/_connection.py", line 78, in inner_send
2022-08-30T09:50:28.053+02:00 result = next(iter(done)).result()
2022-08-30T09:50:28.053+02:00 playwright._impl._api_types.Error: Browser closed.
2022-08-30T09:50:28.053+02:00 ==================== Browser output: ====================
2022-08-30T09:50:28.053+02:00 <launching> /ms-playwright/chromium-1015/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-gHIdYn --remote-debugging-pipe --no-startup-window
<launching> /ms-playwright/chromium-1015/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-gHIdYn --remote-debugging-pipe --no-startup-window
2022-08-30T09:50:28.053+02:00 <launched> pid=129
2022-08-30T09:50:28.053+02:00
[pid=129][err] [0830/075021.572233:WARNING:crashpad_client_linux.cc(362)] prctl: Operation not permitted (1)
[pid=129][err] [0830/075021.572233:WARNING:crashpad_client_linux.cc(362)] prctl: Operation not permitted (1)
2022-08-30T09:50:28.053+02:00 [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00 [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00
[pid=129][err] [129:147:0830/075023.537911:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[pid=129][err] [129:147:0830/075023.537911:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:147:0830/075023.538017:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075023.774216:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075023.774254:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075024.183157:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075024.183204:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.449390:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.453692:ERROR:network_service_instance_impl.cc(461)] Network service crashed, restarting service.
2022-08-30T09:50:28.053+02:00 [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.510964:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:161:0830/075024.511033:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:161:0830/075024.511061:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:161:0830/075024.511109:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:161:0830/075024.512161:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:161:0830/075024.512820:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.570748:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.632383:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.671409:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075024.674327:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075024.854785:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:146:0830/075024.854822:ERROR:bus.cc(398)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075025.701411:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075025.712194:ERROR:network_service_instance_impl.cc(461)] Network service crashed, restarting service.
2022-08-30T09:50:28.053+02:00 [pid=129][err] prctl(PR_SET_NO_NEW_PRIVS) failed
2022-08-30T09:50:28.053+02:00 [pid=129][err] [129:129:0830/075025.793276:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002
2022-08-30T09:50:28.053+02:00 [pid=129][err] [0830/075025.850673:ERROR:scoped_ptrace_attach.cc(27)] ptrace: Operation not permitted (1)
2022-08-30T09:50:28.053+02:00 [pid=129][err] Received signal 11 SEGV_MAPERR 000000000000
2022-08-30T09:50:28.053+02:00 [pid=129][err] #0 0x55fa8ab9efb9 base::debug::CollectStackTrace()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #1 0x55fa8ab11853 base::debug::StackTrace::StackTrace()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #2 0x55fa8ab9eac1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #3 0x7f93d4dc4420 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so+0x1441f)
2022-08-30T09:50:28.053+02:00 [pid=129][err] #4 0x55fa88ae4589 content::WebContentsImpl::SetDeviceEmulationSize()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #5 0x55fa8864714b content::protocol::EmulationHandler::SetDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #6 0x55fa886476e8 content::protocol::EmulationHandler::SetDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #7 0x55fa884a52af content::protocol::Emulation::DomainDispatcherImpl::setDeviceMetricsOverride()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #8 0x55fa89b0c0c3 v8_crdtp::UberDispatcher::DispatchResult::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #9 0x55fa886286fd content::DevToolsSession::HandleCommandInternal()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #10 0x55fa8862866b content::DevToolsSession::HandleCommand()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #11 0x55fa8862ac6e base::internal::Invoker<>::RunOnce()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #12 0x55fa8d741a18 ChromeDevToolsSession::HandleCommand()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #13 0x55fa8d740048 ChromeDevToolsManagerDelegate::HandleCommand()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #14 0x55fa8862858e content::DevToolsSession::DispatchProtocolMessageInternal()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #15 0x55fa8862823f content::DevToolsSession::DispatchProtocolMessage()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #16 0x55fa88624162 base::internal::FunctorTraits<>::Invoke<>()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #17 0x55fa8ab5b010 base::TaskAnnotator::RunTaskImpl()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #18 0x55fa8ab6ebfc base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #19 0x55fa8ab6e7ad base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #20 0x55fa8ab6f172 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #21 0x55fa8ab1834b base::(anonymous namespace)::WorkSourceDispatch()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #22 0x7f93d4c7917d g_main_context_dispatch
2022-08-30T09:50:28.053+02:00 [pid=129][err] #23 0x7f93d4c79400 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6400.6+0x523ff)
2022-08-30T09:50:28.053+02:00 [pid=129][err] #24 0x7f93d4c794a3 g_main_context_iteration
2022-08-30T09:50:28.053+02:00 [pid=129][err] #25 0x55fa8ab18177 base::MessagePumpGlib::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #26 0x55fa8ab6f3e3 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #27 0x55fa8ab37aed base::RunLoop::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #28 0x55fa8859dcad content::BrowserMainLoop::RunMainMessageLoop()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #29 0x55fa8859f4f2 content::BrowserMainRunnerImpl::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #30 0x55fa8859b7dc content::BrowserMain()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #31 0x55fa8a67ab88 content::RunBrowserProcessMain()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #32 0x55fa8a67c0da content::ContentMainRunnerImpl::RunBrowser()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #33 0x55fa8a67bc70 content::ContentMainRunnerImpl::Run()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #34 0x55fa8a678e1e content::RunContentProcess()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #35 0x55fa8a6797de content::ContentMain()
2022-08-30T09:50:28.053+02:00 [pid=129][err] #36 0x55fa86f8f227 ChromeMain
2022-08-30T09:50:28.053+02:00 [pid=129][err] #37 0x7f93d3eef083 __libc_start_main
2022-08-30T09:50:28.053+02:00 [pid=129][err] #38 0x55fa86f8f02a _start
2022-08-30T09:50:28.053+02:00 [pid=129][err] r8: aaaaaaaaaaaaaaaa r9: 00007fff828b2460 r10: 0000000000000003 r11: 00000000000002d0
2022-08-30T09:50:28.053+02:00 [pid=129][err] r12: 00007fff828b2730 r13: 00007fff828b2460 r14: 0000000000000000 r15: 00000000000002d0
2022-08-30T09:50:28.053+02:00 [pid=129][err] di: 00003ac400a8e070 si: 00007fff828b2460 bp: 00007fff828b2280 bx: 00003ac40108cc00
2022-08-30T09:50:28.053+02:00 [pid=129][err] dx: aaaaaaaaaaaaaaaa ax: 0000000000000000 cx: 000055fa90552dd8 sp: 00007fff828b2230
2022-08-30T09:50:28.053+02:00 [pid=129][err] ip: 000055fa88ae4589 efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000004
2022-08-30T09:50:28.053+02:00 [pid=129][err] trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
2022-08-30T09:50:28.053+02:00 [pid=129][err] [end of stack trace]
Related
When running my Flask application through Flask normally, I encounter no issues and the "undetected_chromedriver" runs smoothly. However, to ensure that my Flask application runs continuously, I have created a system service to run it at startup. When I press a button within the Flask application to scrape data from a website, the "undetected_chromedriver" crashes abruptly. The following is the traceback I receive:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/abel/LeadPartner/app/includes/dependencies.py", line 63, in run_subprocess
subprocess.Popen(target_function(*args, **kwargs))
File "/home/abel/LeadPartner/app/scripts/scheduler/verstuur_project.py", line 15, in verstuur_project
driver = get_chromedriver(email=gebruiker_gegevens.email, headless=True)
File "/home/abel/LeadPartner/app/includes/webdriver.py", line 73, in get_chromedriver
driver = webdriver.Chrome( options=chrome_options, seleniumwire_options=proxy_options, version_main=110 )
File "/home/abel/LeadPartner/env/lib/python3.10/site- packages/seleniumwire/undetected_chromedriver/webdriver.py", line 61, in __init__
super().__init__(*args, **kwargs)
File "/home/abel/LeadPartner/env/lib/python3.10/site-packages/undetected_chromedriver/__init__.py", line 411, in __init__
browser = subprocess.Popen(
File "/usr/lib/python3.10/subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1720, in _execute_child
and os.path.dirname(executable)
File "/usr/lib/python3.10/posixpath.py", line 152, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
The command I use to run gunicorn through the terminal (which works) is: gunicorn --workers 3 --bind unix:leadpartner.sock --reload --preload -t 0 wsgi:app
And the service looks like this:
Unit]
Description=Gunicorn instance to serve Lead Partner
After=network.target
[Service]
User=abel
Group=www-data
WorkingDirectory=/home/abel/LeadPartner
Environment="PATH=/home/abel/LeadPartner/env/bin"
ExecStart=/home/abel/LeadPartner/env/bin/gunicorn --workers 3 --bind unix:leadpartner.sock -m 007 --error-logfile /home/abel/lp_error.log --capture-output --reload --preload -t 0 wsgi:app
[Install]
WantedBy=multi-user.target
The flask app itself runs fine with this service, only the chromedriver doesn’t.
I am having difficulty understanding the issue at hand. The only information I could find is that the error might be related to "undetected_chromedriver" not being able to locate the path of Google Chrome. However, when I run the Flask application through gunicorn in the terminal, it works perfectly fine. Upon examining the "undetected_chromedriver" initialization file, I discovered that it looks for "google-chrome," which is the name of the Google Chrome package I am using. I believe this cannot be the issue.
I hope someone here can help me fix this error.
Thanks in advance!
EDIT:
I've come to find out that if i add:
chrome_options.binary_location = '/usr/bin/google-chrome-stable'
Then the script goes a little further. I am now getting another error:
unknown error: cannot connect to chrome at 127.0.0.1:55843 from chrome not reachable Stacktrace: #0 0x55a3e9b2ad93 #1 0x55a3e98f915d #2 0x55a3e98e7a2b #3 0x55a3e9925c63 #4 0x55a3e991d2fc #5 0x55a3e99624f4 #6 0x55a3e9959353 #7 0x55a3e9928e40 #8 0x55a3e992a038 #9 0x55a3e9b7e8be #10 0x55a3e9b828f0 #11 0x55a3e9b62f90 #12 0x55a3e9b83b7d #13 0x55a3e9b54578 #14 0x55a3e9ba8348 #15 0x55a3e9ba84d6 #16 0x55a3e9bc2341 #17 0x7f05da868b43
Attempts
'luckily' for this error, i can at least find something on google to help me.
There are three different posts/articles created about this error. So of course, i've tried all of them. But nothing works unfortunately.
The posts are:
First post
Second post
Third post
Environment: +Centos 7.9 +python2.7 +openstack 4.0.2 (Train)
+ldd /etc/httpd/modules/mod_wsgi.so
___________________
inux-vdso.so.1 => (0x00007ffc57db8000)
libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fb54ccfb000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb54cadf000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb54c8db000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fb54c6d8000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb54c3d6000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb54c008000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb54d2f1000)
____________________
2022-09-27 18:43:08.205406 ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
2022-09-27 18:43:09.925261 mod_wsgi (pid=15104): Target WSGI script '/usr/bin/keystone-wsgi-public' cannot be loaded as Python module.
2022-09-27 18:43:09.925308 mod_wsgi (pid=15104): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
2022-09-27 18:43:09.925332 Traceback (most recent call last):
2022-09-27 18:43:09.925410 File "/usr/bin/keystone-wsgi-public", line 52, in
2022-09-27 18:43:09.925575 application = initialize_public_application()
2022-09-27 18:43:09.925591 File "/usr/lib/python2.7/site-packages/keystone/server/wsgi.py", line 24, in initialize_public_application
2022-09-27 18:43:09.925612 name='public', config_files=flask_core._get_config_files())
2022-09-27 18:43:09.925634 File "/usr/lib/python2.7/site-packages/keystone/server/flask/core.py", line 157, in initialize_application
2022-09-27 18:43:09.925672 keystone.server.configure(config_files=config_files)
2022-09-27 18:43:09.925681 File "/usr/lib/python2.7/site-packages/keystone/server/__init__.py", line 28, in configure
2022-09-27 18:43:09.925696 keystone.conf.configure()
2022-09-27 18:43:09.925703 File "/usr/lib/python2.7/site-packages/keystone/conf/__init__.py", line 137, in configure
2022-09-27 18:43:09.925718 deprecated_since=versionutils.deprecated.STEIN))
2022-09-27 18:43:09.925726 File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2055, in __inner
2022-09-27 18:43:09.925742 result = f(self, *args, **kwargs)
2022-09-27 18:43:09.925751 File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2333, in register_cli_opt
2022-09-27 18:43:09.925764 raise ArgsAlreadyParsedError("cannot register CLI option")
Since Heroku will deprecate the version of Redis my app was using (5.0.12), I upgraded to Redis version 6.2.3.
Since then, the websocket immediately triggers the onclose callback in my client. In the logs, I get this error message:
2021-06-21T18:18:20.761857+00:00 app[web.1]: 2021-06-21 18:18:20,761 ERROR [daphne.server] [asyncioreactor.py:290] - Exception inside application: [Errno 104] Connection reset by peer
2021-06-21T18:18:20.761866+00:00 app[web.1]: Traceback (most recent call last):
2021-06-21T18:18:20.761867+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels/sessions.py", line 183, in __call__
2021-06-21T18:18:20.761867+00:00 app[web.1]: return await self.inner(receive, self.send)
2021-06-21T18:18:20.761868+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels/middleware.py", line 41, in coroutine_call
2021-06-21T18:18:20.761868+00:00 app[web.1]: await inner_instance(receive, send)
2021-06-21T18:18:20.761868+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels/consumer.py", line 59, in __call__
2021-06-21T18:18:20.761869+00:00 app[web.1]: [receive, self.channel_receive], self.dispatch
2021-06-21T18:18:20.761869+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels/utils.py", line 58, in await_many_dispatch
2021-06-21T18:18:20.761870+00:00 app[web.1]: await task
2021-06-21T18:18:20.761870+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels/utils.py", line 50, in await_many_dispatch
2021-06-21T18:18:20.761871+00:00 app[web.1]: result = task.result()
2021-06-21T18:18:20.761871+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels_redis/core.py", line 435, in receive
2021-06-21T18:18:20.761871+00:00 app[web.1]: real_channel
2021-06-21T18:18:20.761872+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels_redis/core.py", line 490, in receive_single
2021-06-21T18:18:20.761872+00:00 app[web.1]: index, channel_key, timeout=self.brpop_timeout
2021-06-21T18:18:20.761872+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels_redis/core.py", line 330, in _brpop_with_clean
2021-06-21T18:18:20.761873+00:00 app[web.1]: async with self.connection(index) as connection:
2021-06-21T18:18:20.761873+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels_redis/core.py", line 835, in __aenter__
2021-06-21T18:18:20.761873+00:00 app[web.1]: self.conn = await self.pool.pop()
2021-06-21T18:18:20.761873+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/channels_redis/core.py", line 73, in pop
2021-06-21T18:18:20.761874+00:00 app[web.1]: conns.append(await aioredis.create_redis(**self.host, loop=loop))
2021-06-21T18:18:20.761886+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/aioredis/commands/__init__.py", line 175, in create_redis
2021-06-21T18:18:20.761886+00:00 app[web.1]: loop=loop)
2021-06-21T18:18:20.761887+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/aioredis/connection.py", line 133, in create_connection
2021-06-21T18:18:20.761887+00:00 app[web.1]: await conn.auth(password)
2021-06-21T18:18:20.761887+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/aioredis/util.py", line 52, in wait_ok
2021-06-21T18:18:20.761887+00:00 app[web.1]: res = await fut
2021-06-21T18:18:20.761887+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/aioredis/connection.py", line 186, in _read_data
2021-06-21T18:18:20.761888+00:00 app[web.1]: obj = await self._reader.readobj()
2021-06-21T18:18:20.761888+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/aioredis/stream.py", line 102, in readobj
2021-06-21T18:18:20.761888+00:00 app[web.1]: await self._wait_for_data('readobj')
2021-06-21T18:18:20.761888+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
2021-06-21T18:18:20.761888+00:00 app[web.1]: await self._waiter
2021-06-21T18:18:20.761889+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/asyncio/selector_events.py", line 814, in _read_ready__data_received
2021-06-21T18:18:20.761889+00:00 app[web.1]: data = self._sock.recv(self.max_size)
2021-06-21T18:18:20.761889+00:00 app[web.1]: ConnectionResetError: [Errno 104] Connection reset by peer
CHANNEL_LAYERS value in settings.py
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [os.environ['REDIS_URL']],
"symmetric_encryption_keys": [SECRET_KEY],
},
},
}
requirements.txt
aiohttp==3.6.2
aioredis==1.3.1
amqp==2.6.0
appdirs==1.4.3
asgi-redis==1.4.3
asgiref==3.2.7
async-timeout==3.0.1
atomicwrites==1.4.0
attrs==19.3.0
autobahn==20.4.3
Automat==20.2.0
autopep8==1.5.2
AWSIoTPythonSDK==1.4.7
billiard==3.6.3.0
CacheControl==0.12.6
cachetools==4.1.0
celery==4.4.6
certifi==2020.4.5.1
cffi==1.14.0
channels==2.4.0
channels-redis==2.4.2
chardet==3.0.4
colorama==0.4.3
constantly==15.1.0
contextlib2==0.6.0
cryptography==2.9.2
daphne==2.5.0
ddtrace==0.40.0
defusedxml==0.6.0
distlib==0.3.0
distro==1.4.0
dj-database-url==0.5.0
Django==3.0.6
django-braces==1.14.0
django-cors-headers==3.2.1
django-datadog-logger==0.3.2
django-filter==2.2.0
django-heroku==0.3.1
django-oauth-toolkit==1.3.2
django-rest-framework-social-oauth2==1.1.0
django-rest-passwordreset==1.1.0
djangorestframework==3.11.0
future==0.18.2
google-api-core==1.22.1
google-api-python-client==1.10.0
google-auth==1.20.1
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.1
googleapis-common-protos==1.51.0
graphqlclient==0.2.4
gunicorn==20.0.4
hiredis==1.0.1
html5lib==1.0.1
httplib2==0.17.3
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.6.0
incremental==17.5.0
intervaltree==3.0.2
ipaddr==2.2.0
JSON-log-formatter==0.3.0
kombu==4.6.11
lockfile==0.12.2
Markdown==3.2.2
more-itertools==8.2.0
msgpack==0.6.2
msgpack-python==0.5.6
multidict==4.7.6
mysql==0.0.2
mysqlclient==1.4.6
numpy==1.19.1
oauth2client==4.1.3
oauthlib==3.1.0
packaging==20.3
paho-mqtt==1.5.0
pandas==1.1.1
pep517==0.8.2
pluggy==0.13.1
progress==1.5
protobuf==3.13.0
psycopg2-binary==2.8.5
py==1.8.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.5.0
pycparser==2.20
pygsheets==2.0.3.1
PyHamcrest==2.0.2
PyJWT==1.7.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pytest==5.4.2
pytest-django==3.9.0
python-dateutil==2.8.1
python3-openid==3.1.0
pytoml==0.1.21
pytz==2020.1
redis==2.10.6
requests==2.22.0
requests-oauthlib==1.3.0
retrying==1.3.3
rsa==4.0
service-identity==18.1.0
six==1.14.0
social-auth-app-django==3.1.0
social-auth-core==3.3.3
sortedcontainers==2.2.2
sqlparse==0.3.1
stripe==2.54.0
tabulate==0.8.7
tenacity==6.2.0
Twisted==20.3.0
txaio==20.4.1
uritemplate==3.0.1
urllib3==1.25.9
vine==1.3.0
wcwidth==0.1.9
webencodings==0.5.1
websockets==8.1
whitenoise==5.0.1
xlrd==1.2.0
yarl==1.4.2
zipp==3.1.0
zope.interface==5.1.0
What needs to change in my configuration, if anything, to get the new version of Redis working?
Thanks in advance.
This is an old question but for anyone else facing similar connection errors on Heroku make you reference the docs:
As of Redis version 6, Hobby plans support both TLS and unencrypted
connections. Production plans require TLS connections. You must enable
TLS in your Redis client’s configuration in order to connect to a
Redis 6 database. For more information, see Heroku Data for Redis
Security and Compliance.
So in other words make sure you are setting up TLS correctly if your Redis instance is on a production plan.
If this is the case, since you are using Python/Django you should reference: https://devcenter.heroku.com/articles/connecting-heroku-redis#connecting-in-python
Pretty new to development - I am seeing the following error when I deploying my .py script. I have referenced my creds as follows:
reference:
# Postgresql initialization
connection = psycopg2.connect(
host= "Host",
dbname= "Database",
port= "Port",
user= "User",
password= "Password")
The error message:
error:
2018-09-16T18:19:45.000000+00:00 app[api]: Build succeeded
2018-09-16T18:19:46.501301+00:00 heroku[web.1]: Process exited with status 1
2018-09-16T18:19:46.521042+00:00 heroku[web.1]: State changed from starting to crashed
2018-09-16T18:19:46.408361+00:00 app[web.1]: /app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
2018-09-16T18:19:46.408378+00:00 app[web.1]: """)
2018-09-16T18:19:46.409434+00:00 app[web.1]: Traceback (most recent call last):
2018-09-16T18:19:46.409439+00:00 app[web.1]: File "app.py", line 24, in <module>
2018-09-16T18:19:46.409629+00:00 app[web.1]: password= "Password")
2018-09-16T18:19:46.409631+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
2018-09-16T18:19:46.409824+00:00 app[web.1]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2018-09-16T18:19:46.409851+00:00 app[web.1]: psycopg2.OperationalError: invalid port number: "Port"
2018-09-16T18:19:46.409855+00:00 app[web.1]:
Can anyone help? Why is my port throwing issue?
In the provided code you're passing strings that have no bearing on establishing a connection. Instead, you should be reading the url, password, port, etc out of the environment. In fact, Heroku puts all of these together into a connection string on your behalf. Try this (heroku documentation):
DATABASE_URL = os.environ['DATABASE_URL']
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
I'm trying to build an app in Django that uses Django Channels. I'm deploying on Elastic Beanstalk. My understanding is that the application load balancer supports websockets and I can route websocket traffic to the appropriate port. The websockets work on my localhost:8000. I'm using the free tier Redis Labs for my channel layer.
I followed this tutorial.
https://blog.mangoforbreakfast.com/2017/02/13/django-channels-on-aws-elastic-beanstalk-using-an-alb/#comment-43
Following this tutorial it appears I can get Daphne and the workers running on the correct ports. I SSHed into the EC2 instance to get the following output.
$ sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status
Daphne RUNNING pid 4240, uptime 0:05:51
Worker:Worker_00 RUNNING pid 4242, uptime 0:05:51
Worker:Worker_01 RUNNING pid 4243, uptime 0:05:51
Worker:Worker_02 RUNNING pid 4244, uptime 0:05:51
Worker:Worker_03 RUNNING pid 4245, uptime 0:05:51
httpd RUNNING pid 4248, uptime 0:05:51
My daphne.out.log and workers.out.log look fine.
$ cat daphne.out.log
2017-07-20 21:41:43,693 INFO Starting server at tcp:port=5000:interface=0.0.0.0, channel layer mysite.asgi:channel_layer.
2017-07-20 21:41:43,693 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2017-07-20 21:41:43,694 INFO Using busy-loop synchronous mode on channel layer
2017-07-20 21:41:43,694 INFO Listening on endpoint tcp:port=5000:interface=0.0.0.0
$ cat workers.out.log
2017-07-20 21:41:44,114 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,120 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,121 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,121 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,126 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,133 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,136 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
From here I updated the security group settings and configured the load balancer and tested my webpage. Instead of /ws/ as the path for rerouting, I used /table/* because those are the pages that need websockets.
If I use /ws/ I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 404
If I use /table/* I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 504
When I changed the security group settings of my load balancer to also allow TCP at port 5000, the worker log changes. With the /ws/ path rule in my load balancer, I now get this in the worker log:
...
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
If I use the /table/* path, I get a very long error in my log.
2017-07-21 01:22:05,270 - ERROR - worker - Error processing message with consumer deal.consumers.ws_connect:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 193, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/worker.py", line 119, in run
consumer(message, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/sessions.py", line 220, in inner
result = func(message, *args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/auth.py", line 71, in inner
message.user = auth.get_user(fake_request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 167, in get_user
user_id = _get_user_session_key(request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 59, in _get_user_session_key
return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 48, in __getitem__
return self._session[key]
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 198, in _get_session
self._session_cache = self.load()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/db.py", line 33, in load
expire_date__gt=timezone.now()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 381, in get
num = len(clone)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 240, in __len__
self._fetch_all()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 1074, in _fetch_all
self._result_cache = list(self.iterator())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 846, in execute_sql
cursor = self.connection.cursor()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 231, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 204, in _cursor
self.ensure_connection()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
Unsurprisingly when I try to use the websockets I get this error:
WebSocket is already in CLOSING or CLOSED state.
Any idea what I'm doing wrong?
Ok so problem was when Daphne runs it doesn't see the environment variables. I added the following code to my supervisord.conf file and it worked:
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && /opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 mysite.asgi:channel_layer"**
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/daphne.out.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && python manage.py runworker"**
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log