Why does this code not produce a log that is readable by tensorboard? - tensorboard

Using Python (3.6) / Jupyter (5.7.8) on WIndows 10 .. I have tried many simple examples of trying to generate log files for tensorboard including this:
logs_base_dir = "C:/tensorlogs"
%load_ext tensorboard.notebook
# %tensorboard --port=6006 --logdir {logs_base_dir}
os.makedirs(logs_base_dir, exist_ok=True)
%tensorboard --port=6008 --logdir {logs_base_dir}
a = tf.constant([10])
b = tf.constant([20])
c = tf.add(a,b)
with tf.Session() as sess:
# or creating the writer inside the session
writer = tf.summary.FileWriter(logs_base_dir, sess.graph)
print(sess.run(c))
writer.close()
sess.close()
I see a logfile created in the 'C:/tensorlogs' folder called: 'events.out.tfevents.1563219145.DESKTOP-5HG12IB'
I can also see tensorboard running in a separate browser window. On the 'scaler' page it lists the correct source folder 'C:/tensorflow'.
But tensorboard indicate that no session is running.
What am I doing wrong?
Is there any way of checking that log file to see if it is legitimate?
Browser window showing tensorboard running - but not finding log file
(Also shows nothing on graph window)
Update: I tried to run tensorboard from the command line using:
tensorboard --logdir='c:/tensorlogs' --port=6006
It has the same behaviour .. page shows in browser with no data. Terminal displays the following messages:
TensorBoard 1.13.1 at http://DESKTOP-5HG12IB:6006 (Press CTRL+C to quit)
I0716 06:56:05.074265 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:05] "[37mGET /data/experiments HTTP/1.1[0m" 200 -
I0716 06:56:05.086722 8400 _internal.py:122] ::1 - - [16/Jul/2019 06:56:05] "[37mGET /data/environment HTTP/1.1[0m" 200 -
I0716 06:56:05.088708 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:05] "[37mGET /data/plugins_listing HTTP/1.1[0m" 200 -
I0716 06:56:05.090195 8400 _internal.py:122] ::1 - - [16/Jul/2019 06:56:05] "[37mGET /data/runs HTTP/1.1[0m" 200 -
I0716 06:56:05.099200 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:05] "[37mGET /data/plugin/scalars/tags HTTP/1.1[0m" 200 -
I0716 06:56:13.198990 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:13] "[37mGET / HTTP/1.1[0m" 200 -
I0716 06:56:14.169489 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /tf-interactive-inference-dashboard/editedexample.png HTTP/1.1[0m" 200 -
I0716 06:56:14.170989 8400 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /tf-interactive-inference-dashboard/distance.png HTTP/1.1[0m" 200 -
I0716 06:56:14.174491 3780 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /tf-interactive-inference-dashboard/pdplots.png HTTP/1.1[0m" 200 -
I0716 06:56:14.176498 12340 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /tf-interactive-inference-dashboard/explorecounterfactuals.png HTTP/1.1[0m" 200 -
I0716 06:56:14.281985 12876 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /data/experiments HTTP/1.1[0m" 200 -
I0716 06:56:14.282483 8400 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /data/runs HTTP/1.1[0m" 200 -
I0716 06:56:14.282483 12340 _internal.py:122] ::1 - - [16/Jul/2019 06:56:14] "[37mGET /data/environment HTTP/1.1[0m" 200 -
.. and lots more ...

You might not have the basics about tensorboard so let me explain a bit.
You created a graph, and logged it to a file. The only thing your log file will contain is a graph with a single operator (Add) and two inputs.
Logging scalars is often done to track your loss function when training neural networks, this seems like a decent guide.
EDIT
1) Are you sure tensorboard is getting your path correctly? Can you go to your log folder in the terminal and run tensorboard --logdir ./ from there?
2) try using "", I tried using the following commands:
tensorboard --logdir ./
tensorboard --logdir="./"
tensorboard --logdir='./'
The bottom one gave an error in browser displaying nothing, other 2 worked fine. The ./ should be replacable with any folder, I just prefer to go into my folder in terminal on windows, to prevent any path from breaking
3) Have you played around with tensorflow installations, is your tensorboard perhaps not fully compatible with your tensorflow? Have you tried uninstalling every tensorboard / tensorflow package, and installing them again? These are my versions:
tensorboard==1.13.1
tensorflow==1.13.1

T.Kelher above supplied the key information to get this working. I am adding this note just to assist others with the same issue:
I completely re-built my anaconda environment running tensorFlow GPU .. several webpages suggested that this can fix the problem I was seeing.
My original code did indeed produce the correct output file .. the error I was seeing was that either tensorBoard was not running, or it was looking in the wrong folder for the log file
One error message from TensorBoard is particularly misleading. The following message seems to be issued not only when there are no log files .. but when tensorboard is not looking in the correct location and, indeed, can't find that location:
No event files found within logdir '[your named log folder]'
One of the reasons that tensorBoard can't find your folder on Windows is that there is an overloading of syntax. Something like "C:/logfolder" may interpret the "C:" not as a drive letter but as a label for the run. It has been suggested ( https://github.com/tensorflow/tensorflow/issues/7856 ) that you can use "training:c:/logfolder" but I had no success with that.
Finally, following t.Kelher's suggestion I made it work by
(a) Running tensorBoard NOT from inside Jupter, but in a terminal window
(b) cd'ing into the log folder before starting tensorBoard
(c) Using the command:
tensorboard --logdir=./ --port=6006
(d) Rather than using the URL: http://localhost:6006, using the URL..exactly as displayed by tensorBoard on startup .... http://[my machine name]:6006

Related

Letsencrypt certificate renewal issue on Ubuntu 18.04 machine using Apache Server

I am using Apache servers to host my Django (v2.1) app. I've installed Letsencrypt certificate for HTTPS. Now the time of renewal has come and it is giving me some unauthorized access error.
When I run sudo certbot command, I got the following output.
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: noppera.tk
2: www.noppera.tk
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 2
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.noppera.tk
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.noppera.tk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.noppera.tk/.well-known/acme-challenge/U0D416-6zOf7YRW0jAVIG8oiLthmpy_xmewRdUlwrQM [34.240.58.158]: 400
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: www.noppera.tk
Type: unauthorized
Detail: Invalid response from
http://www.noppera.tk/.well-known/acme-challenge/U0D416-6zOf7YRW0jAVIG8oiLthmpy_xmewRdUlwrQM
[34.240.58.158]: 400
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Also if I run this for option 1, I got the same error. (If you want, I can paste that log too)
What I've already tried is the following:
Has already installed django-letsencrypt==3.0.1
Added letsencrypt in settings.py
Added the following line in urls.py url(r'^\.well-known/', include('letsencrypt.urls')),
Right now site is accessible using HTTPS. Can anyone help me out renewing the certificate?
EDIT 1
Option 1 Logs:
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: noppera.tk
2: www.noppera.tk
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for noppera.tk
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. noppera.tk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://noppera.tk/.well-known/acme-challenge/y6dj0WW9qDgZiBnDTmXmA5FTSusyjabeE3dZs5eEGpI [34.240.58.158]: "\n\n<html>\n<head>\n <script src=\"https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js\"></script>\n\n\n\n\n\n\n\n<style>\n /*"
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: noppera.tk
Type: unauthorized
Detail: Invalid response from
https://noppera.tk/.well-known/acme-challenge/y6dj0WW9qDgZiBnDTmXmA5FTSusyjabeE3dZs5eEGpI
[34.240.58.158]: "\n\n<html>\n<head>\n <script
src=\"https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js\"></script>\n\n\n\n\n\n\n\n<style>\n
/*"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Django Log for option 1 (noppera.tk)
Invalid HTTP_HOST header: '{{HOST IP}}'. You may need to add '{{HOST IP}}' to ALLOWED_HOSTS.
Bad Request: /console/login/LoginForm.jsp
Not Found: /.well-known/acme-challenge/WRiDAIe3JPBlZXVWduKBYKrmYKbyS3I2eetsth0YBD0
Django Log for option 2 (www.noppera.tk)
Invalid HTTP_HOST header: 'www.noppera.tk'. You may need to add 'www.noppera.tk' to ALLOWED_HOSTS.
Bad Request: /.well-known/acme-challenge/GTX3_zQ6XPymDUn1WVZ_27vO_XtYxPClBD5uA8Y1nhM
Right now, ALLOWED_HOSTS = ["*"]
EDIT 2
Changed ALLOWED_HOSTS = ["*"] to ALLOWED_HOSTS = ["www.noppera.tk", "*"] for Option 2, but same error.
I have found a solution. Posting to help others.
The problem was lying with the duplicate conf in the apache2/sites-available folder. There were 2 default configs and 2 custom config for my site (for each http and https). So what I did was disable the default config and reload Apache using sudo a2dissite default-ssl.conf and sudo a2dissite 000-default.conf.
After that I executed sudo certbot and it renewed certificates successfully.
Few of the useful resources are below:
https://www.jbarrett.me/blog/items/4/setting-ssl-django-app-lets-encrypt-ubuntu-apache-and-mod_wsgi
https://www.digitalocean.com/community/tutorials/how-to-install-the-apache-web-server-on-ubuntu-18-04#step-5-%E2%80%94-setting-up-virtual-hosts-(recommended)

Odoo v9 Webservice Report without layout

After following the documentation in https://www.odoo.com/documentation/9.0/api_integration.html I have encountered a problem with the generated PDF report.
I call the webservice to generate an invoice report and after rendering the pdf report, it returns without its layout ( located: account.report_invoice )
I do the following to render the report:
url = 'http://{0}:{1}/xmlrpc/2/report'.format(self._connect['host'], self._connect['port'])
sock_print = xmlrpclib.ServerProxy(url)
#Here, the 'render_report' function returns the base64 pdf without the specified layout
result = sock_print.render_report(db_name, uid, pwd, report_name, ids, {'model': 'account.invoice', 'report_type': 'qweb-pdf'})
string_pdf = base64.decodestring(report['result'])
return True, string_pdf
After, the function above is done, I save the file in a directory to check if the file was generated with the correct layout.
So far, the pdf was generated but without its layout for account.report_invoice.
Any ideas on what might be happening or what I might be missing?
Thank you for your time.
[EDIT 1]
2018-09-17 14:34:09,599 30522 INFO ? werkzeug: 127.0.0.1 - - [17/Sep/2018 14:34:09] "GET /web/content/323-c1e807b/report.assets_common.0.css HTTP/1.1" 404 -
2018-09-17 14:34:09,617 30522 INFO ? werkzeug: 127.0.0.1 - - [17/Sep/2018 14:34:09] "GET /web/content/328-9a5a204/report.assets_pdf.0.css HTTP/1.1" 404 -
2018-09-17 14:34:09,879 30522 INFO ? werkzeug: 127.0.0.1 - - [17/Sep/2018 14:34:09] "GET /web/content/328-9a5a204/report.assets_pdf.0.css HTTP/1.1" 404 -
2018-09-17 14:34:09,883 30522 INFO ? werkzeug: 127.0.0.1 - - [17/Sep/2018 14:34:09] "GET /web/content/323-c1e807b/report.assets_common.0.css HTTP/1.1" 404 -
Found this when trying to call via webservice.
When I print the reports directly from odoo interface it's O.K, but via webservice it doesn't recognise its own core css.

Grok pattern assistance

Hi i'm in need of some serious help,
I have logs that i wish to Parse using GROK but the problem i'm having is that they are not always consistent in content or spacing here are some obfuscated examples.
title_access_log:ipaddress1, ipaddress2, ipaddress3 - - [14/Nov/2017:08:30:00 +0000] "GET /url HTTP/1.1" 200 198454 - 153261 - 0000fD5b5OSuS2C7ZdhgwqYufJk:GH809 url
title_access_log:ipaddress1, ipaddress2 - - [14/Nov/2017:08:30:00 +0000] "GET /url HTTP/1.1" 200 2326 - 20482 V22843489635e0e42e864037eccb8ad4857500ea 0000BDzHfUFhjJmcs9R4-CyglGS:GH806 url
title_access_log:ipaddress1, ipaddress2 - - [14/Nov/2017:08:30:00 +0000] "POST /url HTTP/1.1" 200 30031 - 17942 - 0000PjpQluI9BZ0w4EDB9o2fow-:GH809 url
I have managed to make a GROK patterns that pull out up to time and date for logs that contain 2 IPs but i get suck going further or when trying to do logs with 3 ips.
Has anyone got any advice on how to tackle this.
i'm using Graylog is what i'm using to extract data to so i do have the option of using other formats than GROK.

Flask extension, failing to return image path when using sub-path of root

I'm working on adding additional functionality to Flask-Resize, specifically adding a feature that should serve the original file instead of generating a cached file if the size and other parameters are the same as the original.
I have everything worked out in the checks and so forth, and everything works fine if the image is in the RESIZE_ROOT directory but if not, when the image generator detects that it doesn't need to do anything and returns the original file path, jinja2 doesn't seem to fetch the image.
Using an image test_img.jpg with a size of 200x300px, in the RESIZE_ROOT directory works fine:
<img src="{{ 'test_img.jpg'|resize('200') }}"></img>
Output:
http://127.0.0.1:5000/static/images/test_img.jpg
127.0.0.1 - - [23/Oct/2015 03:44:28] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [23/Oct/2015 03:44:28] "GET /static/css/main.css HTTP/1.1" 200 -
127.0.0.1 - - [23/Oct/2015 03:44:28] "GET /static/images/test_img.jpg HTTP/1.1" 200 -
-- test_img.jpg fetched and displayed correctly
However when the image is moved to a sub-directory say ad/test_img.jpg, then the console output doesn't even indicate that it is fetching the image
http://127.0.0.1:5000/static/images/ad/test_img.jpg
127.0.0.1 - - [23/Oct/2015 03:58:42] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [23/Oct/2015 03:58:42] "GET /static/css/main.css HTTP/1.1" 200 -
-- ad/test_img.jpg isn't even being fetched
Putting image generates a valid link to the image, so the file path is getting returned correctly so I have no idea what's going on.
This is the relevant code; generate_image raises a exc.StopImageGeneration exception if it detects that it should not generate an image.
if not os.path.exists(full_cache_path):
try:
generate_image(inpath=original_path, outpath=full_cache_path,
format=format, width=width, height=height,
bgcolor=bgcolor, upscale=upscale, fill=fill,
anchor=anchor, quality=quality,
progressive=progressive,
placeholder_reason=placeholder_reason,
force_cache=force_cache)
except exc.StopImageGeneration:
full_cache_url = unicode(resize_url+image_url)
print (full_cache_url)
return full_cache_url.replace('\\', '/')
And my Flask-Resize initialisation parameters if it matters:
(RESIZE_URL='http://127.0.0.1:5000/static/images/', RESIZE_ROOT='static/images/')
So turns out this has nothing to do with Flask/Jinja2, it's Adblock's fault; it was blocking the images silently.
After I added an exception for localhost and 127.0.0.1 there was no problem.

wso2 api manager how to close http_access.log

I finid that at {wso2am_home}repository/logs/ have logs:
http_access_2013-10-28.log
tm.out wso2-apigw-errors.log
wso2-apigw-service.log
wso2-apigw-trace.log
wso2carbon-trace-messages.log
wso2carbon.log
and I configure all the log4j.properties INFO to OFF. I don't know where to close the http_access.log.
I find when I call 1 time api,it write the http_access.log: gwmanager.apim-wso2.com:8280 - - - "GET /direct/1.0.5 HTTP/1.1" - - "-" "Jakarta Commons-HttpClient/3.1" 128.6.X.X:80 - - - "GET http://128.6.X.X:80 HTTP/1.1" - - "-" "Synapse-HttpComponents-NIO so,as I call api time more and more ,the file is more and ---------- more big.
Do you know how to close the http_access.log?
If you want to disable http access logs in WSO2 products then go to catalina-server.xml which is located {CARBON_HOME}/repository/conf/tomcat directory, and remove the following property
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
prefix="http_access_" suffix=".log"
pattern="combined" />
Please refer this for more details