I'm creating script that generates specific files using jinja2 as template engine. It creates file I expect, except for the last line. In template I have specified last empty line, but when file is created it does not have that line.
Template looks like this:
# -*- coding: utf-8 -*-
from openerp import fields, models, api
class {{ class_name }}(models.{{ model_type }}):
"""{{ class_docstring }}"""
_{{ def_type }} = '{{ model }}'
# Here is actually empty line. Note comment does not exist on template. It is just empty line.
So in total there are 10 lines defined in this template. But file that is created using this template will only have 9 lines (that last line will not be created).
Is this expected behavior or it should create me that last line as I am expecting?
Here data and methods that handle rendering:
from jinja2 import Environment, FileSystemLoader
PATH = os.path.dirname(os.path.abspath(__file__))
TEMPLATE_ENVIRONMENT = Environment(
autoescape=True,
loader=FileSystemLoader(os.path.join(PATH, 'templates')),
trim_blocks=False)
...
...
#staticmethod
def render_template(t, context):
# For now it only supports standard templates.
template_filename = TEMPLATE_FILES_MAPPING[t]
return TEMPLATE_ENVIRONMENT.get_template(template_filename).render(
context)
The keep_trailing_newline option may be what you're looking for:
By default, Jinja2 also removes trailing newlines. To keep single
trailing newlines, configure Jinja to keep_trailing_newline.
You can add it to the environment:
TEMPLATE_ENVIRONMENT = Environment(
...
keep_trailing_newline=True)
Another option is to finish the template with two newlines and let jinja2 strip one of them:
File contents
File contents
...
# Actual empty line (without this comment) which is kept by jinja2
# Actual empty line (without this comment) which is stripped by jinja2
Related
I am trying to test a number of Python 2.7 classes using unittest.
Here is the exception:
ScannerError: mapping values are not allowed here
in "<unicode string>", line 3, column 32:
... file1_with_path: '../../testdata/concat1.csv'
Here is the example the error message relates to:
class TestConcatTransform(unittest.TestCase):
def setUp(self):
filename1 = os.path.dirname(os.path.realpath(__file__)) + '/../../testdata/concat1.pkl'
self.df1 = pd.read_pickle(filename1)
filename2 = os.path.dirname(os.path.realpath(__file__)) + '/../../testdata/concat2.pkl'
self.df2 = pd.read_pickle(filename2)
self.yamlconfig = u'''
--- !ConcatTransform
file1_with_path: '../../testdata/concat1.csv'
file2_with_path: '../../testdata/concat2.csv'
skip_header_lines: [0]
duplicates: ['%allcolumns']
outtype: 'dataframe'
client: 'testdata'
addcolumn: []
'''
self.testconcat = yaml.load(self.yamlconfig)
What is the the problem?
Something not clear to me is that the directory structure I have is:
app
app/etl
app/tests
The ConcatTransform is in app/etl/concattransform.py and TestConcatTransform is in app/tests. I import ConcatTransform into the TestConcatTransform unittest with this import:
from app.etl import concattransform
How does PyYAML associate that class with the one defined in yamlconfig?
A YAML document can start with a document start marker ---, but that has to be at the beginning of a line, and yours is indented eight positions on the second line of the input. That causes the --- to be interpreted as the beginning of a multi-line plain (i.e. non-quoted) scalar, and within such a scalar you cannot have a : (colon + space). You can only have : in quoted scalars. And if your document does not have a mapping or sequence at the root level, as yours doesn't, the whole document can only consists of a single scalar.
If you want to keep your sources nicely indented like you have now, I recommend you use dedent from textwrap.
The following runs without error:
import ruamel.yaml
from textwrap import dedent
yaml_config = dedent(u'''\
--- !ConcatTransform
file1_with_path: '../../testdata/concat1.csv'
file2_with_path: '../../testdata/concat2.csv'
skip_header_lines: [0]
duplicates: ['%allcolumns']
outtype: 'dataframe'
client: 'testdata'
addcolumn: []
''')
yaml = ruamel.yaml.YAML()
data = yaml.load(yaml_config)
You should get into the habit to put the backslash (\) at the end of your first triple-quotes, so your YAML document. If you do that, your error would have actually indicated line 2 because the document doesn't start with an empty line anymore.
During loading the YAML parser encouncters the tag !ConcatTransform. A constructor for an object is probably registered with the PyYAML loader, associating that tag with the using PyYAML's add_constructor, during the import.
Unfortunately they registered their constructor with the default, non-safe, loader, which is not necessary, they could have registered with the SafeLoader, and thereby not force users to risk problems with non-controlled input.
I am attempting to export data to a plain text file using Django 1.10 (Python 3.5) views/templates. This text file must be OS agnostic in that Windows users ought to have no trouble viewing the file. Unfortunately, when Django renders my template which has \r\n (Windows friendly) line breaks in the file, the line breaks are magically converted into \n (Mac/Linux friendly) line breaks. What gives?
Here's how I'm attempting to render the plain text file:
from django.template import loader, Context
def myview(request):
my_data = get_my_data()
response = HttpResponse(content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename="export.txt"'
template = loader.get_template('export.txt') # <- this file has \r\n line breaks
context = Context({'data': my_data})
response.write(template.render(context))
return response
Upon downloading the exported file using Chrome or Edge in Windows and opening in Notepad, the line breaks aren't respected, and upon viewing the file in Notepad++ (and showing EOL characters), only the \n character is there! Any help would be greatly appreciated :)
I have faced the same issue, unfortunately this seems to happen in Django's template loader/engine somewhere. If you initiate a Template directly without going through get_template() (or render_to_string() and similar methods which uses the same calls), this does not happen. Here is a simplified version of my testcase:
>>> from django import template
>>> t = template.Template('{% for line in dataset %}{{ line.field1 }}\t{{ line.field2 }}\r\n{{ line.field3 }}\t{{ line.field4 }}\r\n{% endfor %}')
>>> dataset = {'field1': 'F1', 'field2': 'F2', 'field3': 'F3', 'field4': 'F4'}
>>> c = template.Context({'dataset': dataset})
>>> t.render(c)
This outputs the template correctly with \r\n in place:
u'F1\tF2\r\nF3\tF4\r\n'
However if I load the same template from a file, using get_template() like you do, it strips them:
>>> from django.template import loader
>>> t = loader.get_template('test.txt')
>>> t.render(c)
u'F1\tF2\nF3\tF4\n'
I tried wading through the Django code in order to identify where this occurs, but with time constraints on my hand I ended up doing the "quick'n'dirty" fix instead:
>>> t = loader.get_template('test.txt')
>>> out = t.render(c).replace('\n', '\r\n')
Beware, this replaces any \n you might have in the actual data fields too, so use with caution....
I have a module that i want to write into. I'm having several problems. One of which locating a string within the file. Currently I open the file, then use a for line in (filename), then do an if to determine if it finds a string, and all of that works. However before (it is commented out now) i tried to determine what position it was at using tell(). However this gave me an incorrect position, giving me 1118 i believe, instead of 660 something. So i determined the position manually to use seek.
However the second problem was, if i write to this file at the position in the file, it just overwrites all the data from thereon. I would want to insert the data instead of overwriting it.
Unless i insert a string equal in character length where i want the write to happen, it will just override most of the if statements and things like that below.
Is there any way to naively do this?
Here is the file i want to write into
# Filename: neo_usercurves.py
# Created By: Gregory Smith
# Description: A script containing a library of user created curves
# Purpose: A library to store names of all the user curves, and deletes curves
# if specified to do so
import os
import maya.cmds as mc
import module_locator
my_path = module_locator.module_path()
def usercurve_lib(fbxfile=None, remove=None):
"""All control/curve objects created by user
Keyword Arguments:
fbxfile -- (string) name of fbx file to import
remove -- (boolean) will remove an entry from the library and delete the
associated fbx file
"""
curves_dict = {
#crvstart
#crvend
}
if remove is None:
return curves_dict
elif not remove:
try:
name = mc.file(curves_dict[fbxfile], typ='FBX', i=1,
iv=True, pmt=False)
return name[0]
except RuntimeError:
return None
else:
try:
os.remove('%s\%s.fbx' %(my_path, fbxfile))
return '%s.fbx' %(fbxfile)
except OSError:
print 'File %s does not exist.' %(fbxfile)
return None
This is the code below that i'm running in a module called neo_curves.py (this is not the complete code, and 'my_path' is just the path of the current directory neo_curves.py is being run in)
def create_entry(self, crv):
"""Exports user curve to user data directory and adds entry into
neo_usercurves.py
Keyword Arguments:
crv -- (PyNode) the object to export
"""
# set settings
mel.eval('FBXExportFileVersion "FBX201400"')
mel.eval('FBXExportInputConnections -v 0')
select(crv)
mc.file('%s\userdat\%s.fbx' %(my_path, str(crv)), force=True, options='',
typ='FBX export', pr=True, es=True)
with open('%s\userdat\\neo_usercurves.py' %(my_path), 'r+') as usercrvs:
for line in usercrvs:
if line.strip() == '#crvstart':
#linepos = usercrvs.tell()
#linepos = int(linepos)
#usercrvs.seek(linepos, 0)
usercrvs.seek(665, 0)
usercrvs.write("\n "+str(crv)+" : '%s\%s' %(my_path, '"+
str(crv)+".fbx')")
break
This will give me this result below:
# Filename: neo_usercurves.py
# Created By: Gregory Smith
# Description: A script containing a library of user created curves
# Purpose: A library to store names of all the user curves, and deletes curves
# if specified to do so
import os
import maya.cmds as mc
import module_locator
my_path = module_locator.module_path()
def usercurve_lib(fbxfile=None, remove=None):
"""All control/curve objects created by user
Keyword Arguments:
fbxfile -- (string) name of fbx file to import
remove -- (boolean) will remove an entry from the library and delete the
associated fbx file
"""
curves_dict = {
#crvstart
loop_crv : '%s\%s' %(my_path, 'loop_crv.fbx') return curves_dict
elif not remove:
try:
name = mc.file(curves_dict[fbxfile], typ='FBX', i=1,
iv=True, pmt=False)
return name[0]
except RuntimeError:
return None
else:
try:
os.remove('%s\%s.fbx' %(my_path, fbxfile))
return '%s.fbx' %(fbxfile)
except OSError:
print 'File %s does not exist.' %(fbxfile)
return None
In short: on most operating systems you can not insert into files without rewriting if the lengths are not the same.
Have a look at a long discussion here: Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)
I was trying to convert the content scraped into a list for data manipulation, but got the following error: TypeError: 'NoneType' object is not callable
#! /usr/bin/python
from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import os
import re
# Copy all of the content from the provided web page
webpage = urlopen("http://www.optionstrategist.com/calculators/free-volatility- data").read()
# Grab everything that lies between the title tags using a REGEX
preBegin = webpage.find('<pre>') # Locate the pre provided
preEnd = webpage.find('</pre>') # Locate the /pre provided
# Copy the content between the pre tags
voltable = webpage[preBegin:preEnd]
# Pass the content to the Beautiful Soup Module
raw_data = BeautifulSoup(voltable).splitline()
The code is very simple. This is the code for BeautifulSoup4:
# Find all <pre> tag in the HTML page
preTags = webpage.find_all('pre')
for tag in preTags:
# Get the text inside the tag
print(tag.get_text())
Reference:
find_all()
Kinds of filters to put into name field of find()/findall()
get_text()
To get the text from the first pre element:
#!/usr/bin/env python
from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup
url = "http://www.optionstrategist.com/calculators/free-volatility-data"
soup = BeautifulSoup(urlopen(url))
print soup.pre.string
To extract lines with data:
from itertools import dropwhile
lines = soup.pre.string.splitlines()
# drop lines before the data table header
lines = dropwhile(lambda line: not line.startswith("Symbol"), lines)
# extract lines with data
lines = (line for line in lines if '%ile' in line)
Now each line contains data in a fixed-column format. You could use slicing and/or regex to parse/validate individual fields in each row.
I have problems localizing a django-nonrel project, which is deployed to GAE. Because of GAE I have to put everything into my project folder, so it looks like something like this
project
+ django
+ dbindexer
+ registration
+ myapp
...
+ locale
+ templates
I have strings to localize in templates directory, and in the myapp directory.
When I run python manage.py makemessages -l en --ignore django\* from the project dir it crawl through all the directories of the project, including django, so I get a quite big po file. My strings from the templates are there, along with all of the strings from django directory.
after --ignore ( or just -i ) I tried to pu django django/* , but nothing changed.
Any ideas?
./manage.py help makemessages
-i PATTERN, --ignore=PATTERN
Ignore files or directories matching this glob-style
pattern. Use multiple times to ignore more.
I have just tested it, and this command successfully ignored my application:
./manage.py makemessages -l da -i "django*"
But beware that before you test it, you should delete the old .po file, as I think it will not automatically remove the translation lines from your previous makemessages execution.
The problem is with the pattern - maybe the shell was expanding it for you.
In general - it is good to avoid path separators (whether / or \) in the pattern.
If you need to always pass specific options to the makemessages command, you could consider your own wrapper, like this one, which I use myself:
from django.conf import settings
from django.core.management.base import BaseCommand
from django.core.management import call_command
class Command(BaseCommand):
help = "Scan i18n messages without going into externals."
def handle(self, *args, **options):
call_command('makemessages',
all=True,
extensions=['html', 'inc'],
ignore_patterns=['externals*'])
This saves you typing, and gives a common entry point for scanning messages across the project (your translator colleague will not destroy translations by missing out some parameter).
Don't delete the old .po file, once you have cleared it from the totally unwanted (i.e. - those from 'django' directory) messages. This allows gettext to recycle old unused messages, once they are used again (or simmilar ones, which will be marked as #, fuzzy.
Edit - as mt4x noted - the wrapper above doesn't allow for passing the options to the wrapped command. This is easy to fix:
from django.core.management import call_command
from django.core.management.commands.makemessages import (
Command as MakeMessagesCommand
)
class Command(MakeMessagesCommand):
help = "Scan i18n messages without going into externals."
def handle(self, *args, **options):
options['all'] = True
options['extensions'] = ['html', 'inc']
if 'ignore_patterns' not in options:
options['ignore_patterns'] = []
options['ignore_patterns'] += ['externals*']
call_command('makemessages', **options)
Thus - you can fix what needs to be fixed, and flex the rest.
And this needs not be blind override like above, but also some conditional edit of the parameters passed to the command - appending something to a list or only adding it when it's missing.