What I try to do
I'm trying to train a convolutional neural network (CNN) for image-detection using Keras with Tensorflow-GPU as backend in Python 2.7, since I need to use it with ROS kinetic, which only supports Python 2.7 (and not 3.5). My model is a Sequential (code see down below).
What I am using
Pycharm-Community 2018.1.4
Keras 2.2.0
Tensorflow-GPU 1.8.0
60000 input images, 100x100 pixels (3 channels), 3 classes ("train_set")
20000 evaluation images, same dimensions ("evaluation_set")
What works
When training the model on my train_set using Python 3.5 and evaluate it using Python 3.5 it works perfectly fine (train_accuracy: 0.99874, evaluation_accuracy: 0.9993).
What does not work
When training the model on my train_set using Python 2.7 and evaluate it using Python 2.7 my accuracy drops drastically (train_accuracy: 0.695, evaluation_accuracy: 0.543), which is not much more than guessing on 3 classes (which would be 0.3333).
I also tried training the model in Python 3.5 and load it in Python 2.7 for evaluation and prediction, but the results are as worse as before.
In all cases I am using the exact same code:
def build_model(training_input):
model = Sequential()
model.add(Conv2D(32, (3, 3)) # Add some layers
model.compile(optimizer='RMSprop', loss='categorical_crossentropy', metrics=['accuracy'])
def train():
input_training = np.array(input_training_list) # input_training_list is a list containing the imagedata
labels_training = np.array(label_training_list) # label_training_list is a list containing the labels corresponding to the imagedata
model = create_model(input_training)
history = model.fit(input_training, labels_training, epochs=10, shuffle=True, batch_size=20)
model.save(model_directory + "my_model.h5")
def evaluation():
input_evaluation = np.array(input_evaluation_list)
labels_evaluation = np.array(label_evaluation_list)
model = load_model(model_directory + "my_model.h5")
loss, acc = model.evaluate(input_evaluation, labels_evaluation, batch_size=1)
I heard that many people have issues loading the same model in different Sessions(), using different computers or different versions of Python. But here the same architecture gives completely different results in both Python versions.
I found the solution for my problem (thanks to user1735003 for the tip regarding my data).
The reason for my bad results was a wrong data-implementation due to the differences regarding Python 2.x and Python 3.x. When implementing my image-data I use
for i in range(len(evaluation_files)):
input_evaluation = np.divide(ndimage.imread(evaluation_files[i]), 255)
But here is the Problem: In Python 3.x this works perfectly fine, since a division of two integers results in a float, but in Python 2.x the result is an integer as well, so my input_evalution list only consists of zeros. I need to divide by 255.0 (make the result a float).
input_evaluation = np.divide(ndimage.imread(evaluation_files[i]), 255.0)
Alternatively import division from __future__ to get floating point results from integer divisions already in python 2.
from __future__ import division
There are some mayor differences when using Python 2.x or Python 3.x which you can see very nicely for example on http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html .
I also managed training my model on Python 3.5, save it using model.save('my_model') and load it in Python 2.7 using keras.models.load_model('my_model'), which works perfectly fine.
One would also easily just save the weights using model.save_weights('my_weights'), create a new model of the same architecture(!) in Python 2.7 and load the weights into that model using model.load_weights('my_weights'), but since just loading the model itself works perfectly fine that way is much easier.
I have been trying to understand how to properly call a function from inside a subpackage in python. I wanted to be able to call the function the way I call, for example, function isfile from os package, with os.path.isfile(). I made a test package with a structure like this:
sandbox/
-- __init__.py
-- acid.py
-- pack1/
-- __init__.py
-- fly/
-- __init__.py
-- plane.py
-- by/
-- pack2/
There are only two modules there, acid.py and plane.py. Both of them contain just a function, e.g. plane.py is
"""plane module"""
def plane(x):
x=x+4
return x
To use the function in my test.py code, I put
import pack1
in sandbox/__init__.py
import fly
in sandbox/pack1/__init__.py, and
from plane import plane
in sandbox/pack1/fly/__init__.py
The test code was then:
import sandbox
print sandbox.pack1.fly.plane(3)
Is this the right way to import a function from a subpackage, or I'm misunderstanding things?
What you did certainly works, although there are several worthwhile changes to make.
First, a note about importing from packages: importing a module is semantically distinct from accessing something in a module, even though from xml import sax and from datetime import date are syntactically equivalent. It's impossible to import only part of a module, so that
import datetime
datetime.date.today() # OK: date is a class
is guaranteed to work. However, it is possible to import a package but not the modules it contains. This is a good thing for efficiency, but it does mean that
import xml
xml.sax.parse(...) # AttributeError: 'module' object has no attribute 'sax'
is an error. Unfortunately, such errors often go uncaught because some other code has already imported sax, making it available to any other code that imports xml. (The word "from" in from xml import sax is referring to the complete package on disk, not the module object xml — on which it stores the new module as an attribute!)
As an aside, note that your example of os.path is an abberation: writing
import os
os.path.isfile(...)
works, but only because os.path is not actually a module but an alias for one of posixpath, ntpath, etc. (It then gets installed in sys.modules to allow import os.path as if it were a normal module.)
As such, for a package there is a set of public modules that the user must be aware of (because they must be imported by name to be available); the rest are internal modules that the package loads itself when necessary. If a package contains no public modules, it is irrelevant to the user that it is a package (for example, importlib with its one public function is actually implemented as a package for forward compatibility with Python 3).
Now for the suggestions:
Implicit relative imports are deprecated: write from . import pack1 instead of just import pack1 in sandbox/__init__.py, for instance.
The from plane import plane (or from .plane import plane, following the above point) is problematic because it overwrites the reference to the module plane.py with a reference to the function. Instead:
Define the user-visible entry points (like plane()) directly in their package's __init__.py, importing internal functions from private modules as needed, or
Rename the module (to plane_module.py or so) to avoid the collision.
However, it's not generally a good idea to have a package automatically import its public modules anyway: it forces the client to pay for loading unused parts of the package, and it blurs the distinction between public modules and simple nested names. Instead, write client code like
import sandbox.pack1.fly
print sandbox.pack1.fly.plane(3) # the same line you had
or
from sandbox.pack1 import fly
print fly.plane(3)
if you want to avoid repeating sandbox.pack1.
It is often suggested that __init__.py be entirely empty, and in Python 3.3 it became possible to define packages without any __init__.py at all (which by necessity "makes it empty"). This policy does more than the "no automatic import" suggestion in that it precludes loading things from private modules (like plane).
There are sometimes good reasons to have a non-empty __init__.py; for example, it allows reorganzing an existing module into a package without breaking its clients. I personally see no reason to especially restrict its contents; for further discussion see What is __init__.py for?.
The init.py file makes a folder as a package so that you can import it to python prompt. If you just want to call the function "plane" from your file plane.py, add the absolute path of plane.py file to your PYTHONPATH and call the funtion as shown below.
>>> import sys
>>> sys.path.append("D:\\sandbox\\pack1\\fly")
>>> import plane
>>> print plane.__doc__
plane module
>>> print plane.plane(3)
7
If you want all the packages under "sandbox" folder to be used in your script just add the absolute path of "sandbox" folder to your PYTHONPATH and call the funtion as shown below.
>>> import sys
>>> sys.path.append("D:\\")
>>> from sandbox.pack1.fly import plane
>>> print plane.plane(3)
7
You can also import the "plane.py" module and call the function "plane" as shown below:
>>> import sys
>>> sys.path.append("D:\\")
>>> import sandbox.pack1.fly.plane
>>> print sandbox.pack1.fly.plane.plane(3)
7
I'm trying to use type hinting in python 2.7 as described here.
I have a function that expects a callable (a function) with a specific signature as a parameter and I can't figure out how to annotate it.
I've tried
def set_function(self, function):
# type: ((int) -> None) -> None
But PyCharm shows an expected ')' and unexpected tokens errors
I can't seem to find any documentation for this...
The correct way to document a callable within Pycharm (or within any other tool that understands PEP 484 type hints) is like so:
from typing import Callable
def set_function(self, function):
# type: (Callable[[int], None]) -> None
...
Since you're using Python 2, you'll need to install the typing module from PyPi, if you haven't already. (typing was added to Python's standard library in 3.5, the module on PyPi is a backport).
You can find more information on using the typing module in Python's documentation, and within the documentation for mypy.
(If you're not aware, mypy is a command line tool that also understands PEP 484 type hints and will also statically analyze and typecheck your code. It is an independent effort from Pycharm's built-in type checker. Since both Pycharm and mypy use PEP 484 types, mypy's documentation is often a good place to start looking to learn more about using type hints.)
I would like to use scipy's integrate.simps to get a integrated function from a data sample.
Data sample is getting calculated inside For loop (variable fx).
In the same step I store integrated values in the variable intfx.
import numpy as np
import pylab as pl
from scipy import integrate
t=np.arange(0,10.01,0.01)
fx=[]
intfx=[]
counter=0
for i in t:
counter+=1
fx.append(np.sin(i))
intfx.append(fx[-1]+integrate.trapz(fx[-2:], dx=0.1))
pl.plot(t,fx)
pl.plot(t,intfx)
pl.show()
On plots it can be seen that the two functions are very similar, which is obviously wrong.
Can anyone help me with this?
Made a stupid mistake.
Line:
intfx.append(fx[-1]+integrate.trapz(fx[-2:], dx=0.1))
should look like this:
intfx.append(intfx[-1]+integrate.trapz(fx[-2:], dx=0.1))
I want to programmatically edit python source code. Basically I want to read a .py file, generate the AST, and then write back the modified python source code (i.e. another .py file).
There are ways to parse/compile python source code using standard python modules, such as ast or compiler. However, I don't think any of them support ways to modify the source code (e.g. delete this function declaration) and then write back the modifying python source code.
UPDATE: The reason I want to do this is I'd like to write a Mutation testing library for python, mostly by deleting statements / expressions, rerunning tests and seeing what breaks.
Pythoscope does this to the test cases it automatically generates as does the 2to3 tool for python 2.6 (it converts python 2.x source into python 3.x source).
Both these tools uses the lib2to3 library which is an implementation of the python parser/compiler machinery that can preserve comments in source when it's round tripped from source -> AST -> source.
The rope project may meet your needs if you want to do more refactoring like transforms.
The ast module is your other option, and there's an older example of how to "unparse" syntax trees back into code (using the parser module). But the ast module is more useful when doing an AST transform on code that is then transformed into a code object.
The redbaron project also may be a good fit (ht Xavier Combelle)
The builtin ast module doesn't seem to have a method to convert back to source. However, the codegen module here provides a pretty printer for the ast that would enable you do do so.
eg.
import ast
import codegen
expr="""
def foo():
print("hello world")
"""
p=ast.parse(expr)
p.body[0].body = [ ast.parse("return 42").body[0] ] # Replace function body with "return 42"
print(codegen.to_source(p))
This will print:
def foo():
return 42
Note that you may lose the exact formatting and comments, as these are not preserved.
However, you may not need to. If all you require is to execute the replaced AST, you can do so simply by calling compile() on the ast, and execing the resulting code object.
Took a while, but Python 3.9 has this:
https://docs.python.org/3.9/whatsnew/3.9.html#ast
https://docs.python.org/3.9/library/ast.html#ast.unparse
ast.unparse(ast_obj)
Unparse an ast.AST object and generate a string with code that would produce an equivalent ast.AST object if parsed back with ast.parse().
In a different answer I suggested using the astor package, but I have since found a more up-to-date AST un-parsing package called astunparse:
>>> import ast
>>> import astunparse
>>> print(astunparse.unparse(ast.parse('def foo(x): return 2 * x')))
def foo(x):
return (2 * x)
I have tested this on Python 3.5.
You might not need to re-generate source code. That's a bit dangerous for me to say, of course, since you have not actually explained why you think you need to generate a .py file full of code; but:
If you want to generate a .py file that people will actually use, maybe so that they can fill out a form and get a useful .py file to insert into their project, then you don't want to change it into an AST and back because you'll lose all formatting (think of the blank lines that make Python so readable by grouping related sets of lines together) (ast nodes have lineno and col_offset attributes) comments. Instead, you'll probably want to use a templating engine (the Django template language, for example, is designed to make templating even text files easy) to customize the .py file, or else use Rick Copeland's MetaPython extension.
If you are trying to make a change during compilation of a module, note that you don't have to go all the way back to text; you can just compile the AST directly instead of turning it back into a .py file.
But in almost any and every case, you are probably trying to do something dynamic that a language like Python actually makes very easy, without writing new .py files! If you expand your question to let us know what you actually want to accomplish, new .py files will probably not be involved in the answer at all; I have seen hundreds of Python projects doing hundreds of real-world things, and not a single one of them needed to ever writer a .py file. So, I must admit, I'm a bit of a skeptic that you've found the first good use-case. :-)
Update: now that you've explained what you're trying to do, I'd be tempted to just operate on the AST anyway. You will want to mutate by removing, not lines of a file (which could result in half-statements that simply die with a SyntaxError), but whole statements — and what better place to do that than in the AST?
Parsing and modifying the code structure is certainly possible with the help of ast module and I will show it in an example in a moment. However, writing back the modified source code is not possible with ast module alone. There are other modules available for this job such as one here.
NOTE: Example below can be treated as an introductory tutorial on the usage of ast module but a more comprehensive guide on using ast module is available here at Green Tree snakes tutorial and official documentation on ast module.
Introduction to ast:
>>> import ast
>>> tree = ast.parse("print 'Hello Python!!'")
>>> exec(compile(tree, filename="<ast>", mode="exec"))
Hello Python!!
You can parse the python code (represented in string) by simply calling the API ast.parse(). This returns the handle to Abstract Syntax Tree (AST) structure. Interestingly you can compile back this structure and execute it as shown above.
Another very useful API is ast.dump() which dumps the whole AST in a string form. It can be used to inspect the tree structure and is very helpful in debugging. For example,
On Python 2.7:
>>> import ast
>>> tree = ast.parse("print 'Hello Python!!'")
>>> ast.dump(tree)
"Module(body=[Print(dest=None, values=[Str(s='Hello Python!!')], nl=True)])"
On Python 3.5:
>>> import ast
>>> tree = ast.parse("print ('Hello Python!!')")
>>> ast.dump(tree)
"Module(body=[Expr(value=Call(func=Name(id='print', ctx=Load()), args=[Str(s='Hello Python!!')], keywords=[]))])"
Notice the difference in syntax for print statement in Python 2.7 vs. Python 3.5 and the difference in type of AST node in respective trees.
How to modify code using ast:
Now, let's a have a look at an example of modification of python code by ast module. The main tool for modifying AST structure is ast.NodeTransformer class. Whenever one needs to modify the AST, he/she needs to subclass from it and write Node Transformation(s) accordingly.
For our example, let's try to write a simple utility which transforms the Python 2 , print statements to Python 3 function calls.
Print statement to Fun call converter utility: print2to3.py:
#!/usr/bin/env python
'''
This utility converts the python (2.7) statements to Python 3 alike function calls before running the code.
USAGE:
python print2to3.py <filename>
'''
import ast
import sys
class P2to3(ast.NodeTransformer):
def visit_Print(self, node):
new_node = ast.Expr(value=ast.Call(func=ast.Name(id='print', ctx=ast.Load()),
args=node.values,
keywords=[], starargs=None, kwargs=None))
ast.copy_location(new_node, node)
return new_node
def main(filename=None):
if not filename:
return
with open(filename, 'r') as fp:
data = fp.readlines()
data = ''.join(data)
tree = ast.parse(data)
print "Converting python 2 print statements to Python 3 function calls"
print "-" * 35
P2to3().visit(tree)
ast.fix_missing_locations(tree)
# print ast.dump(tree)
exec(compile(tree, filename="p23", mode="exec"))
if __name__ == '__main__':
if len(sys.argv) <=1:
print ("\nUSAGE:\n\t print2to3.py <filename>")
sys.exit(1)
else:
main(sys.argv[1])
This utility can be tried on small example file, such as one below, and it should work fine.
Test Input file : py2.py
class A(object):
def __init__(self):
pass
def good():
print "I am good"
main = good
if __name__ == '__main__':
print "I am in main"
main()
Please note that above transformation is only for ast tutorial purpose and in real case scenario one will have to look at all different scenarios such as print " x is %s" % ("Hello Python").
If you are looking at this in 2019, then you can use this libcst
package. It has syntax similar to ast. This works like a charm, and preserve the code structure. It's basically helpful for the project where you have to preserve comments, whitespace, newline etc.
If you don't need to care about the preserving comments, whitespace and others, then the combination of ast and astor works well.
I've created recently quite stable (core is really well tested) and extensible piece of code which generates code from ast tree: https://github.com/paluh/code-formatter .
I'm using my project as a base for a small vim plugin (which I'm using every day), so my goal is to generate really nice and readable python code.
P.S.
I've tried to extend codegen but it's architecture is based on ast.NodeVisitor interface, so formatters (visitor_ methods) are just functions. I've found this structure quite limiting and hard to optimize (in case of long and nested expressions it's easier to keep objects tree and cache some partial results - in other way you can hit exponential complexity if you want to search for best layout). BUT codegen as every piece of mitsuhiko's work (which I've read) is very well written and concise.
One of the other answers recommends codegen, which seems to have been superceded by astor. The version of astor on PyPI (version 0.5 as of this writing) seems to be a little outdated as well, so you can install the development version of astor as follows.
pip install git+https://github.com/berkerpeksag/astor.git#egg=astor
Then you can use astor.to_source to convert a Python AST to human-readable Python source code:
>>> import ast
>>> import astor
>>> print(astor.to_source(ast.parse('def foo(x): return 2 * x')))
def foo(x):
return 2 * x
I have tested this on Python 3.5.
Unfortunately none of the answers above actually met both of these conditions
Preserve the syntactical integrity for the surrounding source code (e.g keeping comments, other sorts of formatting for the rest of the code)
Actually use AST (not CST).
I've recently written a small toolkit to do pure AST based refactorings, called refactor. For example if you want to replace all placeholders with 42, you can simply write a rule like this;
class Replace(Rule):
def match(self, node):
assert isinstance(node, ast.Name)
assert node.id == 'placeholder'
replacement = ast.Constant(42)
return ReplacementAction(node, replacement)
And it will find all acceptable nodes, replace them with the new nodes and generate the final form;
--- test_file.py
+++ test_file.py
## -1,11 +1,11 ##
def main():
- print(placeholder * 3 + 2)
- print(2 + placeholder + 3)
+ print(42 * 3 + 2)
+ print(2 + 42 + 3)
# some commments
- placeholder # maybe other comments
+ 42 # maybe other comments
if something:
other_thing
- print(placeholder)
+ print(42)
if __name__ == "__main__":
main()
We had a similar need, which wasn't solved by other answers here. So we created a library for this, ASTTokens, which takes an AST tree produced with the ast or astroid modules, and marks it with the ranges of text in the original source code.
It doesn't do modifications of code directly, but that's not hard to add on top, since it does tell you the range of text you need to modify.
For example, this wraps a function call in WRAP(...), preserving comments and everything else:
example = """
def foo(): # Test
'''My func'''
log("hello world") # Print
"""
import ast, asttokens
atok = asttokens.ASTTokens(example, parse=True)
call = next(n for n in ast.walk(atok.tree) if isinstance(n, ast.Call))
start, end = atok.get_text_range(call)
print(atok.text[:start] + ('WRAP(%s)' % atok.text[start:end]) + atok.text[end:])
Produces:
def foo(): # Test
'''My func'''
WRAP(log("hello world")) # Print
Hope this helps!
A Program Transformation System is a tool that parses source text, builds ASTs, allows you to modify them using source-to-source transformations ("if you see this pattern, replace it by that pattern"). Such tools are ideal for doing mutation of existing source codes, which are just "if you see this pattern, replace by a pattern variant".
Of course, you need a program transformation engine that can parse the language of interest to you, and still do the pattern-directed transformations. Our DMS Software Reengineering Toolkit is a system that can do that, and handles Python, and a variety of other languages.
See this SO answer for an example of a DMS-parsed AST for Python capturing comments accurately. DMS can make changes to the AST, and regenerate valid text, including the comments. You can ask it to prettyprint the AST, using its own formatting conventions (you can changes these), or do "fidelity printing", which uses the original line and column information to maximally preserve the original layout (some change in layout where new code is inserted is unavoidable).
To implement a "mutation" rule for Python with DMS, you could write the following:
rule mutate_addition(s:sum, p:product):sum->sum =
" \s + \p " -> " \s - \p"
if mutate_this_place(s);
This rule replace "+" with "-" in a syntactically correct way; it operates on the AST and thus won't touch strings or comments that happen to look right. The extra condition on "mutate_this_place" is to let you control how often this occurs; you don't want to mutate every place in the program.
You'd obviously want a bunch more rules like this that detect various code structures, and replace them by the mutated versions. DMS is happy to apply a set of rules. The mutated AST is then prettyprinted.
I used to use baron for this, but have now switched to parso because it's up to date with modern python. It works great.
I also needed this for a mutation tester. It's really quite simple to make one with parso, check out my code at https://github.com/boxed/mutmut