During a pytest fixture, what is the best way to robustly get the location of a text file for users that may specify different working directories at runtime e.g. I want a person using the cmd line in the test fixture directory find the file as well as an integration server which may work in the project's root. Can I somehow include the text file in a module? What are best practices for including and getting access to non .py files?
I am aware of BASE_DIR = os.path.dirname(os.path.dirname(__file__)), but I am not sure if this will always refer to the same directory given a particular way of running the test suite.
os.path.dirname(os.path.abspath(__file__)) (which is what I think you meant above) worked fine for me so far - it should work as long as Python can figure the path of the file, and with pytest I can't imagine a scenario where that wouldn't be true.
Related
In my C++ application, I have a text file (dataFile.txt) that is installed on the Linux target machine in the following path:
/SoftwareHomeDir/Configuration/Application/dataFile.txt
This file exists on my Rational ClearCase source code environment under the path:
/ProjectName/config/Application/dataFile.txt
I am developping a unitTest in gtest that does following:
Read a specific data from dataFile.txt , if the data does not exist than write it into the file.
1) I am avoiding to create an environment variable to check whether I am in the compilation environment or the target machine. Then add additional test code in the final release. I really want to separate test code from final code.
2) I am not using any IDE (no visual studio, no qt, etc.), just notepad++
3) The compilatio. server is shared (access with a username, however the root folder "/" is shared. Which means that if I create the path "/SoftwareHomeDir/Confiugration/Application/dataFile.txt", it will be visible by all users, and if another user is running his gtest unitTest, he may overwrite my file.
4) In the final code, the path to the dataFile is hard coded, and it is very costly (will take few seconds to run) to implement a filesearch(filename) method to look for the file in the entire hard drive before reading the file.
Question:
I am looking for a solution to unit-test my code in the compilation environment that is using /ProjectName/config/Application/dataFile.txt
The solution to my problem was to combine gmock with gtest as described by the link
https://github.com/google/googletest/blob/master/googlemock/docs/CookBook.md#delegating-calls-to-a-fake
The only modification I made to my code is that instead of defining the path to the configuration data using #define, I created a function getConfigFilePath() that returns the hardcoded path of the configuration file in the installed application. From here, I mocked the class and in my mock, I call a fake getConfigFilePath() that returns, when the real code is executing, the hardcoded path of the config file in the project tree in ClearCase. This is precisely what I was looking for.
After a long time away from an app i wrote in Django and didn't complete, I've come back to it on a new Mac.
I'm struggling to get the code to refer to the apps and the files within them without the explicit path. For instance:
from myproject.app.file import object
Whereas I remember not having to use myproject every time.
Is this something that has changed? I seem to remember being about to add to the path in manage.py which is called every time you run the dev server, but this hasn't worked this time.
sys.path.append /path/to/myproject
Should that fix the issue I'm having?
I started with a simple answer and it grew into more details on how to add subdirectories of your project to python path. Maybe a bit off-topic, but it could be useful to you so I'm pushing the post button anyway.
I usually have a bunch of small re-usable apps of mine I keep inside my project tree, because I don't want them to grow into independent modules. My projet tree will look like this:
manage.py
myproject/apps
myproject/libs
myproject/settings
...
Still, by default, Django only adds the project root to python path. Yet it makes no sense in my opinion to have apps load modules with full path:
from myproject.apps.author.models import Author
from myproject.libs.rest_filters import filters
That's both way too verbose, and it breaks reusability as I only use absolute imports. Not to mention if I someday build an actual python package out of some of the libs, it will break.
So, I took the following steps. I added the relevant folders to the path:
# in manage.py
root = os.path.dirname(__file__)
sys.path.append(os.path.realpath(os.path.join(root, 'myproject', 'apps')))
sys.path.append(os.path.realpath(os.path.join(root, 'myproject', 'libs')))
But you must ensure those packages cannot be loaded from the root of the project, or you will have odd issues as python would load another copy of the module. For instance, isinstance(libs.foo.bar(), myproject.libs.foo.bar) == False
It's not hard though : just remove __init__.py from the folders you add to the path. That will make sure they cannot be descended into from the project.
Also, Django's discover runner will not descend into those paths unless you specify them manually. That may be fine with you (if every module has its own test suite). Or you can extend the runner, so it knows about this: sample code.
I'm new to deploying programs written in C/C++ on Linux and I'm wondering what you'd do in this situation.
I have a binary file (compiled with GNU Make) that needs to read a config file (such as myprogram.conf). But when I write a Makefile to deploy this file to /usr/bin/, where should the config file go? And how does the executable know where it is?
You have endless options, but the best way depends on a couple of things. First, is it a user-specific configuration file, or is it global to all users?
If it's user specific, you could, for example, keep it in ~/.myprogram/config.file and have the program check there. As a service to your users, it's up to you to decide what to do if it's not found -- perhaps copy a default config there from somewhere else, or generate a default, or use hard-coded default options, or display a configuration wizard, or just fail. That's entirely up to you.
If it's global, the traditional place to put it on Linux is in /etc, e.g. /etc/config.file or /etc/myprogram/config.file. See Linux File System Structure. You will generally always have a /etc on Linux. Handling a situation where the file does not exist is the same as above - there's no "right" way to handle that, it's based purely on how convenient you want to make it for a user.
What I usually do for global config files is put them in /etc/wherever on install, have the program default to loading the config file from /etc/wherever, but also give a command line option to override the configuration file (especially useful for testing or other situations).
What I usually do to handle missing config files depends entirely on the application. I'll generally either have hard-coded defaults (if that's appropriate) or simply fail and direct the user to some documentation describing a config file (which I find adequate in situations where my installer installs a config file).
Hope that helps.
It kind of depends on what the configuration parameters are, and whether they are "per system" or "per user" or "per group" or ...
System configurations typically live somewhere in /etc/.... In the same directory that the program lives is a very good place too.
User confgiurations, in the home directory of the user.
Group configurations are the trickiest, as you'll probably need to come up with a scheme where there is a configuration file per "group". /etc/myprog/groups/<groupname>/config or something similar would work.
On Linux, the usual location for configuration files is '/etc', so it is acceptable to deploy a configuration file like /etc/myprog.conf. That requires root privileges however. Other good options include putting a configuration file in the user's home directory, making it something like ~/.myprog.conf or ~/.myprog/.conf to use a folder where you can have several config files, a cache or something else that you want.
As for how the executable knows where the file is, one solution is to look for the file in several common locations. For example, if you decided to place your config in the user's home directory, look for it there first, if not found, look under /etc. And allow a special command line argument that would let a different config file to be loaded. So, say, an invocation of myprog can check for a config file in the home folder, but myprog -c /some/path/config will use /some/path/config as the file. It's also a good idea to have some default settings that you can fall back to if there is no valid config file anywhere.
The config file can go anywhere, but I'd try to put it in the same directory as any other files the program will read or write.
As for how the executable will find it, I'd pass the config file's path to the executable on the command line as an argument, with a default value of "." (which is the current directory, the one you're in when you launch the executable).
Here's the scenario:
I'm running Django 1.3.1, utilizing staticfiles, and django-compressor (latest stable) to, among other things, compile LESS files.
I have an "assets" directory that's hooked into staticfiles with STATICFILES_DIRS (for project-wide static resources). In that directory I have a "css" directory and in that a "lib.less" file that contains LESS variables and mixins.
So the physical path is <project_root>/assets/css/lib.less and it's served at /static/css/lib.less.
In one of my apps' static directory, I have another LESS file that needs to import the one above. The physical path for that is <project_root>/myapp/static/myapp/css/file.less and it would be served at /static/myapp/css/file.less.
My first thought was:
#import "../../css/lib.less"
(i.e. based on the URL, go up to levels from /static/myapp/css to /static/, then traverse down into /static/css/lib.less).
However, that doesn't work, and I've tried just about every combination of URLs and physical paths I can think of and all of them give me FilterErrors in the template, resulting from not being able to find the file to import.
Anyone have any ideas what the actual import path should be?
After tracking down exactly where the error was coming from in the django-compressor source. It turns out to be directly passed from the shell. Which clued me into removing all the variables and literally just trying to get the lessc compiler to parse the file.
Turns out it wants a relative path from the source file to the file to be imported in terms of the physical filesystem path. So I had to back all the way out to my <project_root> and then reference assets/css/lib.less from there. The actual import that finally worked was:
#import "../../../../assets/css/lib.less"
What's very odd though is that lessc would not accept an absolute filesystem path (i.e. /path/to/project/assets/css/lib.less). I'm not sure why.
UPDATE (02/08/2012)
Had a complete "DUH" moment when I finally pushed my code to my staging environment and ran collectstatic. The #import path I was using worked fine in development because that was the physical path to the file then, but once collectstatic has done it's thing, everything is moved around and relative to <project_root>/static/.
I toyed with the idea of using symbolic links to try to match up the pre and post-collectstatic #import paths, but I decided that that was far too complicated and fragile in the long run.
SO... I broke down and moved all the LESS files together under <project_root>/assets/css/, and rationalized moving the LESS files out of the apps because since they're tied to a project-level file in order to function, they're inherently project-level themselves.
I'm sort of in the same bind and this is what I've come up with for the most recent versions of compressor and lessc to integrate with staticfiles. Hopefully this will help some other people out
As far as I can tell from experimenting, lessc doesn't have a notion of absolute or relative paths. Rather, it seems to maintain a search path, which includes the current directory, the containing directory of the less file, and whatever you pass to it via --include-path
so in my configuration for compressor I put
COMPRESS_PRECOMPILERS = (
('text/less', 'lessc --include-path=%s {infile} {outfile}' % STATIC_ROOT),
)
Say, after running collectstatic I have bootstrap living at
STATIC_ROOT/bootstrap/3.2.0/bootstrap.css.
Then from any less file, I can now write
#import (less, reference) "/bootstrap/3.2.0/bootstrap.css"
which allows me to use the bootstrap classes as less mixins in any of my less files!
Every time I update a less file, I have to run collectstatic to aggregate them in a local directory so that compressor can give less the right source files to work on. Otherwise, compressor handles everything smoothly. You can also use collectstatic -l to symlink, which means you only need to collect the files when you add a new one.
I'm considering implementing a management command to smooth out the development process that either subclasses runserver to call collectstatic each time the server is reloaded, or uses django.utils.autoreload directly to call collectstatic when things are updated.
Edit (2014/12/01): My approach as outlined above requires a local static root. I was using remote storage with offline compression in my production environment, so deployment requires a couple extra steps. In addition to calling collectstatic to sync the static files to the remote storage, I call collectstatic with different django config file that uses local storage. After I have collected the files locally, I can call 'compress', having configured it to upload the result files to remote storage, but look in local storage for source files.
I have a django project that I have been working on as a solo developer, and have been using TortoiseSVN to keep the code managed in a repository on a work server. I work on this on a local installation of django etc.
There is now a second person who will be working on this project, and the possibility of working on some other PCs.
Now, there should, for the time being, only be one development version (branch?) of this project, but the configuration file (settings.py) will need to be different on each computer that is being used. I want to create one local version of this file on each PC which should not need to be changed again.
How can I set the repository (preferably within TortoiseSVN) to exclude this one file? E.g. the repository should not include settings.py. When a checkout occurs, it should update all files in the local folder but not change/remove the local copy of settings.py. When a commit occurs, settings.py should be ignored and not uploaded.
At the moment settings.py is overwritten/updated as per any other file in the project folder/repository.
Any nudges in the right direction would be useful - I'm new to SVN generally and would like to know if this is something that's going to need detailed understanding of branching or if there is a simpler way.
Thanks
In TortoiseSVN, when you try to commit your files, in the file list dialog, right click the file and look for the Ignore option. You can ignore by complete filename or extension.
If the file is already in the repository, and you want to remove it from there and ignore it, you can simply right-click the file and in the TortoiseSVN menu look for the 'Delete and add to ignore list' option.
You'll be looking for the svn:ignore property, which tells subversion to not version files matching a pattern or patterns you specify.
There's some guidance on using it with TortoiseSVN at:
http://arcware.net/tortoisesvn-global-ignore-pattern-vs-svn-ignore/
These should help:
I have a file in my project that every developer must change, but I don't want those local mods to ever be committed. How can I make 'svn commit' ignore the file?
Excluding Items from the Commit List
The typical solution is to do what bgever said and ignore the settings file itself, and then commit a file with example values, something like settings.py.example. That file should only be updated when you add or remove settings. When deploying, you'd copy that to settings.py and edit the values.