Eggs from mr.developer are not linked in buildout scripts - django

I have this config in bulidout. (All other info skipped, the \ means same line, just for readability here.)
[buildout]
extensions = mr.developer
auto-checkout = *
[sources]
media_bundler = git git://github.com/culebron/django-media-bundler.git rev=...
M2Crypto.git = git git://github.com/tobiasherp/M2Crypto.git rev=....
This eggs are downloaded and checked out, and built (the .egg folders are created). Also I see in develop_eggs, links to both of them.
M2Crypto then appears in bin/django, but media_bundler does not, and all django commands result in this error message:
$ bin/django runserver
importing
Error: No module named media_bundler
What have I missed in the config?

Turned out all I needed was to include these packages in eggs. M2Crypto was some other package requirement, that's why it was included. media_bundler is not package name, use django-media-bundler instead. Add these packages in eggs in buildout:eggs section:
eggs =
django-media-bundler
M2Crypto
django == 1.6.1
and then see their source folders appear in bin/django.

Related

New AWS Amplify withAuthenticator error related to maplibre-gl-js-amplify

NPM recommended I upgrade aws-amplify to the latest version yesterday, which I did. I then started a new React project in a dev environment (using "npm start"). The package.json file indicates these dependency versions:
"dependencies": {
"#aws-amplify/ui-react": "^2.15.1",
"aws-amplify": "^4.3.18",
"maplibre-gl-js-amplify": "^1.4.1",
... (other irrelevant dependencies removed for this example)
},
Upon calling the line below:
import { withAuthenticator } from '#aws-amplify/ui-react';
I get 12 similar errors of the same issue. Here is one such error:
WARNING in ./node_modules/#aws-amplify/ui-react/node_modules/maplibre-gl-js-amplify/lib/esm/AmplifyMapLibreGeocoder.js
Module Warning (from ./node_modules/source-map-loader/dist/cjs.js):
Failed to parse source map from 'C:\projects\amplify-project\node_modules\#aws-amplify\ui-react\node_modules\maplibre-gl-js-amplify\src\AmplifyMapLibreGeocoder.ts' file: Error: ENOENT: no such file or directory, open 'C:\projects\amplify-project\node_modules\#aws-amplify\ui-react\node_modules\maplibre-gl-js-amplify\src\AmplifyMapLibreGeocoder.ts'
# ./node_modules/#aws-amplify/ui-react/node_modules/maplibre-gl-js-amplify/lib/esm/index.js 3:0-86 6:0-134 6:0-134
# ./node_modules/#aws-amplify/ui-react/dist/esm/components/Geo/MapView/index.js 5:0-69 35:16-17
# ./node_modules/#aws-amplify/ui-react/dist/esm/index.js 13:0-60 13:0-60
# ./src/App.js 9:0-58
# ./src/index.js 8:0-24 16:33-36
maplibre-gl-js-amplify is a dependency, though not one I am even using.
I notice in the error that it's referencing the "C:\projects\amplify-project\node_modules#aws-amplify\ui-react\node_modules\maplibre-gl-js-amplify\src" directory.
However, when I navigate to "C:\projects\amplify-project\node_modules#aws-amplify\ui-react\node_modules\maplibre-gl-js-amplify", there IS no src directory!
The "C:\projects\amplify-project\node_modules#aws-amplify\ui-react\node_modules\maplibre-gl-js-amplify" directory only has dist and lib directories.
When researching, I looked at the github repo and see there is a src directory but no dist or lib directories, with different files in the file structure:
https://github.com/aws-amplify/maplibre-gl-js-amplify
I don't know if the recent changes to aws-amplify caused this new issue or not but bottom line is, it appears that Amplify is calling resources in the src directory erroneously, or is referencing the development version of this repo and not the distribution version.
Any ideas on why this is happening and how to fix?

Python install module from repo v. locally

I am trying to install a homegrown package that will be used , but python package installation is still a bit of a quagmire for me, and I haven't gotten this to work.
I created a package using setup.py sdist, which I uploaded to a repository
I am trying to install my package on another machine. I tried three methods, each time on an entirely clean machine. But none are doing what I want them to.
Method 1
easy_install http://mysite/mypkg.zip
RESULT: mypkg.egg gets added to \Python27\Lib\site-packages. But none of my folder structure is there
Method 2
pip install http://mysite/mypkg.zip
RESULT: two folders, mypkg and mypkg-1.0-py2.7.egg-info, get added to Python27\Lib\site-packages. All of the files seem to be there. But when I got to import or run nosetests on the folder, I get all sorts of import errors that reference mypkg modules. I have played with PATH and PYTHONPATH to get all variations of including the folder, but nothing has worked.
Method 3
download .zip
extract locally
add folder to PATH
run easy_install . in the local dir
RESULT: unpacks pkg locally. When I run nosetests on this folder, everything runs as expected.
Thing is, I don't want each user to have to do all of the steps in Method 3. I will eventually be running nosetests in a .bat file that does various things with the output. I don't want every user to have to modify the .bat file to indicate where the testsuite is located. Which is why Python27\Lib\site-packages appealed to me.
Any insight as to why these three methods behave so differently would be very helpful!

Specifying a django download link in djangorecipe with buildout

I want to deploy my django project using buildout, so I use djangorecipe, but
It will automatically download Django
as says in the readme, and I want to have the option to specify a url to download django (i.e. from a pypi server).
I try to set it with index and find-links buildout options but when they say automatically is true.
I searched through several recipes and found out 2 djangorecipe's forks that seem go in a similar direction, these are thechristmaspig and djbuild. The first left the download responsibility to the recipe zerokspot.recipe.git and the second allow to specify a svn repo. But both of them haven't recent activity (more than 2 years) and a very small community, that poor support make me out.
Can you suggest another way without hacks djangorecipe?
UPDATE:
I had tested with some options without success:
mr.developer:
In djangorecipe mentions use this recipe and I try with fs option of source but no way with an url like this:
[sources]
django = fs http://pypi_server/simple/
I tested with several settings but it seems only works with directory in file system
zc.recipe.egg:
But it also use the same url of the djangorecipe and no found a way change it
UPDATE2:
gp.recipe.pip:
Same results of zc.recipe.egg, but notice something: If remove the use of djangorecipe i.e. (commenting this section):
[buildout]
parts = pip
[pip]
recipe = gp.recipe.pip
install = django
#[django]
#recipe = djangorecipe
#settings = settings
it download django from the local pypi server. With django enabled (uncommenting django section and add django to buildout:parts) it download from www.djangoproject.com server.
In order to execute gp.recipe.pip first (so djangorecipe found django installed) I'm tried with two ways for refering a variable:
Refering a pip's variable from django section (as explain in the section "Automatic part selection and ordering" of buildout docs)
Proceding with section Extending sections (macros) of buildout docs
Note that reorganize sections or change the values order in parts variable no matters for change the execution order.
But again no success
Any ideas?
Thanks
Based on the UPDATE2's comments in the question I figure to have two .cfg files: the one with only pip settings and the second for django settings. Then execute twice buildout using -c option to specify files. So with first buildout call it installs django from pypi server, second encounter that django has been installed and move on. This should works and maybe this is the solution, any other? thx

Django Buildout recipe for pywhois (not on pypi)

I Am trying to create a buildout recipe for pywhois on google code.
The site of which is here:
http://code.google.com/p/pywhois/
Note: if you use easy_install pywhois it installs another package from pypi (python package index) : http://pypi.python.org/pypi/pywhois/0.1
I Am following the documentation on buildout recipe... (http://pypi.python.org/pypi/djangorecipe/1.3)
and I found out I can clone the sourcecode from here:
hg clone https://code.google.com/p/pywhois/
but I can't get it all stick together in a working buildout script... (to be honoust, this is my first buildout script trial... for all other packages I found an existing recipe and normally I try to avoid anything that's not on pypi)
i created 2 issues on their site... on of them is to use another name than in the python package index, the other one is to get their package over there.
Any buildout guru?
EDIT
(already 2 guys trying to help me, thanks!)
I used this in my buildout.conf:
extensions =
buildout.dumppickedversions
mr.developer
auto-checkout = pywhois
and added to my list develop:
develop = src/pywhois
and added to my egg list:
eggs =
myproject
pywhois
and declared the source of the repo:
[sources]
pywhois = hg https://code.google.com/p/pywhois/
and off course add to eggs: pywhois
and it's downloading it and creating an egg...
EDIT2
Whatever I do, I keep getting: Source URL for existing package 'pywhois' differs. Expected 'https://code.google.com/p/pywhois/'.";
It seems to me if the package-name of the google code package is gonna be changed, that this issue will be solved... I aleady got response on the issue tracker about this.
see http://code.google.com/p/pywhois/issues/detail?id=33
EDIT 3: Now on PyPI: python-whois
And, a brilliant quick action from the developer (Richard Penman). So the package is renamed to python-whois (he is thinking about another name, but for now this works). And he put it on pypi!
So this issue is resolved for me, but the answers of Martijn and Reinout learned me a lot more about buildout, thanks!
I'd use mr.developer to grab a local checkout, and use it as a development egg:
[buildout]
extensions = mr.developer
auto-checkout = pywhois
[sources]
pywhois = hg https://code.google.com/p/pywhois/
Now mr.developer will check out pywhois into src/, run it's setup.py to make it a development egg, and tell buildout it's available as such. Now buildout will use that local copy to satisfy any pywhois requirement.

Go build: "Cannot find package" (even though GOPATH is set)

Even though I have GOPATH properly set, I still can't get "go build" or "go run" to find my own packages. What am I doing wrong?
$ echo $GOROOT
/usr/local/go
$ echo $GOPATH
/home/mitchell/go
$ cat ~/main.go
package main
import "foobar"
func main() { }
$ cat /home/mitchell/go/src/foobar.go
package foobar
$ go build main.go
main.go:3:8: import "foobar": cannot find package
It does not work because your foobar.go source file is not in a directory called foobar. go build and go install try to match directories, not source files.
Set $GOPATH to a valid directory, e.g. export GOPATH="$HOME/go"
Move foobar.go to $GOPATH/src/foobar/foobar.go and building should work just fine.
Additional recommended steps:
Add $GOPATH/bin to your $PATH by: PATH="$GOPATH/bin:$PATH"
Move main.go to a subfolder of $GOPATH/src, e.g. $GOPATH/src/test
go install test should now create an executable in $GOPATH/bin that can be called by typing test into your terminal.
Although the accepted answer is still correct about needing to match directories with package names, you really need to migrate to using Go modules instead of using GOPATH. New users who encounter this problem may be confused about the mentions of using GOPATH (as was I), which are now outdated. So, I will try to clear up this issue and provide guidance associated with preventing this issue when using Go modules.
If you're already familiar with Go modules and are experiencing this issue, skip down to my more specific sections below that cover some of the Go conventions that are easy to overlook or forget.
This guide teaches about Go modules: https://golang.org/doc/code.html
Project organization with Go modules
Once you migrate to Go modules, as mentioned in that article, organize the project code as described:
A repository contains one or more modules. A module is a collection of
related Go packages that are released together. A Go repository
typically contains only one module, located at the root of the
repository. A file named go.mod there declares the module path: the
import path prefix for all packages within the module. The module
contains the packages in the directory containing its go.mod file as
well as subdirectories of that directory, up to the next subdirectory
containing another go.mod file (if any).
Each module's path not only serves as an import path prefix for its
packages, but also indicates where the go command should look to
download it. For example, in order to download the module
golang.org/x/tools, the go command would consult the repository
indicated by https://golang.org/x/tools (described more here).
An import path is a string used to import a package. A package's
import path is its module path joined with its subdirectory within the
module. For example, the module github.com/google/go-cmp contains a
package in the directory cmp/. That package's import path is
github.com/google/go-cmp/cmp. Packages in the standard library do not
have a module path prefix.
You can initialize your module like this:
$ go mod init github.com/mitchell/foo-app
Your code doesn't need to be located on github.com for it to build. However, it's a best practice to structure your modules as if they will eventually be published.
Understanding what happens when trying to get a package
There's a great article here that talks about what happens when you try to get a package or module: https://medium.com/rungo/anatomy-of-modules-in-go-c8274d215c16
It discusses where the package is stored and will help you understand why you might be getting this error if you're already using Go modules.
Ensure the imported function has been exported
Note that if you're having trouble accessing a function from another file, you need to ensure that you've exported your function. As described in the first link I provided, a function must begin with an upper-case letter to be exported and made available for importing into other packages.
Names of directories
Another critical detail (as was mentioned in the accepted answer) is that names of directories are what define the names of your packages. (Your package names need to match their directory names.) You can see examples of this here: https://medium.com/rungo/everything-you-need-to-know-about-packages-in-go-b8bac62b74cc
With that said, the file containing your main method (i.e., the entry point of your application) is sort of exempt from this requirement.
As an example, I had problems with my imports when using a structure like this:
/my-app
├── go.mod
├── /src
├── main.go
└── /utils
└── utils.go
I was unable to import the code in utils into my main package.
However, once I put main.go into its own subdirectory, as shown below, my imports worked just fine:
/my-app
├── go.mod
├── /src
├── /app
| └── main.go
└── /utils
└── utils.go
In that example, my go.mod file looks like this:
module git.mydomain.com/path/to/repo/my-app
go 1.14
When I saved main.go after adding a reference to utils.MyFunction(), my IDE automatically pulled in the reference to my package like this:
import "git.mydomain.com/path/to/repo/my-app/src/my-app"
(I'm using VS Code with the Golang extension.)
Notice that the import path included the subdirectory to the package.
Dealing with a private repo
If the code is part of a private repo, you need to run a git command to enable access. Otherwise, you can encounter other errors This article mentions how to do that for private Github, BitBucket, and GitLab repos: https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4
This issue is also discussed here: What's the proper way to "go get" a private repository?
I solved this problem by set my go env GO111MODULE to off
go env -w GO111MODULE=off
Note: setting GO111MODULE=off will turn off the latest GO Modules feature.
Reference: Why is GO111MODULE everywhere, and everything about Go Modules (updated with Go 1.17)
GO111MODULE with Go 1.16
As of Go 1.16, the default behavior is GO111MODULE=on, meaning that if
you want to keep using the old GOPATH way, you will have to force Go
not to use the Go Modules feature:
export GO111MODULE=off
In the recent go versions from 1.14 onwards, we have to do go mod vendor before building or running, since by default go appends -mod=vendor to the go commands.
So after doing go mod vendor, if we try to build, we won't face this issue.
Edit: since you meant GOPATH, see fasmat's answer (upvoted)
As mentioned in "How do I make go find my package?", you need to put a package xxx in a directory xxx.
See the Go language spec:
package math
A set of files sharing the same PackageName form the implementation of a package.
An implementation may require that all source files for a package inhabit the same directory.
The Code organization mentions:
When building a program that imports the package "widget" the go command looks for src/pkg/widget inside the Go root, and then—if the package source isn't found there—it searches for src/widget inside each workspace in order.
(a "workspace" is a path entry in your GOPATH: that variable can reference multiple paths for your 'src, bin, pkg' to be)
(Original answer)
You also should set GOPATH to ~/go, not GOROOT, as illustrated in "How to Write Go Code".
The Go path is used to resolve import statements. It is implemented by and documented in the go/build package.
The GOPATH environment variable lists places to look for Go code.
On Unix, the value is a colon-separated string.
On Windows, the value is a semicolon-separated string.
On Plan 9, the value is a list.
That is different from GOROOT:
The Go binary distributions assume they will be installed in /usr/local/go (or c:\Go under Windows), but it is possible to install them in a different location.
If you do this, you will need to set the GOROOT environment variable to that directory when using the Go tools.
TL;DR: Follow Go conventions! (lesson learned the hard way), check for old go versions and remove them. Install latest.
For me the solution was different. I worked on a shared Linux server and after verifying my GOPATH and other environment variables several times it still didn't work. I encountered several errors including 'Cannot find package' and 'unrecognized import path'. After trying to reinstall with this solution by the instructions on golang.org (including the uninstall part) still encountered problems.
Took me some time to realize that there's still an old version that hasn't been uninstalled (running go version then which go again... DAHH) which got me to this question and finally solved.
Running go env -w GO111MODULE=auto worked for me
Without editing GOPATH or anything, in my case just worked the following:
/app
├── main.go
├── /utils
└── utils.go
Import packages where needed. This can be unintuitive, because it isn't relative to the app path. You need to add the app in the package path too:
main.go:
package main
import(
"app/util"
)
Being in app directory, run:
go mod init app
go get <package/xxx>
go build main.go / go run main.go
You should be good to go.
GOPATH = /home/go
appPath = /home/projects/app
Create a proper go.mod and go.sum with go mod init app (delete old before)
After that resolve all dependencies like missing packages with go get github.com/example/package.
In simple words you can solve the import problem even with GO111MODULE=on with the following syntax for import:
import <your_module_name>/<package_name>
your_module_name -> module name which can be found in the go.mod file of the module as the first line.
example: github.com/nikhilg-hub/todo/ToDoBackend
package_name -> Path to your package within module.
example: orm
So the import statement would look like:
import "github.com/nikhilg-hub/todo/ToDoBackend/orm"
According to me we need to specify the module name + package name because we may need a same package name in two or more different modules.
Note: If you are importing a package from same module still you need to specify the full import path like above.
If you have a valid $GOROOT and $GOPATH but are developing outside of them, you might get this error if the package (yours or someone else's) hasn't been downloaded.
If that's the case, try go get -d (-d flag prevents installation) to ensure the package is downloaded before you run, build or install.
GOROOT should be set to your installation directory (/usr/local/go).
GOPATH should be set to your working directory (something like /home/username/project_folder).
GOPATH should not be set to GOROOT as your own project may need to install packages, and it's not recommended to have those packages in the Go installation folder. Check out this link for more.
For me none of the above solutions worked. But my go version was not the latest one. I have downloaded the latest version and replaced the older version in my mac os after that it worked perfectly.
I had a similar problem when building a docker file:
[1/3] STEP 9/9: RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
api/v1alpha1/XXX.go:5:2: cannot find package "." in:
/workspace/client/YYY/YYY.go
This only appeared when building the Dockerfile, building locally worked fine.
The problem turned out to be a missing statement in my Dockerfile:
COPY client/ client/
I do not understand why this happens, we must be able to import from wherever our file is in its nest, since I have discovered that if we have more than one nest this will throw an error.
package main
import (
"fmt"
"indexer/source/testPackage3" // this will be show a GOROOT error.
"indexer/testPackage"
"indexer/testPackage2"
)
func main() {
fmt.Println("Agile content indexer -")
fmt.Println(testPackage.Greeting())
fmt.Println(testPackage2.Greeting())
fmt.Println(testPackage3.Greeting())
}
├── testPackage2
│ ├── entry2.go
│ └── source
│ └── entry3.go
To conclude, I just want to tell you, the entry3.go file will not work when imported into my main file, which in this case is (main.go), I do not understand why, therefore, I have simply chosen to use a depth folder in the packages I need to export.
entry.go, entry2.go will work perfectly when imported, but entry3.go will not work..
In addition, both the directory and the name of the package must be the same so that they work properly when importing them.
Have you tried adding the absolute directory of go to your 'path'?
export PATH=$PATH:/directory/to/go/