I run: conan install Boost/1.64.0#conan/stable, and it fails.
Output:
C:\temp>conan install Boost/1.64.0#conan/stable
Boost/1.64.0#conan/stable: Not found in local cache, looking in remotes...
Boost/1.64.0#conan/stable: Trying with 'bintray'...
Boost/1.64.0#conan/stable: Trying with 'conan.io'...
ERROR: Unable to find 'Boost/1.64.0#conan/stable' in remotes
Trying other package, works:
C:\temp>conan install fmt/4.0.0#bincrafters/stable
fmt/4.0.0#bincrafters/stable: Not found in local cache, looking in remotes...
fmt/4.0.0#bincrafters/stable: Trying with 'bintray'...
fmt/4.0.0#bincrafters/stable: Trying with 'conan.io'...
Downloading conanmanifest.txt
[==================================================] 121B/121B
Downloading conanfile.py
[==================================================] 1.8KB/1.8KB
fmt/4.0.0#bincrafters/stable: Installing package
Requirements
fmt/4.0.0#bincrafters/stable from conan.io
Packages
fmt/4.0.0#bincrafters/stable:63da998e3642b50bee33f4449826b2d623661505
fmt/4.0.0#bincrafters/stable: Retrieving package 63da998e3642b50bee33f4449826b2d623661505
fmt/4.0.0#bincrafters/stable: Looking for package 63da998e3642b50bee33f4449826b2d623661505 in remote 'conan.io'
Downloading conanmanifest.txt
[==================================================] 938B/938B
Downloading conaninfo.txt
[==================================================] 491B/491B
Downloading conan_package.tgz
[==================================================] 159.8KB/159.8KB
fmt/4.0.0#bincrafters/stable: Package installed 63da998e3642b50bee33f4449826b2d623661505
Any idea why the package isn't found?
How to debug it?
Conan is a decentralized package manager (kind of git-like style), so it can have many remotes. By default it comes configured with 2 remotes:
conan-transit: Is a read-only copy of the old conan.io repository, which contains many different Boost packages, from different authors. Quality is variable, so some packages might work only for certain OS, or might fail for some configurations.
conan-center: It is a moderated/reviewed repository, package creators can submit inclusion requests to share their packages with the community.
So far conan-transit contains several Boost/1.64 packages, so can check it with:
$ conan search Boost* -r=conan-transit
$ conan search Boost* -r=conan-center
As you can see the package you are trying to install doesn't exist in these repositories.
As I said above, conan is decentralized, so you can use different remotes. For example, the "bincrafters" community has a bintray repo that can be added with:
$ conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
$ conan search Boost* -r=bincrafters
You will see they have a large number of Boost/1.64 packages, because they have created a modularized version of boost, in which every library lives in a different package, so you only get installed what you need.
UPDATE: Packages in the central repository are being renamed by the community to lowercase. Try with boost lowercase in the above if necessary.
Related
When using AWS inspector for ECR images, the results mentioned vulnerabilities that don't seem to be installed on the image.
This article from snyk discusses how AWS Inspector is using Snyk to help with it's detecting.
Question:
My question is does AWS inspector or Snyk include package vulnerabilities that are in yarn.lock but not installed?
Is this the convention for other package inspectors, to include vulnerabilities in packages that aren't installed but are listed as dependencies?
This doesn't make sense, why would it be a vulnerability if the package is not actually installed.
More Info:
The reason I don't think the package is installed is because when I run npm list (docs) the vulnerable package is not listed.
Also when I test locally I don't see a node_models folder with the suspected package. I do see it listed in a yarn.lock file.
(The suspected package is a devDependency of a dependency of a dependency of a package that is installed globally)
Side Notes:
When I run npm audit (docs) I don't see any security issues listed. npm audit has it's own issues with listing vulnerabilities that may not be a real issue (see here and enter link description here), but at least it's not listing packages that aren't even installed.
This question may be dependent on this question, regarding devDependencies being installed when installing a package (globally).
We solved this problem by manually deleting all yarn.lock files from our docker images as the last step:
RUN yarn install --production --silent --pure-lockfile --cache-folder /usr/app/yarn-cache \
# get rid of all yarn.lock files as the AWS Inspector thinks they contain vulnerabilities
&& rm -rf ./yarn-cache \
&& find . -type f -name 'yarn.lock' -delete
obviously the bit about the --cache-folder may not be relevant to you, we just keep the cache around through our multistage builds to speed things up
Problem
I am attempting to install Python 2.7.16, openpyxl, and pyinstaller onto a Windows 10 machine that is offline for security reasons. To clarify, I have a mapped network drive on there from which I can transfer the files I need to use.
Question
What is the best way to go about this? I currently have a .msi Python installation file directly from their website. The packages I need are packaged as .tar.gz files. I currently have those on my windows machine, but do not want to proceed until I know for sure what I need to do. Also, do I need to do anything for dependencies? If so, how do I find the dependencies for the packages I need?
Side Notes
The version of Python (2.7.16) comes with pip. Not sure if that makes a difference. Downloading and transferring things requires me to ask my admin, for him to download the files, and then transfer them to my drive so I can have them on my computer. If able, I would like to do this in as little attempts as possible.
Useful links
Python: https://www.python.org/downloads/release/python-2716/
openpyxl: https://pypi.org/project/openpyxl/#files
pyinstaller: https://pypi.org/project/PyInstaller/#files
My solution would be to seek out the offline versions of the python and pip installer and follow this guide
Also a great tip: try the complete procedure (the installing of the required software) on a seperate pc which you have disconnected and do the installation. Note everything you have to do to get it working and use those instruction on your originally intended machine. This will prevent you from having to go back and forth and scratch your head while installing on the target machine.
Please note that I have NO idea how python works and this is just a hunch from me as a programmer.
Installing Python and packages on an Offline Machine: A Comprehensive Guide
The Environment
Let us begin by defining the environment in which this guide may be of some great use. If your situation can be described by one or more of the following, you might have great results following this guide...
The machine you are developing on is offline. (No connection to the internet)
You need to develop and run Python on the machine that is completely offline.
If this sounds like you, read the following cases in which a few minor details may make a big difference in getting you started.
Case1:
You are not allowed to plug in any external media devices into the offline machine. This includes but is not limited to a USB, CD, floppy disk, or any other removable media that may be of some use in helping you transfer Python files to the offline machine.
You are allowed to map a network drive (somewhere else on the local network). This would fix the problem mentioned in number one with removable media.
Answer: In this case, just proceed with the guide, as this was my case and I will explain in detail how I solved my problem.
Case2:
There is no physical way to transfer files onto the development machine that is offline.
Answer: If this is your case, you need to get in touch with the admin team who handles the software on your development machine. Direct them to this guide to proceed.
Let's Get Started
Warning A:
The following must be performed on a computer with an internet connection. It is impossible to download things from any website without an internet connection.
Warning B:
There is a longer way, and there is a shorter way to do the following. To avoid the longer way, you must be able to install python on a different machine that is online. This can be the same machine that you are using to download the packages and python version, or it can even be a home machine. This can be any machine in the world that is on the internet. It's sole purpose will be to help you identify the dependencies of each package.
Installing Python
Visit the python website and identify the version you want. 2.7.9 and up is recommended for this guide. Download the file for your specific system.
Python 2.7.9 : https://www.python.org/downloads/release/python-279/
Python 3.7.3 : https://www.python.org/downloads/release/python-373/
The reason I provided Python 2.7.9 is because that is the earliest 2.7.x version that comes with pip (a package manager).
Visit the python package index to locate the packages you will be using in your python project. https://pypi.org/
Search the package you need, go to the downloads, and get the (.tar.gz) file. Not the .whl files unless you know what you are doing with those.
Tip: If you want to keep track of the packages you are installing, I suggest you put them all in one folder somewhere you can find, or just write them down on paper.
Unpack the .tar.gz package files. You can get rid of the .tar.gz once you unpack them as they will not needed any longer.
Install the version of python that you downloaded for your system in step 1 above.
(This may just be running the .msi file for windows or unpacking some files for linux) If you are not sure how, just look at this brilliant guide
https://realpython.com/installing-python/
Now you should be able to go to your terminal and type "python" and get the python interpreter to open up. If you get a "cannot find python command" you need to setup your path variable.
Windows guide: https://geek-university.com/python/add-python-to-the-windows-path/
Linux guide: https://www.tutorialspoint.com/python/python_environment.htm
Your python installation is done! And your packages should also be ready to install!
Installing Python Packages
What you need to know here is that MOST all python packages have dependencies, which are other packages which packages need installed before they can be installed. If you need more explanation on dependencies, read here: https://www.fullstackpython.com/application-dependencies.html
Before proceeding be sure to add the Python/Scripts folder to your path variable too or pip will not work. Follow this link for instructions. https://appuals.com/fix-pip-is-not-recognized-as-an-internal-or-external-command/
Install packages using pip install [package_name] for every package you need, on your machine that is on the internet, and then do a pip freeze to see all the packages installed.
Once you can see all the packages installed, which will include the dependencies for the ones you ran pip install on, you need to manually download these dependencies from the python package index https://pypi.org/ just like you did with the regular packages.
Moving Offline
Once you have identified all the packages you will need, and all of their dependencies, you will need to download them, unpack all of them, and move them into one folder, which I will call "OFFLINE_SETUP_FOLDER".
To be clear:
The packages we installed before was only to find out the dependencies we were going to need. You do not have to re-download the packages you have already downloaded before running pip install. You should only need to download the dependencies you have found during the pip freeze command.
Finally you need to copy into the "OFFLINE_SETUP_FOLDER" your python installation file, be it a .msi file for windows, or the .tar file for linux.
Your "OFFLINE_SETUP_FOLDER" should contain the following...
In the following, package can be the name of any package that you downloaded, and the a and b inpackage1a and package1b just represent dependencies for that package. These file names are just examples for packages
python.msi (installation file for python)
/package1 (normal package folder)
/package1a (package dependency folder)
/package1b (package dependency folder)
/package2 (normal package folder)
/package3 (normal package folder)
/package3a (package dependency folder)
Once this is complete, you need to move that folder onto the machine that is completely offline form the network.
Then run the installation for python as you did before and install it on the machine. Do no forget to setup the path variable. Refer back to the Installing Python section if needed.
Open your terminal or CMD and CD into the "OFFLINE_SETUP_FOLDER".
Now you need to CD into each individual package folder, and run this command: python setup.py install and let it run.
If the package install fails, it will be because one of the dependencies has not been installed. If this is the case, CD into the dependency that is says is missing, and run python setup.py install in there first.
Keep repeating these steps until all packages and dependencies have been installed.
This is the end of this python guide for installing python on an offline machine. I hope this helped :)
I have a specific version of google-cloud-core in my server and I don’t know how to choose the other libraries in a way to make them suit with my google cloud core version.
I can’t just move to the latest versions because old programs cannot run into new versions for example BigQuery library.
My specific need is to know « how to know » which version of Google Cloud Storage should I choose according to the 0.26.0 core version.
Is there some repositories where we can find packages grouped by google-cloud-core versions?
In my case the version 1.6 of google-cloud-storage works but I found it just by downgrade and try again method !
Best regards
What you can do as a workaround is create a virtual environment, install a specific library - like google-cloud-storage - and check the dependencies installed with that library version. I made a quick test and installed a few versions of google-cloud-storage. For version 1.3.0, the google-cloud-core 0.26.0 dependency was installed.
You can do so by following these steps:
virtualenv env-name
source env-name/bin/activate
pip freeze (to check there is nothing there)
pip install google-cloud-storage==1.3.0
pip freeze (again)
Once finished you’ll see google-cloud-core 0.26.0 was installed.
Can I use apt-get or other package managers in Cloud Foundry buildpacks or .profile scripts that come with apps; and if I can, how to do it? I expect to do it the same way as in a dockerfile, but it doesn't work with or without sudo in my case.
Can I use apt-get or other package managers in Cloud Foundry buildpacks or .profile scripts that come with apps; and if I can, how to do it?
No. Running apt-get or a package manager would typically require root access and you do not get root access when the build pack runs or when your application runs (this is a difference w/Docker).
That said, you can do anything that doesn't require root access, so if you found a package manager that installed in the vcap user's home directory and didn't need root then you could use that.
It depends on what you're trying to install, but in some cases you can work around this by downloading the .deb or .rpm file and manually extracting the binaries. This typically works OK for things like shared libraries. Just download the precompiled binary that matches your stack (cflinuxfs2 == Ubuntu Trusty). For other things, you can build your own binaries from source. This is what the build pack's do, see binary-builder.
Hope that helps!
So far I installed deployment version of wso2 AM. Now I would like to build it from source and try running it instead of the binaries I downloaded from the site.
Based on WSO2 documentation, I understand the steps are:
1) Download the carbon kernel source:
git clone -b 4.4.x https://github.com/wso2/carbon-kernel.git
2) Download the APIM source:
git clone https://github.com/wso2/product-apim
3) Build APIM from source
cd <SOURCE-DIR>\product-apim
mvn clean install
Are these steps sufficient, or am I missing something?
Should I build carbon-kernel in addition to building apim-manager?
On previous stackoverflow question, I read that carbon-kernel is not really necessary, and instead i should download and build carbon-apimgmt. Is this correct?
After I build the sources, how do I "package" all the compiled binaries along with all other necessary artifacts, in order to form an equivalent package to the wso2am-1.10.0.zip which I download from the site? Or is there another way to install and run the built code?
Github projects related to API manger can be found in following locations
apimgt component repo:
https://github.com/wso2/carbon-apimgt
This repository contains org.wso2.carbon.apimgt component related source code.
product repo:
https://github.com/wso2/product-apim
This repository contains all the resources needed to build the product package and intergration tests for the product.
master branch of these repositories are used for current development. (if you open parent pom.xml file you would find SNAPSHOT versions). If you build the default branches you would build the current development version of the api manager. (at this time, 1.10.1-SNAPSHOT). To build already released product you need to build released tag.
Steps to Build API manager 1.10.0
clone product:
git clone https://github.com/wso2/product-apim
Checkout release tag v1.10.0:
git checkout v1.10.0
Build the product:
mvn clean install (or mvn clean install -Dmaven.test.skip=true to skip integration tests)
get the product from
product-apim\modules\distribution\product\target
You do not have to build the 'carbon-apimgt' repository because the component build using that is already released and can be found in the nexus repo.
If you want to build the component (say need to provide a fix for a bug) build the 'v5.0.3' tag from the 'carbon-apimgt' repo.
git clone https://github.com/wso2/product-apim
git checkout v5.0.3
I'm posting the steps I did:
git clone https://github.com/wso2/carbon-appmgt
git clone https://github.com/wso2/product-apim
cd <SRC>/carbon-appmgt
mvn clean install
cd <SRC>/product-apim
mvn clean install
The ZIP file was found in
<SRC>\product-apim\modules\distribution\product\target
It is similar to the ZIP file that you download from the site.