I am trying to reproduce my development environment in a docker image. I am able to get simple dependencies met such as python+a couple standard packages, largely through the builds from docker hub. But when it comes to installing xgboost or pandas I am having great difficulty.
After looking into the error messages it looked like I had the wrong version of g++ installed. The build had 4.7, but xgboost requires 4.9+. As I tried to update g++ I kept running into a wall where I couldn't update g++ because I needed another package (apt-add-repository), but to install that package I needed another (apt-utils) etc.
Does anyone have any general advice with setting up a Docker image or for this specific problem of upgrading the g++.
Here is the Docker file:
FROM continuumio/anaconda
MAINTAINER maintainer
RUN apt-get install -y g++-4.9
One test would be to start from a gcc:4.9 image (which uses wheezy), and try to add what anaconda Dockerfile does.
That way, you start from an image with the right gcc.
You first need to make sure your source list is up-to-date. The line with RUN command in the dockerfile should be
RUN apt-get update && apt-get install -y g++
Related
Background
I have a CI pipeline for a C++ library I've been developing. So far, I can distribute this lib to Linux and Windows systems. Since I use GitLab to build, test and package my lib, I'd like to have my Windows builds running faster and I have no clue on how to do that.
Currently, I use the following script for my Windows builds:
.windows_template:
tags:
- windows
before_script:
- choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
- choco install python --pre -y
- choco install git -y
- $env:ChocolateyInstall = Convert-Path "$((Get-Command choco).Path)\..\.."; Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"; refreshenv
- python -m pip install --upgrade pip
- pip install conan monotonic
The problem
Any build with the script above can take up to 10 minutes; worse: I have two stages, each one taking the same amount of time. This means that my whole CI pipeline will take 20 minutes to finish because of slowness in Windows builds.
Ideal solution
EVERYTHING in my before_script can be cached or stored as an image. I only need some hints on how to do it properly.
Additional information
I use the following tools for my builds:
CMake: to support my building process;
Python3: to test and build packages;
Conan (requires Python3): to support the creation of packages with several features, as well as distribute them;
Git: to download Googletest in CMake configuration step This is already provided in the cookbooks - I might just remove this extra installation step in my before_script;
Googletest (requires Python3): testing library;
Visual Studio DEV Tools: to compile the library This is already in the cookbooks.
Installing packages like this (whether it's OS packages though apt-get install... or pip, or anything else) is generally against best practices for CI/CD jobs because every job that runs will have to do the same thing, costing a lot of time as you run more pipelines, as you've seen already.
A few alternatives are to search for an existing image that has everything you need (possible but not likely with more dependencies), split up your job into pieces that might be solved by an image with just one or two dependencies, or create a custom docker image to use in your jobs. I answered a similar question with an example a few weeks ago here: "Unable to locate package git" when running GitLab CI/CD pipeline
But here's an example Dockerfile with Windows:
# Dockerfile
FROM mcr.microsoft.com/windows
RUN ./install_chocolatey.sh
RUN choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
RUN choco install python --pre -y
RUN choco install git -y
...
The FROM line says that our new image extends the mcr.microsoft.com/windows base image. You can extend any image you have access to, even if it already extends another image (in fact, that's how most images work: they start with something small, like a base OS installation, then add things needed for that package. PHP for example starts on an Ubuntu image, then installs the necessary PHP packages).
The first RUN line is just an example. I'm not a Windows user and don't have experience installing Chocolatey, but you'd do here whatever you'd normally do to install it locally. The rest are for installing whatever else you need.
Then run
docker build /path/to/dockerfile-dir -t mygroup/mytag:version
The path you supply needs to be the directory that contains the Dockerfile, not the Dockerfile itself. The -t flag sets the image's tag after it's built (though you can do that with a separate command, docker tag too).
Next, you'll have to log into whichever registry you're using (Docker Hub (https://docs.docker.com/docker-hub/repos/), Gitlab Container Registry (https://docs.gitlab.com/ee/user/packages/container_registry/), a private registry your employer may support, or any other option.
docker login my.docker.hub.com
Now you can push the image to the registry:
docker push my.docker.hub.com/mygroup/mytag:version
You'll have to review the information in the docs about telling your Gitlab runner or pipelines how to authenticate with the registry (unless it's Public on Docker Hub or you use the Gitlab Container Registry) https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-an-image-from-a-private-container-registry
Once all that's done, you can use your new image in your CI jobs, and everything we put into the image will be ready to use:
.windows_template:
image: my.docker.hub.com/mygroup/mytag:version
tags:
- windows
...
I'm trying to build a basic Docker container based on a tutorial. I am on Windows 10 Home version 2004, and I am using the standard command line. I've created the following Docker file to facilitate this, with the only change from the tutorial's version being my older version of gcc:
FROM gcc:6.3.0
RUN apt-get -qq update
RUN apt-get -qq upgrade
RUN apt-get -qq install cmake
RUN apt-get install libboost-all-dev=1.62.0.1
RUN apt-get -qq install build-essential libtcmalloc-minimal4 && \
ln -s /usr/lib/libtcmalloc_minimal.so.4 /usr/lib/libtcmalloc_minimal.so
Once the script gets to the step where it tries to install libboost-all-dev I get the following output:
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.62.0.1' for 'libboost-all-dev' was not found
The command '/bin/sh -c apt-get install libboost-all-dev=1.62.0.1' returned a non-zero code: 100
and the build stops.
I've tried updating the build script to use the current version of Boost (1.74.0) as well and get the same issue. I'm not really finding any solutions in my research online and the output is not very helpful in trying to figure out what the issue is. Could anyone with more experience with installing Boost as part of the Docker process point me in the right direction?
The package manager will only be able to install versions of Boost that it knows exist, based on the enabled package manager repositories. There is typically only one version of Boost in the default repositories. In my experience, this applies to any Linux OS that supplies Boost, not only those that are run within a Docker container.
The Docker image you started with, gcc:6.3.0, appears to have only Boost version 1.55.0.2, so requesting any other version will yield the same error.
If you want a different version of Boost in your image, you can follow the typical steps for installing a different version of Boost outside a Docker container. These steps are well-documented on Stack Overflow, or you might find a repository such as this to enable in your package manager to directly install it from apt-get.
I try to install the package "data.table" (and "aws.s3)" via Rstudio Server on an Amazon Linux instance following this guide:
http://stanke.co/category/r/
Unfortunately, I get the following error message. I really don't know what else to do.
Can anybody help? I installed devtools and I am able to install other packages such as xml2, devtools and deplyr.
I had the same issue on AWS and already fixed.
You need first install gcc64 and openmp shared support library.
sudo yum install gcc64
sudo yum install libgomp
Then under your user home create an .R folder with a Makevars file in it, with the following content (it will tell to R which compiler to use):
CC = /usr/bin/gcc64
CXX = /usr/bin/g++
SHLIB_OPENMP_CFLAGS = -fopenmp
I hope it's working for you as well ...
You need to install dmlc-core.
This link will provide more information:
A common bricks library for building scalable and portable distributed machine learning
based on https://github.com/RcppCore/RcppArmadillo/issues/200, I think this issue is due to a g++ compatability issue. It might also explain why when I installed devtools it kept giving me [-Wdeprecated-declarations]
so run:
sudo yum remove gcc72-c++.x86_64 libgcc72.x86_64
yum install R-devel
Then you should be able to run the installation command.
I am trying to install docker from the source code downloaded from github.com/docker/docker
I am unable to install it from the source code .
The Makefile present creates a image , but i want to install it in my system.
Can anyone suggest solution ?
I am using UBUNTU 14.04
Well, idk if this works for your linux distro. (looks like it is ubuntu) but i run kali linux and even if we have different commands to use the process is just as same in every linux distro.
first, before we jump on, we need to update our linux repos.(repositories)
sudo apt update
and,
sudo apt-get update
then,
sudo apt install git
[This installs git]
Now we can start cloning git repos. into our system
go to your desired folder/working directory and type:
sudo git clone "link of the git repo. without the commas"
i would better suggest you to just:
sudo apt install docker.io
[To install docker by apt]
it's better to install it via the docker package and update it to the last version. This is the best way to install docker.
sudo apt-get install python-Orange
or
sudo apt-get install python-orange
doesn't work
sudo python setup.py install
sudo python setup.py build
is not working as well.
Can anyone help??
Python has two tools for easy installation of all programs that are listed on the Python Package Index, also known as PyPi: These are easy_install and pip. Both retrieve very recent versions of Orange (and of any other package that is updating its PyPi entry regularly).
I installed Orange on Ubuntu 12.04 (LTS) with
pip install orange.
You will see lots of log lines indicating that Pip is downloading and compiling Orange for you. Simply wait. When pip is ready, fire up python and try to import orange. If that works, quit python and try the GUI with python /usr/local/lib/python2.7/dist-packages/Orange/OrangeCanvas/orngCanvas.pyw (you probably want to create a shell alias or bash script for that one :-)
NOTE: on 12.04 I needed to first upgrade 'distribute' itself with sudo easy_install -U distribute but this was clearly indicated by pip.
https://pypi.python.org/pypi/Orange/2.6/
You need to extract the dowloaded tarball on that page to a folder and then change directory to that folder. Then the sudo python setup.py... instructions will work (but you should 'build' the application before you 'install' it).
go to the given link "https://pypi.python.org/pypi/Orange/2.6/"
download the package and extract the file
install with given command
python setup.py build
python setup.py install
note:- during installation make sure that your net is working because it downloads required packages. Also it may ask for C++ or gcc compilers while installing and could be terminate just read the errors care fully and install requires packages from the synaptic package manage in ubuntu.