I am trying to install the lustre clients on Unbuntu 20.04 nodes I have in GCP. Im using linux kernel version 5.15.0-1021-gcp.
I'm trying to install the client with the following code:
cd /home/apps/
mkdir lustre
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout 2.15.0
sh autogen.sh
./configure --prefix=/home/apps/lustre --disable-server --enable-client ## doesnt run! Fails at ./configures with error message "error: Run make config in /lib/modules/5.15.0-1021-gcp/build"
make debs
The configure step fails with an error about running make config in /lib/modules/5.15.0-1021-gcp/build. I tried running make config in /lib/modules/5.15.0-1021-gcp/build but was asked to input some values that I was unsure of.
I also tried downloading the deb package of the client software at
https://downloads.whamcloud.com/public/lustre/lustre-2.15.0/ubuntu2004/client/lustre-client-modules-5.4.0-96-generic_2.15.0-1_amd64.deb. However this is for the wrong linux kernel and I'm not sure what env variables need to be set for this package.
Anyone know how to install the client modules for lustre on Ubuntu?
You need to have the kernel sources or kernel-devel package that exactly match the kernel that you are installing on. This should also include the .config file that describes all of the options used when building your kernel.
Alternately, you could try a pre-built package, but it isn't clear if this will install on your kernel or not.
https://build.whamcloud.com/job/lustre-b2_15/40/arch=x86_64,build_type=client,distro=ubuntu2204,ib_stack=inkernel/
Related
Background
I have a CI pipeline for a C++ library I've been developing. So far, I can distribute this lib to Linux and Windows systems. Since I use GitLab to build, test and package my lib, I'd like to have my Windows builds running faster and I have no clue on how to do that.
Currently, I use the following script for my Windows builds:
.windows_template:
tags:
- windows
before_script:
- choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
- choco install python --pre -y
- choco install git -y
- $env:ChocolateyInstall = Convert-Path "$((Get-Command choco).Path)\..\.."; Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"; refreshenv
- python -m pip install --upgrade pip
- pip install conan monotonic
The problem
Any build with the script above can take up to 10 minutes; worse: I have two stages, each one taking the same amount of time. This means that my whole CI pipeline will take 20 minutes to finish because of slowness in Windows builds.
Ideal solution
EVERYTHING in my before_script can be cached or stored as an image. I only need some hints on how to do it properly.
Additional information
I use the following tools for my builds:
CMake: to support my building process;
Python3: to test and build packages;
Conan (requires Python3): to support the creation of packages with several features, as well as distribute them;
Git: to download Googletest in CMake configuration step This is already provided in the cookbooks - I might just remove this extra installation step in my before_script;
Googletest (requires Python3): testing library;
Visual Studio DEV Tools: to compile the library This is already in the cookbooks.
Installing packages like this (whether it's OS packages though apt-get install... or pip, or anything else) is generally against best practices for CI/CD jobs because every job that runs will have to do the same thing, costing a lot of time as you run more pipelines, as you've seen already.
A few alternatives are to search for an existing image that has everything you need (possible but not likely with more dependencies), split up your job into pieces that might be solved by an image with just one or two dependencies, or create a custom docker image to use in your jobs. I answered a similar question with an example a few weeks ago here: "Unable to locate package git" when running GitLab CI/CD pipeline
But here's an example Dockerfile with Windows:
# Dockerfile
FROM mcr.microsoft.com/windows
RUN ./install_chocolatey.sh
RUN choco install cmake.install -y --installargs '"ADD_CMAKE_TO_PATH=System"'
RUN choco install python --pre -y
RUN choco install git -y
...
The FROM line says that our new image extends the mcr.microsoft.com/windows base image. You can extend any image you have access to, even if it already extends another image (in fact, that's how most images work: they start with something small, like a base OS installation, then add things needed for that package. PHP for example starts on an Ubuntu image, then installs the necessary PHP packages).
The first RUN line is just an example. I'm not a Windows user and don't have experience installing Chocolatey, but you'd do here whatever you'd normally do to install it locally. The rest are for installing whatever else you need.
Then run
docker build /path/to/dockerfile-dir -t mygroup/mytag:version
The path you supply needs to be the directory that contains the Dockerfile, not the Dockerfile itself. The -t flag sets the image's tag after it's built (though you can do that with a separate command, docker tag too).
Next, you'll have to log into whichever registry you're using (Docker Hub (https://docs.docker.com/docker-hub/repos/), Gitlab Container Registry (https://docs.gitlab.com/ee/user/packages/container_registry/), a private registry your employer may support, or any other option.
docker login my.docker.hub.com
Now you can push the image to the registry:
docker push my.docker.hub.com/mygroup/mytag:version
You'll have to review the information in the docs about telling your Gitlab runner or pipelines how to authenticate with the registry (unless it's Public on Docker Hub or you use the Gitlab Container Registry) https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-an-image-from-a-private-container-registry
Once all that's done, you can use your new image in your CI jobs, and everything we put into the image will be ready to use:
.windows_template:
image: my.docker.hub.com/mygroup/mytag:version
tags:
- windows
...
I hope I'm clear in my question, if not please tell me.
I am using OpenImageIO's python bindings (pybind11) for some scripts that will run on hundreds of computers. Unfortunately it took me a lot of time to install OpenImageIO and make it work with my Python2 installation. I'd like to know if there's a way to create a file/folder that I could send to other computers so they can install the Python module simply with "pip install file/folder".
Thanks ofr your help
Are you running the scripts on a compute cluster with a shared filesystem? If so, then there's no need to create separate installations of python for each machine. The simplest solution is to create ONE python environment in a location that is accessible by all of your machines. An easy way to create a Python environment in a non-system location is to use Miniconda. Install it to a shared (network) location, and create an environment for all of your machines to use.
If your machines do NOT have a shared file system, then you'll need to somehow reproduce the environment on all of them independently. In this case, there's no simple way to do that with pip.**
But if you can use conda instead, then there's a very straightforward solution. First, install everything you need into a single conda environment. Then you have a choice: You can export the list of conda packages, or simply copy the entire conda environment directory to the other machines.
OpenImageIO is available from the conda-forge channel, a community-developed repository of conda packages. The name of the package is py-openimageio. They have stopped updating the python-2.7 version, but the old versions are still available.
Here's how to do it.
Install Miniconda-2.7
Create a new environment with python 2.7, OpenImageIO, and any other packages you need:
conda create -n jao-stuff -c conda-forge py-openimageio python=2.7
conda activate jao-stuff
python -c "import OpenImageIO; print('It works!')"
Do ONE of the following:
a. Export the list of packages in your environment:
conda env export -n jao-stuff -f jao-stuff-packages.yaml
Then, on the other machines, install Miniconda, then create the environments using the package list from the previous step:
conda create -n jao-stuff --file jao-stuff-packages.yaml
OR
b. Just copy all of the files in the environment to the other machines, and run them directly. Conda environments are self-contained (except for a few low-level system libraries), so you can usually just copy the whole thing to another machine and run it without any further install step.
tar czf jao-stuff.tar.gz $(conda info --prefix)/envs/jao-stuff
On the other machine, unpack the tarball anywhere and just run the python executable it contains:
tar xzf jao-stuff.tar.gz
jao-stuff/bin/python -c "import OpenImageIO; print('It works!')"
**That's because OpenImageIO is a C++ project, with several C++ dependencies, and they don't provide binaries in the wheel format. I don't blame them -- pip is not well suited to this use-case, even with wheels. Conda, on the other hand, was designed for exactly this use-case, and works perfectly for it.
Following these instructions to install the DataStax C++ Driver on MacOS High Sierra, as a pre-requisite to installing the DataStax PHP Driver for Cassandra.
Everything runs great until I get to the line "make install" in the "Building and installing the C/C++ driver" section. That's where I get the message: "make: *** No rule to make target `install'. Stop."
Can someone help me get past this step?
** SOLVED ** a friend helped me stumble across the solution. Two things to remember when installing on MacOS High Sierra:
1.) You need to run the install of cpp-driver (which isn't a step in the DataStax instructions referenced in the question) and then
2.) You have to fully qualify the cmake .. command to point to the OpenSSL install.
Here are the amended instructions that worked for me:
# Datastax C++ driver dependencies
brew install libuv cmake
brew install openssl
brew link --force openssl
# Install git if you dont have it
brew install git
# Retrieve the cpp
git clone https://github.com/datastax/cpp-driver.git --depth=1
mkdir cpp-driver/build
cd cpp-driver/build
# Build with qualified path to OpenSSL location
cmake -DOPENSSL_ROOT_DIR=/usr/local/opt/openssl/ -DOPENSSL_LIBRARIES=/usr/local/opt/openssl/lib ..
make
make install
# Install pecl driver
pecl install cassandra
Once that's complete you should be good to go.
I try to install the package "data.table" (and "aws.s3)" via Rstudio Server on an Amazon Linux instance following this guide:
http://stanke.co/category/r/
Unfortunately, I get the following error message. I really don't know what else to do.
Can anybody help? I installed devtools and I am able to install other packages such as xml2, devtools and deplyr.
I had the same issue on AWS and already fixed.
You need first install gcc64 and openmp shared support library.
sudo yum install gcc64
sudo yum install libgomp
Then under your user home create an .R folder with a Makevars file in it, with the following content (it will tell to R which compiler to use):
CC = /usr/bin/gcc64
CXX = /usr/bin/g++
SHLIB_OPENMP_CFLAGS = -fopenmp
I hope it's working for you as well ...
You need to install dmlc-core.
This link will provide more information:
A common bricks library for building scalable and portable distributed machine learning
based on https://github.com/RcppCore/RcppArmadillo/issues/200, I think this issue is due to a g++ compatability issue. It might also explain why when I installed devtools it kept giving me [-Wdeprecated-declarations]
so run:
sudo yum remove gcc72-c++.x86_64 libgcc72.x86_64
yum install R-devel
Then you should be able to run the installation command.
I am trying to reproduce my development environment in a docker image. I am able to get simple dependencies met such as python+a couple standard packages, largely through the builds from docker hub. But when it comes to installing xgboost or pandas I am having great difficulty.
After looking into the error messages it looked like I had the wrong version of g++ installed. The build had 4.7, but xgboost requires 4.9+. As I tried to update g++ I kept running into a wall where I couldn't update g++ because I needed another package (apt-add-repository), but to install that package I needed another (apt-utils) etc.
Does anyone have any general advice with setting up a Docker image or for this specific problem of upgrading the g++.
Here is the Docker file:
FROM continuumio/anaconda
MAINTAINER maintainer
RUN apt-get install -y g++-4.9
One test would be to start from a gcc:4.9 image (which uses wheezy), and try to add what anaconda Dockerfile does.
That way, you start from an image with the right gcc.
You first need to make sure your source list is up-to-date. The line with RUN command in the dockerfile should be
RUN apt-get update && apt-get install -y g++