I have a CDK codepipeline, which, simplified, looks something like this:
const pipeline = new CodePipeline(this, 'Pipeline', {
synth: new CodeBuildStep('InstallStep', {
commands: ['npm install -g some-package'],
}),
});
const initStep = new CodeBuildStep(`InitStep`, {
commands: ['some-package']
})
An NPM package is installed globally during synth step. Is there a way to use it in other build steps without reinstalling it again? I know I can easily pass build artifacts betwen the steps, but not sure about globally installed things.
Considering each code build step is a new compute instance, I don't think you can cache global node_modules. You may be able to instead do something like
const initStep = new CodeBuildStep(`InitStep`, {
commands: ['npx some-package']
})
Using npx to call the package avoids having to have the extra install step, assuming you call it like a cli command.
I have installed conda using miniforge. Since my mac has a m1 chip, i had to install conda using Miniforge3-MacOSX-arm64.sh, inorder to get tensorflow working. unfortunately this version (minforge/minconda arm64) doesn't have python2 for some reason. As I require python2 for another project (doesnot require tensorflow) I have decided to install anaconda3.
But now I am unaware how to switch between the two conda versions (anaconda3 and miniconda/miniforge3).
For example when I enter activate conda in the terminal, it activates the base environment of the miniforge version.
How do I activate base environment of the anaconda version. So that I can create python2 environment there (anaconda3).
According to this post, one solution is to change the content of your .zshrc file, save your changes, close and reopen your terminal. I tested on a MacBook Pro M1 where Miniforge3 and Anaconda3 are currently installed and it works. In the following, just replace --PATH-- with the path of the requested environment management system. For example, I replace --PATH-- with opt/anaconda3 for Anaconda3 and miniforge3 for .. Miniforge3.
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/username/--PATH--/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/Users/username/--PATH--/etc/profile.d/conda.sh" ]; then
. "/Users/username/--PATH--/etc/profile.d/conda.sh"
else
export PATH="/Users/username/--PATH--/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
A quick fix for switching between environments is to pick out the path you get from the output of conda env list. Here is what I get from both miniforge and miniconda:
(base) user#machine script % conda env list
# conda environments:
#
base * /Users/user/miniforge3
nmgp /Users/user/miniforge3/envs/nmgp
scphere /Users/user/miniforge3/envs/scphere
/opt/miniconda3
/opt/miniconda3/envs/gpcounts
/opt/miniconda3/envs/gpy
/opt/miniconda3/envs/test
/opt/miniconda3/envs/nmgp
/opt/miniconda3/envs/scphere
/opt/miniconda3/envs/ssdgp
To activate the miniforge environments you can use the name directly:
conda activate nmgp
To activate a miniconda environment you can use the absolute path:
conda activate /opt/miniconda3/envs/nmgp
The question
I'm trying to write a package to install libmonome, a toolkit for using the monome hardware (OSC controllers with LEDs, mostly for music).
My efforts are based on these instructions for building libmonome from source. Note that those instructions use waf, not make.
My hello world packages that use make build successfully. But my libmonome package that tries to use python waf analogously is not building:
[jeff#jbb-dell:~/nix/jbb-config/custom-packages/libmonome]$ nix-build
these derivations will be built:
/nix/store/kkf4c8l0njqdapnm2qgk6ffmybmafrpv-libmonome.drv
building '/nix/store/kkf4c8l0njqdapnm2qgk6ffmybmafrpv-libmonome.drv'...
Error: Cannot unpack waf lib into /nix/store/dk3pnwg7z9q14f4yj35y2kaqdmahnhhh-libmonome/.waf-2.0.17-6b308e91b5eb321c61bd5469cd6d43aa
Move waf in a writable directory
builder for '/nix/store/kkf4c8l0njqdapnm2qgk6ffmybmafrpv-libmonome.drv' failed with exit code 1
error: build of '/nix/store/kkf4c8l0njqdapnm2qgk6ffmybmafrpv-libmonome.drv' failed
[jeff#jbb-dell:~/nix/jbb-config/custom-packages/libmonome]$
Some maybe-helpful, maybe-redundant information
If you're cloning the repo, note that you'll need to fetch submodules to get all the libmonome code. Here's one way to do that:
git clone --recurse-submodules https://github.com/JeffreyBenjaminBrown/nixos-experiments
git checkout c20581f839f8e0fb39b2762baeea7d0a7ab10783
I already put absolute links to my code above, but in case you would rather see that code on this page, here is my default.nix file:
{...}:
with (import <nixpkgs> {});
derivation {
name = "libmonome";
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = [ git
coreutils
liblo
python2
];
# I would like to use fetchGit but haven't gotten it to work.
# src = fetchGit {
# url = "https://github.com/monome/libmonome.git";
# };
repo = ./libmonome;
system = builtins.currentSystem;
}
and here is the builder.sh script it calls:
set -e # Exit the build on any error
unset PATH # because it's initially a non-existant path
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done
cd $repo
# I've tried with python2 and python3
python2 ./waf configure
python2 ./waf
sudo python2 ./waf install
I am attempting to make two AWS Lambda functions (written in typescript). Both of these functions share the same code for interacting with an API. In order to not have to copy the same code to two different Lambdas, I would like to move my shared code to a local module, and have both my Lambdas depend on said module.
My initial attempt at staring code between the two lambdas was to use a monorepo and lerna. My current project structure looks like this:
- lerna.json
- package.json
- packages
- api
- package.json
- lambdas
- funcA
- package.json
- func B
- package.json
lerna.json:
{
"packages": [
"packages/api",
"packages/lambdas/*"
],
"version": "1.0.0"
}
In each of my package.json for my Lambda functions, I am able to include my local api module as such:
"dependencies": {
"#local/api": "*"
}
With this, I've been able to move the common code to its own module. However, I'm now not sure how to bundle my functions to deploy to AWS Lambda. Is there a way for lerna to be able to create a bundle that can be deployed?
As cp -rL doesn't work on the mac I had to come up with something similar.
Here is a workflow that works if all of your packages belong to one scope (#org):
In the package.json of your lerna repo:
"scripts": {
"deploy": "lerna exec \"rm -rf node_modules\" && lerna bootstrap -- --production && lerna run deploy && lerna bootstrap"
}
In the package that contains your lambda function:
"scripts":{
"deploy": "npm pack && tar zxvf packagename-version.tgz && rm -rf node_modules/#org && cp -r node_modules/* package/node_modules && cd package && npm dedupe"
}
Now replace "packagename-version" and "#org" with the respective values of your project. Also add all of the dependent packages to "bundledDependencies".
After running npm run deploy in the root of your lerna mono repo you end up with a folder "package" in the package that contains your lambda function. It has all the dependencies needed to run your function. Take it from there.
I had hoped that using npm pack would allow me to utilize .npmignore files but it seems that that doesn't work. If anyone has an idea how to make it work let me know.
I have struggled with this same problem for a while now, and I was finally forced to do something about it.
I was using a little package named slice-node-modules, as found here in this similar question, which was good enough for my purposes for a while. As I have consolidated more of my projects into monorepos and begun using shared dependencies which reside as siblings rather than being externally published, I ran into shortcomings with that approach.
I've created a new tool called lerna-to-lambda which was specifically tailored to my use case. I published it publicly with minimal documentation, hopefully enough to help others in similar situations. The gist of it is that you run l2l in your bundling step, after you've installed all of your dependencies, and it copies what is needed into an output directory which is then ready to deploy to Lambda using SAM or whatever.
For example, from the README, something like this might be in your Lambda function's package.json:
"scripts": {
...
"clean": "rimraf build lambda",
"compile": "tsc -p tsconfig.build.json",
"package": "l2l -i build -o lambda",
"build": "yarn run clean && yarn run compile && yarn run package"
},
In this case, the compile step is compiling TypeScript files from a source directory into JavaScript files in the build directory. Then the package step bundles up all the code from build along with all of the Lambda's dependencies (except aws-sdk) into the directory lambda, which is what you'd deploy to AWS. If someone were using plain JavaScript rather than TypeScript, they could just copy the necessary .js files into the build directory before packaging.
I realize this question is over 2 years old, and you've probably figured out your own solutions and/or workarounds since then. But since it is still relevant to me, I assume it's still relevant to someone out there, so I am sharing.
Running lerna bootstrap will create a node_modules folder in each "package". This will include all of your lerna managed dependencies as well as external dependencies for that particular package.
From then on, your deployment of each lambda will be agnostic of the fact that you're using lerna. The deployment package will need to include the code for that specific lambda and the node_modules folder for that lambda - you can zip these and upload them manually, or use something like SAM or CloudFormation.
Edit: as you rightly point out you'll end up with symlinks in your node_modules folder which make things awkward to package up. To get around this, you could run something like this prior to packaging for deployment:
cp -rL lambdas/funcA/node_modules lambdas/funcA/packaged/node_modules
The -L will force the symlinked directories to be copied into the folder, which you can then zip.
I have used a custom script to copy the dependencies during the install process.. This will allow me to develop and deploy the application with the same code.
Project structure
In the package.json file of the lambda_a, I have the following line:
"scripts": {
"install": "node ./install_libs.js #libs/library_a"
},
#libs/library_a can be used by the lambda code using the following statement:
const library_a = require('#libs/library_a')
for SAM builds, I use the following command from the lmbdas frolder:
export SAM_BUILD=true && sam build
install_libs.js
console.log("Starting module installation")
var fs = require('fs');
var path = require('path');
var {exec} = require("child_process");
if (!fs.existsSync("node_modules")) {
fs.mkdirSync("node_modules");
}
if (!fs.existsSync("node_modules/#libs")) {
fs.mkdirSync("node_modules/#libs");
}
const sam_build = process.env.SAM_BUILD || false
libs_path = "../../"
if (sam_build) {
libs_path = "../../" + libs_path
}
process.argv.forEach(async function (val, index, array) {
if (index > 1) {
var currentLib = libs_path + val
console.log(`Building lib ${currentLib}`)
await exec(`cd ${currentLib} && npm install` , function (error, stdout, stderr){
if (error) {
console.log(`error: ${error.message}`);
return;
}
console.log(`stdout: ${stdout}`);
console.log('Importing module : ' + currentLib);
copyFolderRecursiveSync(currentLib, "node_modules/#libs")
});
}
});
function copyFolderRecursiveSync(source, target) {
var files = [];
// Check if folder needs to be created or integrated
var targetFolder = path.join(target, path.basename(source));
if (!fs.existsSync(targetFolder)) {
fs.mkdirSync(targetFolder);
}
// Copy
if (fs.lstatSync(source).isDirectory()) {
files = fs.readdirSync(source);
files.forEach(function (file) {
var curSource = path.join(source, file);
if (fs.lstatSync(curSource).isDirectory()) {
copyFolderRecursiveSync(curSource, targetFolder);
} else {
copyFileSync(curSource, targetFolder);
}
});
}
}
function copyFileSync(source, target) {
var targetFile = target;
// If target is a directory, a new file with the same name will be created
if (fs.existsSync(target)) {
if (fs.lstatSync(target).isDirectory()) {
targetFile = path.join(target, path.basename(source));
}
}
fs.writeFileSync(targetFile, fs.readFileSync(source));
}
I want to install and configure DoctrineFixturesBundle and doctrine-fixtures in Symfony 2.1.4. Can anyone give me a guide.
Your question may be already in answered in DoctrineFixturesBundle issues section, anyway I will summarize the steps to install and configure DoctrineFixturesBundle in symfony 2.1.4 version.
Step 1 :
Open the composer.json and append below code in require section. This file is located in root folder of the project
{
"require": {
"doctrine/doctrine-fixtures-bundle": "dev-master",
"doctrine/data-fixtures": "dev-master"
}
}
Step 2:
Run below command in your terminal
php composer.phar update
Final Step:
append the below in your app/AppKernel.php
// ...
public function registerBundles()
{
$bundles = array(
// ...
new Doctrine\Bundle\FixturesBundle\DoctrineFixturesBundle(),
// ...
);
// ...
}
Hope it helps.