I would like to add the following to pre-commit for a team:
- repo: https://github.com/pre-commit/mirrors-mypy
rev: 'v0.720'
hooks:
- id: mypy
args: [--ignore-missing-imports]
My team is worried that this might be too strict. To have a gradual introduction, I would like this hook not to make the commit fail, but only to show the issues. Is that possible?
you can, but I wouldn't suggest it -- warning noise is likely to have your whole team ignore the entire output and the entire tool
here's how you would do such a thing (note that it has reduced portability due to bash -- mostly because the framework intentionally does not suggest this)
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.720
hooks:
- id: mypy
verbose: true
entry: bash -c 'mypy "$#" || true' --
two pieces make this work:
verbose: true always produces the output -- this option is really only intended for debugging purposes, but you can turn it on always (it can be noisy / annoying though)
bash + || true -- ignore the exit code
disclaimer: I am the author of pre-commit
Also note that you can temporarily disable hooks by setting the environment variable SKIP. For example:
SKIP=flake8 git commit -m 'fix thing - work in progress'
This is especially useful when you just want to make local "checkpoint" commits that you'll fix later.
Side note on mypy specifically: there's a potentially big issue with using mypy in a non-blocking way like this. If you allow commits with type errors to be merged, everyone else will start to see those type errors in their pre-commit checks.
When developers are making further changes, it's confusing whether the mypy errors that appear were there from before, or due to their further changes. This can be a recipe for frustration/confusion, and also for allowing further type errors to accumulate.
I think the mypy guide on using mypy with an existing codebase is pretty good advice.
If you just need to temporarily skip mypy checks so you can checkpoint your work, push a PR for initial review, or whatever, you can just do SKIP=mypy as mentioned above.
Related
We have multiple release branches in our product (currently this is unavoidable). For this question, suppose we have two:
master
release/r-858
Both have classic CI builds. Now I want to replace them with YAML builds. Our requirement is simple - have two distinct build definitions pointing to a YAML script - one for master, one for release/r-858
At the beginning I thought it is a trivial exercise:
Create YAML build script in master. Set the CI trigger to master.
Cherry-pick (never mind why not merge) to release/r-858 - set the CI trigger to release/r-858.
Not ideal, because the two scripts only differ in their CI trigger. But "I am learning, it is good enough for now" saying me to myself.
However, this simple scheme does not work! The build I created for release/r-858 is triggered on changes in master!
I double check every setting I know about builds - all look correct.
Please, observe:
The master build
The release/r-858 build
Uh oh, look at that. It shows the YAML on the master branch! Well, maybe it is an innocent presentation bug? Let us check the branch I am supposed to build from:
Yup, the file is different - I am playing with the trigger trying to solve the very same problem this question is about. The original code had release/r-858 instead of $(Build.SourceBranch) as the CI trigger, but since it did not help I started playing with all kinds of trigger values.
To remove any doubt, here is the proof the branch corresponds to release/r-858:
C:\xyz\58 [arch/shelve/798914 ≡ +0 ~17 -0 !]> git lg -2
cfdb6a9a86a | (HEAD -> arch/shelve/798914, origin/arch/shelve/798914) Rename azure-pipelines-ci-58.yml to azure-pipelines-ci.yml and sync it with the master version (68 seconds ago) [Kharitonov, Mark] (2020-08-14 09:09:46 -0400)
a931e3bd96b | (origin/release/r-858, release/r-858) Merged PR 90230: 793282 Work Assignments Merge (28 minutes ago) [Mihailichenco, Serghei] (2020-08-14 12:02:20 -0400)
C:\xyz\58 [arch/shelve/798914 ≡ +0 ~17 -0 !]>
Anyway, more build properties:
The problem
So a developer pushed some code to master and now the release/r-858 build is running:
Why is this? One of our guys asked a similar question in the Microsoft Developer Community forum, but that thread does not make sense to me.
What am I doing wrong here?
Edit 1
Imagine a big enterprise monolithic application. It is deployed in production at version 858. At the same time, developers work on the next version and also hot fixes and service packs for the version already deployed in prod.
A change can be made only in master or only in release/r-858 or in both (not at the same time, though). Many teams are working at the same time on many different aspects of the application and hence QA has many pods where the application is deployed. As I have mentioned above - about 150 pods for the bleeding edge (master) and about the same amount for the already released code, because there is active work to test hot fixes and service packs.
I appreciate this arrangement is not ideal. It is such not because we love it, but because one has to deal with decade old decisions. We are working to change it, but it takes time.
Anyway, the current process is to have 2 build definitions (in reality there are more for different reasons). So far we used classic CI builds, now we want to migrate to YAML (which we already use for micro services, but not the monolith).
Now I understand that we can have different release pipelines based off the same build definition, but different branch filters.
And maybe we will. But I do not understand why it is wrong to have different build definitions here, given that each branch is a long living release branch.
Edit 2
You can ignore $(Build.SourceBranch) and imaging release/r-858 instead. The net result is exactly the same. In the scenario I bring above code is committed to master, not release/r-858.
Edit 3
It is very confusing. Suppose I am creating a new YAML build. The dialog says "select YAML in any branch", but they point is that once selected this branch becomes the default branch of the build. That is the branch we can see here:
If I have a single YAML file in the master branch, the build with the default branch release/r-858 cannot even use it, unless it is merged to release/r-858. I tried it - I:
created a new YAML build
selected the YAML file from the master branch
ran and right away cancelled the build
then went to edit the build and changes the branch of the build from master to release/r-858 - it allowed me to save the build, even if the YAML does not exist in that branch
But then when I tried to run the build again I got this:
An error occurred while loading the YAML build pipeline. File /Build/azure-pipelines-ci.yml not found in repository bla-bla-bla branch refs/heads/release/r-858 version 5893f559292e56cf6db48687fd910bd2916e3cef.
And indeed, looking at the raw build definition, the process section contains the YAML file path, but not the branch:
"process": {
"yamlFilename": "Build/azure-pipelines-ci.yml",
"type": 2,
"resources": {},
"target": null
},
The branch only appears in the repository section of the definition:
"repository": {
"defaultBranch": "refs/heads/release/r-858",
...
},
It is clear to me that a single build definition can be used to CI build many branches. But this model I need to implement is build definition per release branch. I cannot have a single build definition for the following reasons:
Different release branches have different agent pools, because of the different development intensity. Remember, this is on on-prem Azure DevOps Server with self hosted agents. Can we express this requirement with a single build definition?
Different build variable values which we want to control without sending a Pull Request to YAML file repository. How do you do it with a single build definition? For example, one of the variables controls the version Major.Minor. They are different in each release branch.
So, I do not see any way to avoid multiple build definitions in our situation. The root cause for this are the release branches, but we cannot throw them away in the near future.
So, we have 2 build definitions. That forces us to have 2 YAML - one per branch, because a build definition with the default branch of release/r-858 expects to find YAML in that branch, otherwise we cannot trigger the build manually. Which is a must, even if the build has a CI trigger.
So, 2 build definitions, 2 YAMLs (one per branch). So far my hands were forced. But now I am told that the release branch build would be triggered by the master YAML just because the release branch build is linked to the same YAML file name ignoring the default branch of the build!
Because this is what happens - a commit is checked in to master and the release branch build is invoked in addition to the master branch build! Both build definitions build exactly the same branch (master) using the master YAML script. But because the release branch build has different set of variables the end result is plain wrong.
This is not reasonable. I am going to create a dummy repo to reproduce it cleanly and post here.
Edit 4
As promised - a trivial reproduction. Given:
master branch build test-master-CI
release branch build test-r58-CI
Since having two build definitions necessarily means two YAMLs (one per branch), here they are:
C:\xyz\DevOps\Test [master ≡]> cat .\azure-pipelines.yml
trigger:
branches:
include:
- master
name: $(BuildVersionPrefix).$(DayOfYear)$(Date:HH)
steps:
- script: echo master
C:\xyz\DevOps\Test [master ≡]> git co release/r-858
Switched to branch 'release/r-858'
Your branch is up to date with 'origin/release/r-858'.
C:\xyz\DevOps\Test [release/r-858 ≡]> cat .\azure-pipelines.yml
trigger:
branches:
include:
- release/r-858
name: $(BuildVersionPrefix).$(DayOfYear)$(Date:HH)
steps:
- script: echo release/r-858
C:\Dayforce\DevOps\Test [release/r-858 ≡]>
Where BuildVersionPrefix = 59.0 for master and 58.3 for release/r-858
When I trigger each build manually I get this:
Now I commit a change to master. Lo and behold - both builds are triggered:
In both cases the YAML from the master branch is used. BUT the release branch defines BuildVersionPrefix = 58.3 and so the master build executed by the release branch build definition has bogus version.
Is this really how the feature is supposed to work? That makes the CI YAML trigger useless for my scenario. Thank you Matt for helping me to realize that.
I think I get where the confusion comes from. When you are configuring the pipeline, you are specifying the branch (notice the description says the file in any branch) and the file name.
What you are doing is just duplicating the monitoring though. If you were to really inspect it, I think you will see that when you push to release branch, it isn't trigger the master YAML pipeline ... it is just triggering the release YAML steps a second time. That is because the pipeline is just monitoring changes to the repo and responding based on the YAML configuration. In this case, you pushed to release and it evaluated that there was a YAML that matched that trigger (the release branch's copy) and triggered for both build definitions.
I verified this on a mocked-up pipeline. I had selected different branches on the creation, but the only thing that really impacts I believe is the default branch it would use for scheduled builds. I created a simple echo statement in both of these it was using the release branches YAML configuration.
I think if you really want to achieve the desired results you are expecting, you will want to use the override triggers that you define on the definition instead of relying on what is in the YAML trigger.
I had the same issue and Matt helped me solve this.
I'm only writing this as the only way to get this working for me was to create a build YAML file on one branch (with the correct configuration). Then create the other YAML file on another branch. And then create the pipelines in the new shiny YAML editor within Devops.
The key is, when in the "Configure" section of a new pipeline, select:
"Existing Azure Pipelines YAML file" which allows you to select a branch and a YAML file within that branch.
This allowed me to have the SystemOne branch build and test the system one site and the SystemTwo branch build and test the system two site.
I also added triggers inside the SystemOne.yml using a wild card. EG
trigger:
batch: true
branches:
include:
- SystemOne/*
And the same for the SystemTwo.yml.
This question already has answers here:
How do I run a code formatter over my source without modifying git history?
(6 answers)
Closed 4 years ago.
I have a project with a rich git history from multiple users, it is never been auto-formatted and I'd like to run clang-format on it. It is important to preserve git history.
Some examples of what I mean.
When there was a block of code from Joe and then 'a+b' was converted to a + b. It stays the Joe's line in git blame.
When there was
void foo()
{
return k;
}
and it was formatted to
void foo() { return k; }
It still stays the Joe's code.
etc.
Any known solutions?
Basically, you cannot have it both ways (that is, preserve the history and auto-format wholesale), but you have some options.
1) You could in principle re-create the repository by replaying all the commits one-by-one with auto-formatting applied, but this new repository will be a different one: all the commits SHAs will be different. Some conflicts are possible, especially in case of non-linear history (merges). This may not be a trivial and completely automatic operation.
2) You can also just apply the formatting as a new commit (single, huge one), but this will make using git blame harder.
3) You can also apply auto-formatting on the go as the files are modified during development. This will have somewhat smaller impact, but is not perfect either.
There are two aspects of "preserving history", both of which can be accomplished with git:
Make changes on top of existing history: this will emburden a git blame with a one single commit of formatting changes. This is easy and no different from your normal workflow. I recommend one single commit and then enforcing formatting as a pre-merge and pre-commit hook on the server and client sides.
Rewrite all history, individually formatting each commit. This makes any references to the pre-rewrite hashes of commits etc. invalid, but gives you a clean git blame/log. This needs some magic using (maybe low-level) git commands to make this happen. Or a "simple" "interactive" rebase where you amend each commit with the formatting changes (although keeping the original committer might rake some extra magic).
A last option is fixing code up as it is committed. This leads to a horrible disarray of styles within a single file and doesn't help anything.
I have a clojure script that I've been running for several years now.
Recently I had to do a reinstall of leiningen and java 8.
When I run the code I now get multiple StackOverflowErrors on third party libraries.
E.g.
StackOverflowError org.apache.commons.math3.util.FastMath.cosQ (FastMath.java:1850)
and
StackOverflowError clojure.lang.Numbers$DoubleOps.combine (Numbers.java:571)
I would post code but it happens now at multiple points in my code and on third party libraries??
without a stack trace and some code, a specific anser will be hard to find, so here is a general method for these things:
Part 1 "what was running before":
recreate your old environment, by digging through the logs, reverting to a backup etc. then run:
lein deps :tree 2>&1 > old-lein-dependencies
the 2>&1 part ensures that the version range and conflict warnings
are included in the output.
Mark down the leiningen version and java version:
lein version
Part 2: "what is running now"
Repeat the steps and record the same information as before:
lein deps :tree 2>&1 > new-lein-dependencies
lein version
java -version
Part 3: Diff and Compare
pick through all the differences
diff -u old-lein-dependencies new-lein-dependencies
there will be a big block of differences at the top where lein prints all the important warnings. The final clue is almost always here, though it's often not easy to recognise up right away.
Part 4: Do Science
go through every version change, starting from the initial configuration by pinning the versions in the project.clj until you find the change that breaks things. A convenient way to pin these is with the :managed-dependencies block in the project.clj file. It looks something like this:
:managed-dependencies [[http-kit "2.3.0-alpha4"]]
and repeat the process of switching out versions till you get a handle on where the change was introduced. For me this has almost always been the result of using a version range in a dependency rather than a specific version. I'm not too enthusiastic about version ranges anymore :-/
So I could not compare the previous setup as it was on a machine that was wiped clean.
I found that an error had crept into one of the math formulas executed by the tool which basically called itself repeatedly resulting in the stack overflow error.
The issue: sometimes, but not every time, Git deletes the static directory of a repo. We're not sure what triggers it, but it appears to happen when either merging between branches or sometimes even just checking out branches. It does this without asking, and eats tracked files.
The background:
I have a (private) project which has a few branches, 'release', 'develop', multiple feature lines.
There are two of us (me and #stevejalim) working on the repo. This problem happens to both of us.
I am using purely the command line for my git commands; Steve is using a mixture of the command line and Git Tower.
It's a Django project with a static directory. We may have git rmed the static directory at some point in the past, or put it in .gitignore, but not recently. And the head of our develop branch doesn't have static in .gitignore and has files in static tracked.
This happens so infrequently that we're not sure if it's something we're doing, or an intermittent problem, a bug with Git, or a corrupted tree
It might happen only when merging another branch back into develop. But branches are always branched from develop and back into develop. But we're not sure.
We are using git-flow, but the issue happens when using non-git-flow commands, too.
As examples of when this can strike:
1) Steve had a develop branch that was clean (no changes to commit or stage) and stable. He cut a new release with git flow release start|finish and in the process (possibly the back-merge from master to develop), the entire /static/ tree got deleted.
2) Steve repaired the delete by discarding the changes (to essentially undelete the file). But then, Steve simply switched from master back to develop and the /static/ dir got zapped again (this was with Git Tower)
3) Sometimes just merging from a feature branch to develop as an interim merge can trigger it. It does seem to happen most often when cutting a new release, though
Could it be related to how we repair the zapping of the /static/ dir? What is the best way to bulk-undelete things that have been deleted? Neither discarding local changes or a hard reset to HEAD seems to cure things. Might a rebase help us?
UPDATE We've just experienced this again simply with a git add . - no changing branches, no merging. Does that help diagnosis at all?
Here is the content of Steve's .git/config:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = git#github.com:foobarbazbam/bar.git
[branch "master"]
remote = origin
merge = refs/heads/master
[gitflow "branch"]
master = master
develop = develop
[gitflow "prefix"]
feature = feature/
release = release/
hotfix = hotfix/
support = support/
versiontag =
[difftool "tower"]
cmd = \"/Applications/Tower.app/Contents/Resources/CompareScripts/kaleidoscope.sh\" \"$LOCAL\" \"$REMOTE\"
Here is the content of .gitignore:
.DS_Store
*.pyc
*.log
*.log.*
*.bak
*~
settings_local.py
/build/
/static_collected/*
/static/uploads/*
/static/theme_files/*
/static/picture/*
pip-log.txt
*.tmproj
*.dot
*.db
*.sublime-project
*.sublime-workspace
/docs/_*
Okay ladies and gents I have a public apology to make. Git was not to blame. I will leave this question here as a lesson to other people who may pass the same way.
We were using the django-storages backend (a 'plugin' to enable Django to store files on Amazon S3 transparently). This has a test called HashPathStorageTest. The tear-down this test deletes settings.MEDIA_ROOT, which was set to ./static. This is faulty, in my opinion. It has no business blanket-deleting files that it didn't create.
We were running our tests, like good citizens, before checking in. Most of the time we ran only tests for our code, but occasionally we ran tests for the whole project (including 3rd party plugins). This was producing the behaviour in the question. Because we ran test and git things together, it wasn't easy to pin down which command was doing the deleting (and the deleted files only showed up when we ran git status).
So problem solved. Again, sorry for casting aspersions on the good name of Git!
This would appear to be git deciding to merge to a branch which A) has the folder deleted B) it thinks is more recent than the branch you are merging to. Might there be a machine which has american date type and one which has european date type? Or does any of your testing involve resetting machine date time?
I would
a) check the regions of all machines included in your developement.
b) choose a fixed point in your development, copy the files out and create a new reprository and go from there.
Ok so this just happened - I'm adding it a separate answer because its so silly. We have a repositiory for our SQL scripts and a new developer accidentally edited .gitignore to contain *.sql - needless to say changes stopped being propagated on that branch - and fortunately it was caught prior to any signifficant loss of development time. However its not inconcievable that this could produce the issues above - so have you checked your git directive files? If the folder is ignored (and .gitignore is in there as well - i.e ignored) it could dissapear and reappear almost by magic.
The only thing I care about is C++ file. (Don't worry about binary files, text, etc ... you can assume taht everything is C++ code_).
I have these branches:
* dev
master
Now, I can to create a new branch "magic", where branch magic is equiv to "dev" (in terms of C++ code generated), but minimizes useless whiteline diffs (like inserting extra newlines) from master.
Is this possible?
Thanks!
Your question isn't completely clear to me. I think you want the new branch to contain series of commits that are "equivalent" to the ones in dev but don't contain unnecessary whitespace changes.
The simplest way to do this is with git rebase --interactive. This allows you to edit a series of commits manually. You can get the hash of the first ("root") commit with git rev-list HEAD | tail -n 1. If you want to edit the first commit as well, that's more difficult but there's a SO answer for that.
git checkout dev
git checkout -b magic
git rebase --interactive $(git rev-list HEAD | tail -n 1)
This brings up an editor on the list of commits in forward chronological order. You change pick to edit on the commits that you wish to change. git will then pause before processing each of these commits, allowing you to modify it with git commit --amend and then continue the rebase with git rebase --continue. You could of course run a script to clean up the files rather than editing them manually. If you want to do this based on the differences from the previous commit you would need to use something like git cat-file blob HEAD^:filename to fetch the previous revision, or use git diff HEAD^ filename.
If you want to automate the whole process then you can use git filter-branch --tree-filter script. Look at the man page for git-filter-branch for details.
If you want to prevent future commits from containing whitespace errors, you can set up a pre-commit hook to disallow them.
Once you're happy with the new branch magic you can use it to replace dev:
git branch -m dev dev-old
git branch -m magic dev
However, note that this might cause problems for people who already have a copy of dev in their own repos. It would then be better to do a merge instead.
git diff --ignore-space-change ...
You could try setting up a post-merge hook which will:
only do anything if the current branch is magic
remove all whitelines of *.cpp files
make a new commit (since a post-merge hook cannot directly affect the outcome of git merge)