Can someone help on this how to automate the ci/cd pipeline to update and create new files in databricks repos from bitbucket repositories..
If you want to sync changes from BitBucket repository into Databricks Repos, then you have following possibilities:
Use databricks repos update command of the databricks-cli package, like this:
databricks repos update --path /Repos/user/repository --branch <branch_name>
Use Update command of Repos API, but it's too low-level because it doesn't work with paths, and you need to know Repository ID that could obtained via Workspace API. So it's several commands instead of single one:
curl -s -n -X GET -o /tmp/staging-repo-info.json "$DATABRICKS_HOST/api/2.0/workspace/get-status" -H "Authorization: Bearer $DATABRICKS_TOKEN" -d '{"path":"/Repos/Staging/databricks-nutter-projects-demo"}'
export STAGING_REPOS_ID=$(cat /tmp/staging-repo-info.json|grep '"object_type":"REPO"'|sed -e 's|^.*"object_id":\([0-9]*\).*$|\1|')
curl -s -n -X PATCH -o "/tmp/$(Build.SourceBranchName)-out.json" "$DATABRICKS_HOST/api/2.0/repos/$STAGING_REPOS_ID" \
-H "Authorization: Bearer $DATABRICKS_TOKEN" -d "{\"branch\": \"$(Build.SourceBranchName)\"}"
P.S. You can find end-to-end CI/CD demo with Repos and Azure DevOps in this repository. Although it's not a BitBucket, but the structure of the pipeline remains the same.
I have a maven Artifact Registry and am able to add dependency in pom.xml and get the jar.
I have another usecase where I would like to only download the jar using CLI something which you can easily do with other external maven repos eg curl https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime/0.7.0-incubating/iceberg-spark-runtime-0.7.0-incubating.jar --output temp.jar
I don't see any instructions about how to do this.
I needed this too.
I have configured a service account following gcp guide
Then, I have executed the following command to get authbasic credz :
gcloud artifacts print-settings gradle \
[--project=PROJECT] \
[--repository=REPOSITORY] \
[--location=LOCATION] \
--json-key=KEY-FILE \
[--version-policy=VERSION-POLICY] \
[--allow-snapshot-overwrites]
In the output you have the artifactRegistryMavenSecret.
Finally you get your artifact with :
curl -L -u _json_key_base64:{{ artifactRegistryMavenSecret }} https://{{ region }}-maven.pkg.dev/{{ projectId }}/{{ repository }}/path/of/artifact/module/{{ version }}/app-{{ version }}.jar -o file.jar
It seems like this feature as mentioned does not exist yet for Artifact Registry based on this open feature request (this feature request has currently no ETA). However, you can try to implement a Cloud build automation not to only save your built artifact in Artifact Registry, but also to store them in Google Cloud Storage or other Storage repositories; so you can easily access the JARs (since Cloud Storage supports direct downloading).
In order to do this, you would need to integrate Cloud Build with Artifact Registry. The documentation page has instructions to use Maven projects with Cloud Build and Artifact Registry. In addition, you can configure Cloud Build to store built artifacts in Cloud Storage.
Both of these integrations are configured through a Cloud Build configuration file. In this file, the steps for building a project are defined, including integrations to other serverless services. This integration would involve defining a target Maven repository:
steps:
- name: gcr.io/cloud-builders/mvn
args: ['deploy']
And a location to deploy the artifacts into Cloud Storage:
artifacts:
objects:
location: [STORAGE_LOCATION]
paths: [[ARTIFACT_PATH],[ARTIFACT_PATH], ...]
Additional to #Nicolas Roux's answer:
artifactRegistryMavenSecret is basically an encode64 of the Service Account json key.
So instead of runnig gcloud artifacts print-settings gradle and curl -u _json_key_base64:{{ artifactRegistryMavenSecret }}, another way is you can directly use the token from gcloud auth print-access-token, then apply this token to cURL.
For example:
1. gcloud auth activate-service-account SERVICE_ACCOUNT#DOMAIN.COM \
--key-file=/path/key.json --project=PROJECT_ID
2. curl --oauth2-bearer "$(gcloud auth print-access-token)" \
-o app-{{ version }}.jar \
-L https://{{ region }}-maven.pkg.dev/{{ projectId }}/{{ repository }}/path/of/artifact/module/{{ version }}/app-{{ version }}.jar
By that, if you're working with Google Auth Action (google-github-actions/auth#v0) in Github Actions Workflow, then you can easily run the curl command without needing to extract artifactRegistryMavenSecret.
I am having the same problem as described in this jira about CORS headers. It has been fixed, but the release is showing as Fixed with r205117 (the commit).
Is there a way to obtain service packs, or do I have to build the product myself ?
As #Asela said you can build this fix for yourself.
Personally I found the instructions listed in the documentation rather lacking so thought I'd post an update on how I managed to patch it. Once I found a way to get this to work it was simple, but took a while to discover how to.
Download source
As per docs, but be aware it's a 4G+ download
svn checkout https://svn.wso2.org/repos/wso2/carbon/platform/tags/turing-chunk11 ~/wso2.chunk11
Patch code
cd to ~/wso2.chunk11/components/apimgt/org.wso2.carbon.apimgt.gateway/1.2.2/
edit src/main/java/org/wso2/carbon/apimgt/gateway/handlers/Utils.java
Build JAR
I used mvn clean compile install
Patch product
cd to product home, in my case ~/wso2am-1.7.0
cd to patches ./repository/components/patches/
mkdir and cd for the patch, in my case mkdir patch0009 ; cd patch0009
copy in the new jar there cp ~/wso2.chunk11/components/apimgt/org.wso2.carbon.apimgt.gateway/1.2.2/target/org.wso2.carbon.apimgt.gateway-1.2.2.jar .
start the product and the patch should apply.
Test patch
Assuming you've added the '*' to ~/wso2am-1.7.0/repository/conf/api-manager.xml:
<Access-Control-Allow-Origin>*</Access-Control-Allow-Origin>
You can just curl an API and see the correct header:
curl -i -X OPTIONS --header 'Content-Type: application/json' --header 'Origin: http://somewhere.com' http://192.168.1.64:8280/myApi
An check the correct domain setting is returned:
Access-Control-Allow-Origin: *
It has been fixed in the APIM 1.8.0 which has not been released yet. Therefore you may need to wait for 1.8.0 official release. I think, if you can obtain support from WSO2, they would give it as a patch. If not, you have following options.
You can find the source change with r205117 and apply it to the older APIM version (say APIM 1.7.0). To do that you need to build the relevant jar file with above diff. Ordinal source for jar file can be found at here and Source diff can be found at here. You can apply the source diff and build the jar file, then replace the orignal jar file which can be found in the APIM 1.7.0
You can build the 1.8.0 product by yourself. It would be little bit hard to build all the product and get a new pack of 1.8.0.
Every GitHub repository can have (or be) a GitHub Pages website, that can be built with Jekyll. GitHub builds the site every time you push a new commit.
Is there a way to force the refresh of the Github Pages website without pushing a new commit?
From GitHub support, 2014-06-07:
It's not currently possible to manually trigger a rebuild, without pushing a commit to the appropriate branch.
Edit:
As Andy pointed out in the comments, you can push an empty commit with the command:
git commit -m 'rebuild pages' --allow-empty
git push origin <branch-name>
Edit 2:
Thanks to GitHub Actions, it's fairly easy to trigger a daily publish: https://stackoverflow.com/a/61706020/4548500.
If you want a quick script solution, here it is. Just do the following tasks only once, and run the script whenever you want to rebuild your GitHub page.
1. Create a personal access token for the command line:
Follow the official help here to create a personal access token. Basically, you have to log in your GitHub account and go to: Settings > Developer settings > Personal access tokens > Generate new token.
Tick repo scope.
Copy the token.
2. Create the following script:
Create a file called RebuildPage.sh and add the lines:
#!/bin/bash
curl -u yourname:yourtoken -X POST https://api.github.com/repos/yourname/yourrepo/pages/builds
Here,
Replace yourname with your GitHub username.
Replace yourtoken with your copied personal access token.
Replace yourrepo with your repository name.
3. Run the script:
If you use Windows 10:
You need to setup Windows Subsystem for Linux, if not already done. Follow this to do so.
Remove the first line (#!/bin/bash) from the script and save the script as RebuildPage.bat. (i.e., replace .sh with .bat in the script file name)
Alternative to the above point: To get the double-click feature for running the .sh file:
Set bash.exe as the default program for .sh files.
Open regedit.exe and edit HKEY_CLASSES_ROOT\Applications\bash.exe\shell\open\command. Set the (Default) value to:
"C:\Windows\System32\bash.exe" -c " \"./$(grep -oE '[^\\]+$' <<< '%L')\";"
Now double-click the script wheneven you want to rebuild your GitHub page. Done!
If you use Linux/Mac, running the script is as same as running other scripts. Done!
Additional notes for the solution:
This solution utilizes a API of GitHub REST API v3. Here is the official documentation for the API.
Now that GitHub Actions are available, this is trivial to do:
# File: .github/workflows/refresh.yml
name: Refresh
on:
schedule:
- cron: '0 3 * * *' # Runs every day at 3am
jobs:
refresh:
runs-on: ubuntu-latest
steps:
- name: Trigger GitHub pages rebuild
run: |
curl --fail --request POST \
--url https://api.github.com/repos/${{ github.repository }}/pages/builds \
--header "Authorization: Bearer $USER_TOKEN"
env:
# You must create a personal token with repo access as GitHub does
# not yet support server-to-server page builds.
USER_TOKEN: ${{ secrets.USER_TOKEN }}
Sample repo that does this: https://github.com/SUPERCILEX/personal-website/actions
Pages API: https://developer.github.com/v3/repos/pages/#request-a-page-build
I had this problem for a while, and pushing to master branch didn't change anything on myapp.github.io, for two reasons :
1 - Build
No matter how many time I tried to push my work on master, build would not start. I found a workaround by modifying my file in Github online editor (open your index.html and edit it on Github website, then commit)
2 - Caching issues
Even after a successful build, I would still see the exact same page on myapp.github.io, and hard reloading with Ctrl + Shift + R wouldn't solve it. Instead, if using Chrome, inspect your page, head into the Application tab, select "Clear storage" in the left menu, and click on "Clear site data" at the bottom of the menu.
Even after I pushed my changes to GitHub repository, I was not able to see the changes today. Then I checked my repository settings for more information, there I could see, all these times the build was failing and that was the reason I was not able to see the changes.
You may also see a message as "Your site is having problems building: Unable to build page. Please try again later."
Then I was checking my recent commits and tried to find out what causes this issue. At the end I was able to fix the issue.
There was an additional comma in the tags (,) and that caused this issue.
You will not get relevant error messages if there are any issues in your .md file. I recommend you to check for the build status and compare the changes if you are facing the same issue.
This is doable as of v3 of the GitHub API, though it is currently in preview
https://developer.github.com/v3/repos/pages/#request-a-page-build
POST /repos/:owner/:repo/pages/builds
The empty commit didn't work for me, but based on #benett answer, this worked for me:
Open Postman, create a new request with this URL: https://api.github.com/repos/[user_name]/[repo_name]/pages/builds (replace with your name and repo), and select POST method.
Before you run it, go to the headers tab and add a new key Accept with the value application/vnd.github.mister-fantastic-preview+json
Now you can run it and visit your pages again.
I was having trouble refreshing even though my Github Actions was showing that my site has been deployed.
Toggling the publishing source did the trick for me. I switched the publishing source from master to content and then back to master. You can check how to change the publishing source of the branch here
I went through the same problem, to solve it I developed a githu action that works with scheduler and supports updating multiple gh-pages at the same time.
https://github.com/marketplace/actions/jekyll-update-github-pages-without-new-commit, the action update gh-pages without generate new commits.
name: Update all github pages
on:
schedule:
- cron: "30 0 * * *"
jobs:
github-pages:
runs-on: ubuntu-latest
name: Update Github Pages Initiatives
steps:
- name: Jekyll update github pages without new commit
uses: DP6/jekyll-update-pages-action#v1.0.1
with:
DEPLOY_TOKEN: ${{ secrets.GH_PAGES_DEPLOY_TOKEN }}
USER: ${{ secrets.GH_PAGES_USER }}
FILTER: 'is%3Apublic%20org%3Adp6'
Log action
Alternative Solution
You may have received an email from GitHub telling you that Jekyll did not succeed at building your site when you pushed it to your gh-pages. If this is the case, you can try to force push to trigger another build.
If you use a dedicated folder for the final website, let's say a public folder, you can try to rebuild your folder and add the folder to your commited changes. After that, you'll need to split those file into your gh-pages branch and force them to trigger another build even if the files did not change at all. The rest of the code bellow just removes the commits for the public folder for convenience and removes it from the local filesystem.
Code
git add public
git commit -am ":bug: triggering another jekyll build"
git push origin $(git subtree split --prefix public master):gh-pages --force
git reset HEAD~1
rm -rf public
Tips
If there are uncommited changes that are not part of the final site, you can stash them with the following command.
git stash
Then do the above command to manually force the Jekyll build and unstash them.
git stash pop
References
Online Git Manual
I surmise from other answers that this was once difficult?
Go to Settings->Pages
Just under "Change theme" you'll see a link to the actual Github action labeled "pages build and deployment workflow".
Click Re-run all jobs
Hi I am having a Jenkins build pipeline like this: 1. builds the app and deploys to Artifactory; 2. runs an SSH exec command on the test server (remote) to download the artifacts and deploys them into the right directory; 3. runs web tests against the test server, if passed, changes the build status in Artifactory to something like pre-staging for further manual UAT testing. My question is, how to change the build status in Artifactory from a Jenkins job. If using Artifactory's RESTAPI is necessary, can someone share an example? Much appreciated!
Yes, REST API is the easiest way.
You need to perform a Build Promotion call. Please note it requires Artifactory Pro.
It's a POST request, accepting simple json string, in which only two properties are mandatory: status and ciUser.
The call should look something like this:
curl -X POST -u admin:password -H "Content-Type: application/json" -d '{"status":"tests passed","ciUser":"jenkinsAdmin"}' "http://localhost:8081/artifactory/api/build/promote/buildName/buildNumber"