I am trying to create a version of the model and link it to my exported Tensorflow model. however it gives me the following error : health probe timeout: generic::unavailable: The fetch failed with status 3 and reason: UNREACHABLE_5xx Check the url is available and that authentication parameters are specified correctly.
I have made my SaveModel directory public and have attached service-xxxxxxxxxxxx#cloud-ml.google.com.iam.gserviceaccount.com to my bucket with Storage Legacy Bucket Reader. My service account service-xxxxxxxxxxxx#cloud-ml.google.com.iam.gserviceaccount.com has role ML Engine Admin and Storage Admin. The bucket and ml-engine are part of the same project and region us-central1. I am initialising the model version with the following config:
Python version: 2.7
Framework: TensorFlow
Framework version: 1.12.3
Runtime version: 1.12
Machine type: n1-highmem-2
Accelerator: Nvidia Tesla K-80
Accelerator count: 1
Note : I used python 2.7 for training and runtime version 1.12
Can you verify Saved model is valid by using CLI.
Check that Serving tag-sets are available in your saved model, use the SavedModel CLI:
saved_model_cli show --dir <your model directory>
Related
How do I use a custom builder image in Cloud Build which is stored in a repository in Artifact Registry (instead of Container Registry?)
I have set up a pipeline in Cloud Build where some python code is executed using official python images. As I want to cache my python dependencies, I wanted to create a custom Cloud Builder as shown in the official documentation here.
GCP clearly indicates to switch to Artifact Registry as Container Registry will be replaced by the former. Consequently, I have pushed my docker image to Artifact Registry. I also gave my Cloud Builder Service Account the reader permissions to Artifact Registry.
Using the image in a Cloud Build step like this
steps:
- name: 'europe-west3-docker.pkg.dev/xxxx/yyyy:latest'
id: install_dependencies
entrypoint: pip
args: ["install", "-r", "requirements.txt", "--user"]
throws the following error
Step #0 - "install_dependencies": Pulling image: europe-west3-docker.pkg.dev/xxxx/yyyy:latest
Step #0 - "install_dependencies": Error response from daemon: manifest for europe-west3-docker.pkg.dev/xxxx/yyyy:latest not found: manifest unknown: Requested entity was not found.
"xxxx" is the repository name and "yyyy" the name of my image. The tag "latest" exists.
I can pull the image locally and access the repository.
I could not find any documentation on how to integrate these images from Artifact Registry. There is only this official guide, where the image is built using the Docker image from Container Registry – however this should not be future proof.
It looks like you need to add your Project ID to your image name.
You can use the "$PROJECT_ID" Cloud Build default substitution variable.
So your updated image name would look something like this:
steps:
- name: 'europe-west3-docker.pkg.dev/$PROJECT_ID/xxxx/yyyy:latest'
For more details about substituting variable values in Cloud Build see:
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values
I'm Attempting to deploy a Flask (python3) App to Google App Engine using Google Cloud Build, but am getting the following error:
failed unmarshalling build config app.yaml: unknown field "runtime"
My app.yaml is as follows:
runtime: python37
resources:
memory_gb: 2.0
The weird things is that I can deploy the service correctly using the CLI, but not using the activators.
I'm currently using Bitbucket and it seems that the activator is ok (webhook is activated as soon as I push a commit to the desired branch, but the mentioned message appears and the build is unsuccessful).
I followed this tutorial: https://cloud.google.com/appengine/docs/standard/python3/building-app/writing-web-service
I want my custom Policy Information Point (PIP) to be able to connect to OpenStack Swift (an Object Storage application) thanks to this connector. The connector is able to retrieve metadata from an object, which would be sent back to the Policy Decision Point (PDP) when the right AttributeId is requested.
In our context, Swift holds information about our resources (objects here).
Below are all the steps I followed to try and use this connector in my custom PIP.
I got the connector to work locally (no integration with WSO2IS yet) following only the first 3 steps and calling the swift_test function in a debug-purposed main class.
I followed this guide to implement a custom PIP, which suggests using a database offering a JDBC driver (such as mariadb). My issue is that Swift does not offer a JDBC driver, hence the use of the openstack4j connector.
I added the needed openstack4j dependency to the maven project linked in the guide (here).
I also added the following imports to the extended class (named KMarketJDBCAttributeFinder in the guide):
package org.xacmlinfo.xacml.pip.jdbc;
import org.openstack4j.api.OSClient.OSClientV3;
import org.openstack4j.model.common.Identifier;
import org.openstack4j.model.storage.object.SwiftAccount;
import org.openstack4j.model.storage.object.SwiftContainer;
import org.openstack4j.model.storage.object.SwiftObject;
import org.openstack4j.model.storage.object.options.ObjectListOptions;
import org.openstack4j.openstack.OSFactory;
import org.wso2.carbon.identity.entitlement.pip.AbstractPIPAttributeFinder;
...
and this function to test and retrieve an object's metadata:
public void swift_test() {
OSClientV3 os = OSFactory.builderV3()
.endpoint(our_keystoneV3_url)
.credentials(our_keystone_username, password, domain_identifier)
.scopeToProject(Identifier.byName(our_tenant), Identifier.byName(our_domain))
.authenticate();
SwiftAccount account = os.objectStorage.account().get();
Map<String, String> md =
os.objectStorage.objects().getMetadata("our_container", "our_object");
System.out.println(md.toString());
}
which I call in the overriden getAttributeValues of the custom PIP class.
I then built the class using mvn package to generate the .jar which I copied in <IS_HOME>/repository/components/lib.
I downloaded all my dependencies using mvn dependency:copy-dependencies, and copied them all to the same <IS_HOME>/repository/components/lib folder.
I start the wso2is server, send a XACML request which involves calling my custom PIP, and see the following error in <IS_HOME>/repository/logs/wso2carbon.log:
ERROR {org.openstack4j.core.transport.internal.HttpExecutor} - No OpenStack4j connector found in classpath
ERROR {org.wso2.carbon.identity.entitlement.pip.CarbonAttributeFinder} - Error while retrieving attribute values from PIP attribute finder: org.openstack4j.api.exceptions.ConnectorNotFoundException: No OpenStack4j connector found in classpath
ERROR {org.wso2.balana.finder.AttributeFinder} - Error while trying to resolve values: Error while retrieving attribute values from PIP attribute finder: No OpenStack4j connector found in classpath
Our wso2is server, version 5.7.0 runs on a CentOS 7.6 VM.
My question is then: can we use this kind of connector in a custom PIP with wso2is? If so, how would I go about resolving the classpath issues between my dependencies?
P.S. : I previously added another custom PIP, connecting this time to a MariaDB database which is working well. The .jar I create when using mvn package contains both custom PIPs, and they are both recognized in the section "PDP > Extension" in the wso2is web interface.
Here is the list of dependencies obtained with the mvn command:
activation-1.1.1.jar
btf-1.2.jar
classworlds-1.1-alpha-2.jar
commons-codec-1.9.jar
commons-io-2.3.jar
commons-lang-2.6.jar
commons-logging-1.2.jar
guava-20.0.jar
httpclient-4.5.3.jar
httpcore-4.4.6.jar
jackson-annotations-2.7.0.jar
jackson-core-2.7.3.jar
jackson-core-asl-1.9.7.jar
jackson-coreutils-1.6.jar
jackson-databind-2.7.3.jar
jackson-dataformat-yaml-2.7.3.jar
jackson-jaxrs-base-2.7.3.jar
jackson-jaxrs-json-provider-2.7.3.jar
jackson-mapper-asl-1.9.7.jar
jackson-module-jaxb-annotations-2.7.3.jar
jboss-annotations-api_1.2_spec-1.0.0.Final.jar
jboss-jaxrs-api_2.0_spec-1.0.1.Beta1.jar
jboss-logging-3.3.0.Final.jar
jcip-annotations-1.0.jar
jcl-over-slf4j-1.7.2.jar
jdom2-2.0.6.jar
joss-0.10.2.jar
json-patch-1.9.jar
jsr305-2.0.0.jar
junit-3.8.1.jar
maven-artifact-2.0.jar
maven-compiler-plugin-2.0.jar
maven-plugin-api-2.0.jar
msg-simple-1.1.jar
openstack4j-3.1.0.jar
openstack4j-core-3.1.0.jar
openstack4j-resteasy-3.1.0.jar
org.wso2.carbon.identity.entitlement-4.2.0.jar
plexus-compiler-api-1.5.1.jar
plexus-compiler-javac-1.5.1.jar
plexus-compiler-manager-1.5.1.jar
plexus-container-default-1.0-alpha-8.jar
plexus-utils-1.0.4.jar
resteasy-client-3.1.4.Final.jar
resteasy-jaxrs-3.1.4.Final.jar
resteasy-jaxrs-services-3.1.4.Final.jar
slf4j-api-1.7.2.jar
snakeyaml-1.15.jar
I have trained both my own model and the one from the official tutorial.
I'm up to the step to deploy the model to support prediction. However, it keeps giving me an error saying:
"create version failed. internal error happened"
when I attempt to deploy the models by running:
gcloud ml-engine versions create v1 \
--model $MODEL_NAME \
--origin $MODEL_BINARIES \
--python-version 3.5 \
--runtime-version 1.13
*the model binary should be correct, as I pointed it to the folder containing model.pb and variables folder, e.g. MODEL_BINARIES=gs://$BUCKET_NAME/results/20190404_020134/saved_model/1554343466.
I have also tried to change the region setting for the model as well, but this doesn't help.
Turns out your GCS bucket and the trained model needs to be in the same region. This was not explained well in the Cloud ML tutorial, where it only says:
Note: Use the same region where you plan on running Cloud ML Engine jobs. The example uses us-central1 because that is the region used in the getting-started instructions.
Also note that a lot of regions cannot be used for both the bucket and model training (e.g. asia-east1).
Is there a way to check the runtime version of a GCMLE prediction service model? From the UI, I can see the model and model location, but I can't remember if the model was pushed with 1.6, 1.7 or 1.8 and would like to confirm.
You can use the gcloud ml-engine models describe command from the gcloud CLI tool in order to obtain more details about the model and its current default version.
Running this command, will return something like:
$ gcloud ml-engine models describe census
defaultVersion:
createTime: '2018-06-05T11:54:35Z'
deploymentUri: gs://GCS/model/location
framework: TENSORFLOW
isDefault: true
name: projects/PROJECT_ID/models/MODEL_NAME/versions/VERSION_NAME
pythonVersion: '2.7'
runtimeVersion: '1.7' <---- This is what you are interested in
state: READY
name: projects/PROJECT_ID/models/MODEL_NAME
regions:
- us-central1
Otherwise, you can find more details about specific ML Engine model versions with the gcloud ml-engine versions describe command too.