I've been using Dataprep for months, and have a lot of different flows built in one of my projects. I was working with it this morning, but now when I log in, the project in Dataprep is blank, like I'm a brand new user. I'm starting to panic because months of work has vanished! Does anyone have any suggestions on what to do?
Things I've tried without success:
I switched into a different project and I can see that project's
flows listed.
Logged out/in
restarted browser
Thank you for your help, you are correct. It turns out we received an email from google with the subject "[Action Required] Please migrate off JSON-RPC and Global HTTP Batch Endpoints" (specifically storage#v1). We were not using this API with the solutions we developed within this project, so one of our developers deactivated it. It showed the affected dependencies, which included the Dataflow API. DataPrep was not disabled, nor did it need to be reenabled before accessing it again...it just lost it's metadata like both Ali T and James commented.
Google Cloud Support recommends exporting the recipes and flows (manually I believe) as the best way to prevent DataPrep working file loss in the future.
Related
I have been using the Google Cloud speech recognition service for some time, through a python application.
Due to accidentally copying my Google Cloud json file to a GitHub shared location (I was doing a backup), I suddenly got a warning from Google Cloud that I was violating the rules as json is private. Then, I promptly removed the file, but nevertheless, I got an email saying that my resources for my project "santo1" were suspended, saying some reasons of "cryptocurrency mining" which I have no idea.
I applied to reactivate and my appeal was accepted promptly, saying that my resources about santo1 were reinstated.
Unfortunately, the speech recognition still didn't work.
Launching it from python, it records from the microphone but no answer from the service - and no error messages at all.
Then I attempted the following:
regenerate API
create a new json
create a new project with its own json under my same google account
as suggested by the Google Cloud chat operator, I manually clicked play to the VM resource that appeared stopped
create a new gmail account, with another new project, setup with billing and everything (also reconfigured through "gcloud init")
None of these attempts worked.
I need assistance on this, as the chat operator didn't seem capable of telling me more.
Thank you in advance
Best regards
I would recommend you to contact GCP support for this case as your cloud service could be still in suspended status regardless your access is OK
Apparently, the access key is stolen and used by hackers and they did crypto mining using your GCP account, hence your service account was banned
If it's your testing account/project, you should consider to create a new project rather than continue with it, the hacker could create some other services which you may not realize until too late
Worse case is it's your PROD service, then you'd better review the bill and transaction report thoroughly
I have several questions regarding the Google Spanner Export / Import tool. Apparently the tool creates a dataflow job.
Can an import/export dataflow job be re-run after it had run successfully from the tool? If so, will it use the current timestamp?
How to schedule a daily backup (export) of Spanner DBs?
How to get notified of new enhancements within the GCP platform? I was browsing the web for something else and I noticed that the export / import tool for GCP Spanner had been released 4 days earlier.
I am still browsing through the documentation for dataflow jobs and templates, etc.. Any suggestions to the above would be greatly appreciated.
Thx
My response based on limited experience with the Spanner Export tool.
I have not seen a way to do this. There is no option in the GCP console, though that does not mean it cannot be done.
There is no built-in scheduling capability. Perhaps this can be done via Google's managed Airflow service, Cloud Composer (https://console.cloud.google.com/composer)? I have yet to try this, but it is next step as I have similar needs.
I've made this request to Google several times. I have yet to get a response. My best recommendation is to read the change logs when updating the gcloud CLI.
Finally-- there is an outstanding issue with the Export tool that causes it to fail if you export a table with 0 rows. I have filed a case with Google (Case #16454353) and they confirmed this issue. Specifically:
After running into a similar error message during my reproduction of
the issue, I drilled down into the error message and discovered that
there is something odd with the file path for the Cloud Storage folder
[1]. There seems to be an issue with the Java File class viewing
‘gs://’ as having a redundant ‘/’ and that causes the ‘No such file or
directory’ error message.
Fortunately for us, there is an ongoing internal investigation on this
issue, and it seems like there is a fix being worked on. I have
indicated your interest in a fix as well, however, I do not have any
ETAs or guarantees of when a working fix will be rolled out.
When I open the Google Cloud Shell Code Editor it is not loading the resources and hence I am unable to work. I have attached a screenshot below with a view of the developer tools console. Please help me out. Thanks.
This issue seems to be related to an internal project/billing configuration. Since this kind of access errors are thrown when the accounts have payment issues, I think that you should firstly verify that your billing account is in a good status; however, if you continue getting these error messages after this validation, I suggest you to take a look the Issue Tracker tool that you can use to raise a Cloud Shell ticket in order to verify this scenario with the Google Technical Support Team.
A couple things could cause this:
An interfering browser extension
Are you using any browser extensions that could be interfering
(e.g., an ad blocker)
A bug.
As #Armin_SC suggested, use Issue Tracker to file an issue in this case.
As a workaround, you might want to try gcloud compute ssh to connect to your instances.
I've deployed a copy of opserver, and it is working perfectly when using alladmin as the security setting. However, once I switch it to ad and configure the groups, the SQL tab goes away and I get an access denied message if I try browsing directly to it. The dashboard still displays all Solar Winds data as expected.
The build I'm using is actually from November. I tried a more recent build, but I lose the network information from Solar Winds (the CPU and Mem graphs show, but Net is all blank)
Is there a separate place to configure the SQL permissions that I'm missing?
I think perhaps there was some caching going on for the hub that wasn't happening for the provider, because they are both working now. Since it was a new security group, perhaps it hadn't replicated yet (causing the SQL auth to fail) but the dashboard provider was still using the previous authentication?
I also did discover a neat option while researching this though - the GitHub page mentions that you can also specify security at a provider level in the JSON using the AdminGroups and ViewGroups properties!
Are there any best practices on virus scanning all files being uploaded to the Sitecore media library (and ultimately stored in Sitecore's DB)?
I searched all over the web but there is too much noise caused by the word virus since many people seem to have performance issues on server that have anti-virus software installed.
I don't know if it is an established best practice, but I would probably add a processor for the uiUpload pipeline that used an API or command line process for a commercial antivirus product. Other than the fact that it is in a pipeline processor, it shouldn't really be much different from how you would do it in any other ASP.NET application. Performance will definitely be a concern, but you could create a dialog with a psuedo progress bar to give some feedback to the user.
Take a look at this post by Mike Reynolds. It may help you out:
http://sitecorejunkie.com/2013/11/09/perform-a-virus-scan-on-files-uploaded-into-sitecore/
I am not aware of any published best practices, but if you are able to add a step in the upload process, you might want to take a look at Metascan, which provides API level integration to multiple antivirus engines. Using this, you could build a workflow for those uploaded files to scan them prior to them hitting your Sitecore media library by establishing rules based on the results of the antivirus engines used in your Metascan deployment. There's also a hosted version at metascan-online(dot)com
Disclaimer /// I am an employee of OPSWAT, who produces Metascan, but it appears to be a potential solution to your issue
In one of our recent Projects, we were faced with a requirement to scan incoming files for virus. The problem in the project was that the files after begin uploaded, were made public available on the website.
The way we solved the problem was to implementing https://www.virustotal.com/. Its a free online virus scanner that has a public API. You can send files via SSL.
We implemented the solution by adding newly uploaded files to a Sitecore workflow. The workflow would handle the scanning of files, and move the files to the final stage of the workflow, if the files wasn't infected. If a file was infected, the file would be deleted.
A Scheduler is running every 5 minutes to check for new incoming files with the workflow.
This also means that the files aren't available straight away, as the scheduler has to check the file, but you should be able to implement the functionality directly when the user has uploaded the file, by adding your custom code to the upload pipeline.