I am trying to execute a SQL statement on BQ Database by initializing the BQ Client. This has been running smoothly for a while but lately seeing an issue.
My code specifically fails when its trying to wait for the results from the DB
query_job = client.query(QUERY) # API request to start the query
db_rslt = query_job.result() # Wait for the query to return results
Here is error message:
File "/opt/conda/default/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line xxx, in result super(QueryJob, self).result(retry=retry, timeout=timeout) File "/opt/conda/default/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line xxx, in result return super(_AsyncJob, self).result(timeout=timeout) File "/opt/conda/default/lib/python3.7/site-packages/google/api_core/future/polling.py", line xxx, in result self._blocking_poll(timeout=timeout, retry=retry, polling=polling) TypeError: _blocking_poll() got an unexpected keyword argument 'retry'
Added the timeout parameter to the result method but did not help
It depends on the version of your installed google-cloud-bigquery library, if you are using a 1.x version, make sure to set google-api-core to a version less than 1.34.0.
What is actually happening here is that when you install google-cloud-bigquery, a bunch of other google related libraries are installed automatically with it, google-api-core is one of these libraries. However, a higher version is being installed thus causing this problem.
In my case, setting my google-api-core==1.33.2 solves the issue. Other solution would be is to upgrade google-cloud-bigquery to a later version .
7 days ago Google released an update to google-api-core==1.34.0 for bug fixes. I guess this introduced some breaking changes if you are not using the later versions of google-cloud-bigquery. You may either:
Pin google-api-core to the previous working version (eg. 1.33.2)
Upgrade google-cloud-bigquery to the latest versions.
I solved it by changing the bigquery version to the latest: 'google-cloud-bigquery==3.4.0'.
This also happened to me.
My python image is 3.8.13:bulleye.
It was using the following stack of google library.
google-auth==1.33.0
google-cloud-bigquery==1.25.0
google-cloud-firestore==2.3.1
Changing them to
db-dtypes==1.0.0
google-auth==2.6.0
google-cloud-bigquery==3.0.1
google-cloud-firestore==2.5.3
solved the problems.
db-dtypes is required for new version of google cloud library.
This did the trick for me, although I was having this issue on Kaggle notebooks:
pip install -U google-cloud-bigquery google-api-core==1.33.2
Also the v2 version of google-api-core seems to work:
google-api-core==2.11.0
https://github.com/googleapis/python-api-core/releaseshttps://github.com/googleapis/python-api-core/releases
Related
I am trying to set up mlops for Vertex AI, following this notebook. It works until, near the end, I try:
vertex_ai.init(
project=PROJECT,
location=REGION)
which gives:
module 'google.cloud.aiplatform.constants' has no attribute 'SUPPORTED_REGIONS
I am using us-central1 which is supported. I wondered if maybe from google.cloud import aiplatform as vertex_ai has been changed but don't know how to find out. Any help is much appreciated.
I followed the same Notebook as you, even though I didn't have any issue. What could be happening to you is that you are using an older version of the library.
You can use the command to upgrade the library that is the following one: pip3 install google-cloud-aiplatform --upgrade.
Sometimes this happens with the basic installation of the library; the problems could be in the dependencies, versions and indirectly permissions.
So, I'm working on a robotframework test project, and the goal is to run several test suites in parallel. For this purpose, pabot was chosen as the solution. I am trying to implement it, but with little success.
My issue is: after installing Pabot (which, I might say, I did by cloning the project and running "setup.py install", instead of using pip, since the corporate proxy I'm behind has proven an obstacle I can't overcome), I created a new directory in the project tree, moved some suites there, and ran:
pabot --processes 2 --outputdir pabot_results Login*.robot
Doing so results in the following error message:
2018-10-10 10:27:30.449000 [PID:9676] [0] EXECUTING Suites.LoginAdmin
2018-10-10 10:27:30.449000 PID:400 EXECUTING Suites.LoginUser
2018-10-10 10:27:30.777000 PID:400 FAILED Suites.LoginUser
2018-10-10 10:27:30.777000 [PID:9676] [0] FAILED Suites.LoginAdmin
WARN: No output files in "pabot_results\pabot_results"
Output:
[ ERROR ] Reading XML source '' failed: invalid mode ('rb') or filename
Try --help for usage information.
Elapsed time: 0 minutes 0.578 seconds
Upon inspecting the stderr file that was generated, I have this message:
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\robotframework-3.1a2.dev1-py2.7.egg\robot\running\runner.py", line 22, in
from .context import EXECUTION_CONTEXTS
ValueError: Attempted relative import in non-package
Apparently, this has to do with something from the runner.py script, which, if I'm not mistaken, came with the installation of robotframework. Since manually modifying that script does not seem to me the optimal solution, my question is, what am I missing here? Did I forget to do anything while setting this up? Or is this an issue of compatibility between versions?
This project is using Maven as the tool to manage dependencies. The version I am running is 3.5.4. I am using a Windows 10, 64bit system; I have Python 2.7.14, and Robot Framework 3.1a2.dev1. The Pabot version is 0.44. Obviously, I added C:\Python27 and C:\Python27\Scripts to the PATH environment variable.
Edit: I am also using robotframework-maven-plugin version 1.4.0.8, if that happens to be relevant.
Edit 2: added the error messages in text format.
I believe I've come across an issue similar when setting up parallel execution on my machine. Firstly I would confirm that pabot is installed using pip show robotframework-pabot.
Then you should define the directory your results are going to using -d.
I then modified the name of the -o to Output.xml to make it easy to identify.
This is a copy of the code I use. Runs optimally with 8 processes
pabot --processes 8 -d results -o Output.xml Tests
Seems that you stumbled on a bug in the prerelease version of robot framework (3.1a2.dev1).
Please install a release version of robot framework. For example 3.0.4.
Just in case anyone happens to stumble upon this issue in the future:
Since I can't use pip, and I tried a good deal of workarounds that eventually made things more unstable, I ended up saving my project and removing everything Python-related from my system, so as to allow me to install everything from scratch. In a Windows 10, 64bit system, I used:
Python 2.7.14
wxPython 2.8.12.1, win64, unicode, for py27
setuptools 40.2.0 (to allow me to use the easy_install command)
Robot Framework 3.0.4
robotremoteserver 1.1
Selenium2Library 3.0.0
and Pabot version 0.45.
I might add that, when installing the Selenium2Library the way I described above, it eventually tries to download some things from the pip repositories - which, if you have a proxy, will cause you trouble. I solved this problem by browsing https://pypi.org/simple/selenium/, manually downloading the 2.53.6 .tar.gz file, then extracting it and running setup.py install on the command line.
PS: Ideally, though, anyone should be able to use proxy settings from the command line (--proxy http://user:password#server:port) to get pip and then use it; however, for some reason, probably related to network security configurations that I didn't want to lose time with, this didn't work in my case.
In Anaconda I am trying to create an environment using an environment.yml file which begins with the lines:
name: mytest
dependencies:
- anaconda=4.0.0=np110py27_0
However when trying to create the environment, I get the error:
Fetching package metadata .........
Solving package specifications: ....
Error: The following specifications were found to be in conflict:
- anaconda 4.0.0 np110py27_0
Use "conda info <package>" to see the dependencies for each package.
I encountered no problems when I did this seven days ago, but when I tried this yesterday I got the error.
I am running on Windows 7 64-bit as administrator, Anaconda 2.2.0 (Python 2.7 version). The "conda list" output includes conda 4.1.11 and conda-env 2.5.2.
To try to isolate the error, I installed Miniconda2 on a different 64-bit Windows 7 computer (as administrator) that had never had any Anaconda/Miniconda installed before. This is the most recent 64-bit Python 2.7 series (Miniconda2-4.1.11-Windows-x86_64.exe).
But trying to install anaconda=4.0.0=np110py27_0, either to a new environment or to the root environment, both produce the same error I received before:
C:\>conda install anaconda=4.0.0=np110py27_0
Fetching package metadata .........
.Solving package specifications: ....
The following specifications were found to be in conflict:
- anaconda 4.0.0 np110py27_0
Use "conda info <package>" to see the dependencies for each package.
C:\>conda create --name test400 anaconda=4.0.0=np110py27_0
Fetching package metadata .........
.Solving package specifications: ....
The following specifications were found to be in conflict:
- anaconda 4.0.0 np110py27_0
Use "conda info <package>" to see the dependencies for each package.
How can I determine what is causing the conflict, and how could I resolve it, given that conda is not naming a second package in its error message? I have seen responses to other "specifications in conflict" questions in which the answer is often "Install the problematic package to a separate python environment", but in this case the new environment could not be created with the package. Starting from a clean Miniconda install did not work either. I suspect something has changed in the Anaconda repository (which would be consistent with the original environment.yml working in the past but not now), but how would I determine if this is the underlying issue?
Thanks.
The underlying issue was a temporary error in the https://repo.continuum.io/pkgs/free/win-64/repodata.json file, which has since been fixed.
For reference for anyone investigating Anaconda dependency conflicts, here are the details of the investigation, and the workaround for this case:
The cause:
The repodata.json file (linked above) is essentially a 'master list' of the dependencies of the various libraries in https://repo.continuum.io/pkgs/free/win-64/. The "conda" command uses this repodata.json file.
While the problem was occurring, the repodata.json file incorrectly listed "_nb_ext_conf" as a dependency for each version of ipywidgets. (The /info/index.json file inside "ipywidgets-4.1.1-py27_0.tar.bz2" did not list "_nb_ext_conf" as a dependency, however I think newer versions of ipywidgets require it.)
The "_nb_ext_conf-0.2.0-py27_0.tar.bz2" and "_nb_ext_conf-0.3.0-py27_0.tar.bz2" files list "notebook >=4.2.0" as a dependency in their info/index.json files.
The info/index.json file in anaconda-4.0.0-np110py27_0.tar.bz2 file (which is used when you specify "anaconda=4.0.0=np110py27_0" in an environment.yml) lists "ipywidgets 4.1.1 py27_0" as a dependency.
Due to the temporary problem in repodata.json, this "ipywidgets 4.1.1 py27_0" caused conda to think "_nb_ext_conf" needed to be installed, thus causing conda to think "notebook >=4.2.0" also needed to be installed.
But the info/index.json file in anaconda-4.0.0-np110py27_0.tar.bz2 file also specifies that the specific version "notebook 4.1.0 py27_2" must be installed.
The conflicting requirements for "notebook" versions (4.1.0 and >=4.2.0) caused the "specifications were found to be in conflict" error.
The workaround:
First, remove the line "- anaconda=4.0.0=np110py27_0" from the environment.yml file.
Next, replace that line in environment.yml with every library listed in the "depends" section of the info/index.json file from anaconda-4.0.0-np110py27_0.tar.bz2. (Remove the quotation marks, replace the spaces with equals signs, etc. to convert the .json syntax to the environment.yml syntax.)
Finally, remove the "- notebook=4.1.0=py27_2" line from this list.
This new environment.yml file will now list every library which would have been installed by "anaconda=4.0.0=np110py27_0", with the exception of "notebook", but "notebook" will get installed anyway due to the "notebook >=4.2.0" requirement in "_nb_ext_conf" due to "ipywidgets", and/or the "notebook" requirement in "ipywidgets" itself.
Investigative tools:
The command "conda info anaconda=4.0.0=np110py27_0" gives the list of libraries required by the specified package, according to repodata.json. I put this list of libraries into a temporary_environment.yml file. Attempting to create an environment from that temporary_environment.yml file caused conda to specify that "notebook" was involved in the conflict, which gave the hint to try omitting "notebook".
Running "conda info" lists all the libraries currently installed in the active environment. The output for the environment created by temporary_environment.yml was compared to the output from an environment from a computer where "anaconda=4.0.0=np110py27_0" had previously installed successfully. This highlighted "_nb_ext_conf" as one difference.
I created a batch file which ran "conda info" for every library listed in anaconda=4.0.0=np110py27_0, and I looked for instances of "notebook" and "_nb_ext_conf" in the output. This pointed to "ipywidgets" as a suspect.
I have been trying to connect with a MySql database using Wt::Dbo::backend::MySQL latest version Wt-3.3.5 but I keep getting the error:
"MySQL error performing query: 'SET storage_engine=INNODB;': Unknown system variable 'storage_engine'"
MySql is running on windows 7, with latest version MySql Server 5.7.10.
I have tried to execute:
SET storage_engine=INNODB;
in the MySql command line and of course get the same Unknown system variable 'storage_engine'". The command that does work is:
SET default_storage_engine=INNODB;
Does this means Wt::Dbo::backend::MySQL does not support this version of MySql? I could not find anything on their documentation nor the internet.
Thanks,
I have found in the following links that indeed the variable "storage_engine" has been renamed to "default_storage_engine", which has caused some software to start getting issues with recent versions of MySql.
https://dba.stackexchange.com/questions/101908/migrate-jira-to-mysql-unknown-system-variable-storage-engine
https://confluence.atlassian.com/display/CONFKB/Confluence+fails+to+start+with+error+'Unknown+system+variable+'storage_engine''+using+MySQL+5.7.x
There are to possible solutions:
1) Change Wt::Dbo source by modifying the file "MySQL.C" around line 980 in function :
void MySQL::init()
{
executeSql("SET sql_mode='ANSI_QUOTES,REAL_AS_FLOAT'");
executeSql("SET storage_engine=INNODB;");
executeSql("SET NAMES 'utf8';");
}
By first somehow testing first which MySql version the server has and then SET the "default_storage_engine" variable instead "storage_engine". And then recompile the full Wt::Dbo library.
OR
2) Go back to the most recent version of MySql in which the variable "storage_engine" has not yet been renamed.
I opted for solution 2 and uninstalled MySql 5.7 and installed MySql 5.4 and now everything works just fine.
Hope this helps somebody out there... cheers!
I am trying to capture outgoing packets (i.e., from my localhost) by using Scapy capturing function sniff, then parse them as they get captured but I keep getting the following warning:
WARNING: fileno: pcapy API does not permit to get capure file descriptor. Bugs ahead! Press
Enter to trigger packet reading
Is there a fix for this issue? I need the process to be fully automatic without the user interaction.
Any help is greatly appreciated.
p.s. my system setup is:
OS: Mac OS X 10.7.5
python: 2.7.1
scapy: 2.2.0
Thanks
Just in case somebody else ran into the same issue, I managed to solve it by installing pypcap module to replace pcapy. I referred to http://comments.gmane.org/gmane.comp.security.scapy.general/4026
you will need to have the following dependencies installed:
C compiler + make (Xcode will have these settled)
libpcap
pyrex ( can be installed by package manager e.g., pip ).
you might run into an issue when you install pypcap after the dependencies are installed, refer to this patch http://code.google.com/p/pypcap/issues/detail?id=13, it worked for me!