I have already host of my wordpress website on Google Cloud Platform using Google Cloud Compute Engine. Now I want to split my existing wordpress database and move to Google SQL Cloud to improve my website performance.
I'm creating successfully SQL instance on Google Cloud SQL cloud. I refer to this link but I got error when I'm uploading my wordpress database backup.
After creating database on Google Cloud SQL when I click on import button, it take few minutes and show import failed : error 1031 (hy000) table storage engine for wp_wcfm_daily_analysis doesn't have this option error.
Thanks in advance.
In one of your import file there is the command that tries to change the storage engine from InnoDB to some other storage engine, probably to MyISAM.
As it is stated in the CloudSQL documentation:
InnoDB is the only supported storage engine for Second Generation instances because it is more resistant to table corruption than other MySQL storage engines, such as MyISAM.
You need to check in your sql file that you want to import, if you have the option : ENGINE = MyISAM attached to any CREATE TABLE command, and remove it.
You can also try to convert all your tables to InnoDB by using the following SQL code:
SET #DATABASE_NAME = 'name_of_your_db';
SELECT CONCAT('ALTER TABLE `', table_name, '` ENGINE=InnoDB;') AS sql_statements
FROM information_schema.tables AS tb
WHERE table_schema = #DATABASE_NAME
AND `ENGINE` = 'MyISAM'
AND `TABLE_TYPE` = 'BASE TABLE'
ORDER BY table_name DESC;
You can find here a related discussion.
Related
I am trying a simple operation where I import a csv file from my cloud storage bucket into my cloud sql database. However I repeatedly receive the error message HTTPError 400: Unknown export file type..
I am following https://cloud.google.com/sdk/gcloud/reference/sql/import/csv as reference.
What can be wrong?
The database Prod exists as does the fundamental table
The actual .csv data I am trying to import
CSV is currently not a supported file type in Cloud SQL, SQL Server. As stated in this documentation,
In Cloud SQL, SQL Server currently supports importing databases using SQL and BAK files.
You can do one of the following:
Switch database engine to either MySQL or PostgreSQL where CSV files are supported.
If the data on your CSV file came from an on-premise SQL Server database table, you can create an SQL file from it, then use it to import into Cloud SQL, SQL Server.
I have created one Composer in gcp project. I want to access the Metadatadb of Airflow which runs at background on Cloud SQL.
How can i access that?
Also i want to create one table inside that metadatadb which i will be using to store some data query by one of airflow dag. Is it ok to create any table inside that metadatadb or that metadatadb is only for airflow server use?
You can access Airflow internal DB via UI using Data Profiling -> Ad Hoc Query
There you can see all the tables with a SQL query like :
SHOW tables;
I wouldn't recommand creating a new table or manually inserting rows into existing tables thought.
You should also be able to access this DB in your DAGs operators and sensors by using airflow-db connexion.
We have google cloud MySQL Instance and we are getting error like:
InnoDB: Cannot add field CE_Lead__c in table client_1067.#sql-ib1179556-1243586840 because after adding it, the row size is 8148 which is greater than maximum allowed size (8126) for a record on index leaf page.
The workaround to fix this issue is to set internal_tmp_disk_storage_engine=MYISAM
I can't see this flag in Database flags section, also as per the google documents this variable is not exist in google cloud, however I can see it exist in MySQL 5.7
Database flag not exist in google cloud supported flag:
https://cloud.google.com/sql/docs/mysql/flags#list-flags-mysql
Database flag exist in MySQL 5.7
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_internal_tmp_disk_storage_engine
Thanks
SKS
Some flags are not supported in Cloud SQL MySQL due to the managed nature of the service (stability/streamline reasons, etc.) You could try other workarounds like decreasing the memory usage mentioned further below on this post, if you decide you need to use flags that are not supported on Cloud SQL you can choose to create your own self managed MySQL instance in order to bypass the Cloud SQL limitations. (for example using Compute Engine)
I have build a Django WebApp. It has an sql database. I would like to analyze this data and share the analysis using online platform Jupyter notebook such as Kaggle.
I have already deployed to Google App Engine as an SQL instance, but I don't know how to view this SQL instance tables in Kaggle. There is an option to view BigQuery databases in Kaggle, but I don't know how to get the data from my SQL instance to BigQuery.
To be able to access the data with Kaddle you would need to import the data from the CloudSQL instance into BigQuery.
Currently there are some options for importing data into BigQuery, the best choice would depend on what type of analysis you want to do with it.
If you just want to import the data from the CloudSQL instance into BigQuery, the easiest way to do it would be to first export the data in CSV format and then import the CSV file into BigQuery.
In case you are working with a large database, you can also do it programmatically by using the Client Libraries.
I have several Django (python) based, back-end web applications that I would like to start piping data into Google Big Query in an automated fashion. The relational database on the backend is MySQL, these applications are not public facing and not in Google App Engine.
We already have Google Apps for Business along with a Google Big Data project set up. With that said I can manually dump tables into CSV and import into Big Query but is there some best practices on automating this kind of data delivery into Google? I've poured over the documentation and don't really see any definitive writing on this matter.
Any advice would be appreciated.
Thanks for reading
Recently WePay started a series of articles on how they use BigQuery to run their analytics. Their second article highlights how they use Apache AirFlow to move data from MySQL to BigQuery:
https://wecode.wepay.com/posts/airflow-wepay
As they mention "We have only a single config-driven ETL DAG file. It dynamically generates over 200 DAGs", and "The most important part is the select block. This defines which columns we pull from MySQL and load into BigQuery".
See the article for more details.
You can use Python robots, which run on Linux with crontab.
For loading into Google Cloud Platform BigQuery, I use pandas_gbq.to_gbq library:
Create your dataframe (df) according to this or this
In order to get the Token.jsonfile:
Create a Google Cloud Platform BigQuery service account.
Load the JSON file:
from google.oauth2 import service_account
import pandas as pd
import pandas_gbq
DIR = os.path.dirname(os.path.realpath(__file__))
TOKEN_AUTH = DIR + '/token.json'
CREDENTIALS = service_account.Credentials.from_service_account_file(TOKEN_AUTH)
#df is a pandas dataframe
pandas_gbq.to_gbq(df, '<dataset>.<table_name>', project_id='<project_id>',
if_exists=<replace or append> , credentials=CREDENTIALS)
Once you have created your tocken, install crontab on Linux and schedule your load-robot task:
Using crontab to execute script every minute and another every 24 hours
Finally, you can also use Apache Airflow (for advanced users with Docker skills)