multiline regex match string replacement on large file - regex

I have a large mysqldump (4+ gigs), and we have an archive type table which suffers from this bug Net result is that I need to reset the AUTO_INCREMENT counter to zero. Did manage to do the replacement, but it was ugly, involving splitting the file into smaller chunks, then grepping to find the table, looking to see the number I wanted to change and then using sed on the original file to replace just that match on the auto increment. Like I said, horrible, but it worked.
So - I have tried to decipher multiline sed and didn't get that far. What I want to do is to seek to the table name I'm interested in, and then from that point find the next AUTO_INCREMENT= , and then match the number in it, and make it zero. Here's the table: (assume there is scads of data before this point, and after it)
DROP TABLE IF EXISTS `archive_exported_problems`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `archive_exported_problems` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`export_id` int(11) DEFAULT NULL,
`problem_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=ARCHIVE AUTO_INCREMENT=478 DEFAULT CHARSET=latin1;
What I want to do, is to (automatically) scan the file until it matches
(?:CREATE TABLE `archive_exported_problems).*?AUTO_INCREMENT=(\d+)
(regex which seems to work) and then replace the capture group with 0
I assume this is possible - any help most appreciated!

If perl is an option then it is easier using DOTALL flag in perl like this:
perl -00 -pe
's/(?s)(CREATE TABLE `archive_exported_problems`.*?AUTO_INCREMENT)=\d+/$1=0/' file.sql
DROP TABLE IF EXISTS `archive_exported_problems`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `archive_exported_problems` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`export_id` int(11) DEFAULT NULL,
`problem_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=ARCHIVE AUTO_INCREMENT=0 DEFAULT CHARSET=latin1;
Options used are:
-00 # slurps whole file
(?s) # enable DOTALL flag for regex

Consider this:
$ sed -r '/CREATE TABLE `archive_exported_problems`/,/AUTO_INCREMENT=/ {s/(AUTO_INCREMENT)=[[:digit:]]+/\1=0/;}' file
DROP TABLE IF EXISTS `archive_exported_problems`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `archive_exported_problems` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`export_id` int(11) DEFAULT NULL,
`problem_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=ARCHIVE AUTO_INCREMENT=0 DEFAULT CHARSET=latin1;
For Mac OSX (BSD), try:
$ sed -E -e '/CREATE TABLE `archive_exported_problems`/,/AUTO_INCREMENT=/ {s/(AUTO_INCREMENT)=[[:digit:]]+/\1=0/;}' file
How it works
/CREATE TABLEarchive_exported_problems/, /AUTO_INCREMENT=/
This restricts the subsequent commands to ranges on lines that start with a line containing CREATE TABLE 'archive_exported_problems' and end with a line containing AUTO_INCREMENT=.
s/(AUTO_INCREMENT)=[[:digit:]]+/\1=0/
This performs the substitution that you wanted.
Limitation
This approach assumes that the CREATE TABLE phrase and the AUTO_INCREMENT= phrase will never be on the same line. If that is not true, we need to make some minor changes.

Related

Django and mariadb not working with istartswith correctly using german umlauts

the following query should only return all cities starting with "Ö" (German umlaut).
letter = 'Ö'
City.objects.filter(name__istartswith=letter)
But it returns cities starting with O and Ö.
I use django 1.11 and mariadb.
I allready set COLLATE on that table to utf8_bin but this haven't changed the behavior within django.
This is the simplified SQL query
SELECT `cities_city`.`name` FROM `cities_city` WHERE `cities_city`.`name` LIKE "Ö%";
and here the SHOW CREATE TABLE output:
SHOW CREATE TABLE `cities_city`
-> ;
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| cities_city | CREATE TABLE `cities_city` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(200) CHARACTER SET utf8 NOT NULL,
`slug` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`name_std` varchar(200) CHARACTER SET utf8 NOT NULL,
`location` point NOT NULL,
`population` int(11) NOT NULL,
`elevation` int(11) DEFAULT NULL,
`kind` varchar(10) CHARACTER SET utf8 NOT NULL,
`timezone` varchar(40) CHARACTER SET utf8 NOT NULL,
`country_id` int(11) NOT NULL,
`region_id` int(11) DEFAULT NULL,
`subregion_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `cities_city_country_id_2f07e352_uniq` (`country_id`,`region_id`,`subregion_id`,`id`,`name`),
KEY `cities_city_b068931c` (`name`),
KEY `cities_city_16c3f481` (`name_std`),
KEY `cities_city_region_id_0227cdac_fk_cities_region_id` (`region_id`),
KEY `cities_city_subregion_id_9fbab97d_fk_cities_subregion_id` (`subregion_id`),
CONSTRAINT `cities_city_country_id_779ae117_fk_cities_country_id` FOREIGN KEY (`country_id`) REFERENCES `cities_country` (`id`),
CONSTRAINT `cities_city_region_id_0227cdac_fk_cities_region_id` FOREIGN KEY (`region_id`) REFERENCES `cities_region` (`id`),
CONSTRAINT `cities_city_subregion_id_9fbab97d_fk_cities_subregion_id` FOREIGN KEY (`subregion_id`) REFERENCES `cities_subregion` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=11468436 DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
The problem is a subtle one.
The clue is here:
SELECT * FROM information_schema.`COLUMNS` WHERE table_name = 'cities_city';
The explanation...
`name` varchar(200) CHARACTER SET utf8 NOT NULL,
is COLLATE utf8_general_ci because that is the default collation for utf8.
This table default:
) ENGINE=InnoDB AUTO_INCREMENT=11468436 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
gives utf8_bin to any newly added rows.
Perhaps you did the obvious ALTER TABLE to change to _bin? Instead:
ALTER TABLE cities_city
CONVERT TO CHARACTER SET utf8
COLLATE utf8_bin;
this will go into each string column and make the change. Note that indexes (etc) must be rebuilt when the collation changes.

REGEX: get Create table queries in sql dump

I have an sql dump of different tables each with different amount of fields, I want to insert a query after each one, so I'm trying to find a regex statment that would retreive:
CREATE TABLE cms_audit (
aud_id bigint NOT NULL IDENTITY(1,1),
user_id int DEFAULT NULL,
client_id int NOT NULL,
aud_event varchar(500) NOT NULL,
aud_type varchar(150) NOT NULL,
aud_string varchar(1000) DEFAULT NULL,
aud_date datetime DEFAULT NULL
)
-- --------------------------------------------------------
My regex is CREATE TABLE .*-- (in notepad++ I've checked the box that say's . matches newline) in my head this means get all that starts with "create table" and whatever is after it until you reach "--".
However this statement is retrieving the entire file instead of getting each "create table" query separately, what am I missing?.
I have also tried CREATE TABLE (.*|\n)*--.. didn't work.
You need to use a regex with any character except --. To achieve this you can do:
CREATE TABLE (?:(?!--).)*
EDIT
The ?! is to make a Negative Lookahead of the string --. Nothing with this string will match this expression.
You can see and test it with this link (it's very well explained and a good tool):
https://regex101.com/r/mR9fD4/1

mysql c++ connector reduces my strings when I insert into the database

I have a problem with mysql c++ connector when i want to insert a string with a prepared statement he reduce my string in the database(saved in a longtext) . I have enormous loss of data because I want to save a longtext.
here is my code :
void RequetteBDD::add(Files::Fichier file)
{
string query = "INSERT INTO files(titre,url,type,txt,lastcrawl) VALUES (?,?,?,?,?)";
sql::PreparedStatement *prep_stmt;
prep_stmt = con->prepareStatement(query);
prep_stmt->setString(1,file.getNom()); //title
prep_stmt->setString(2,file.getURL().getUri()); //url
prep_stmt->setInt(3,file.getTypeInt()); //type
//i also try :
istringstream stream(file.getTextFull());
prep_stmt->setBlob(4,&stream);
//but the saved length was exactly the same.
prep_stmt->setString(4,file.getTextFull()); //here is the probleme
prep_stmt->setInt(5,time(NULL)); //timstamp
prep_stmt->execute();
delete prep_stmt;
}
mysql ddb:
CREATE TABLE IF NOT EXISTS `files` (
`id` int(11) NOT NULL AUTO_INCREMENT ,
`titre` varchar(256) COLLATE utf8_unicode_ci NOT NULL,
`url` varchar(512) COLLATE utf8_unicode_ci NOT NULL,
`type` int(1) NOT NULL DEFAULT '0' COMMENT ,
`txt` longtext COLLATE utf8_unicode_ci NOT NULL,
`lastcrawl` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=7 ;
thanks for your help.
It was a encoding problem; they are two solution to solve the probleme:
change the encoding from the database to ASCII
Or change the encoding from the string can be easely do with boost::locale::conv
I hope it will help other people.

Error Message after attempted .sql import

I use Joomla and I recently installed a new Template.
To make the demo-content of said Template work, I need to import a .sql file into my database using phpMyAdmin.
However, after trying to import the file I get this error message:
SQL-Befehl [SQL-Order]:
--
-- Datenbank *[database]*: `j17_jp_investment`
--
-- --------------------------------------------------------
--
-- Tabellenstruktur *[table structure]* für Tabelle *[table, sheet]* `jos_assets`
--
DROP TABLE IF EXISTS `jos_assets` ;
MySQL meldet [reports]: Dokumentation [documentation]
#1046 - No database selected
i had the same issue with a template i just bought!
Add the MySQL sentense: USE jml_d52ka2dc1f; where jml_d52ka2dc1f; is your database name.
Your dump.sql file should looks like this:
USE jml_d52ka2dc1f;
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
DROP TABLE IF EXISTS `jml_assets`;
CREATE TABLE IF NOT EXISTS `jml_assets` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`parent_id` int(11) NOT NULL DEFAULT '0' COMMENT 'Nested set parent.',
`lft` int(11) NOT NULL DEFAULT '0' COMMENT 'Nested set lft.',
`rgt` int(11) NOT NULL DEFAULT '0' COMMENT 'Nested set rgt.',
`level` int(10) unsigned NOT NULL COMMENT 'The cached level in the nested tree.',
`name` varchar(50) NOT NULL COMMENT 'The unique name for the asset.\n',
`title` varchar(100) NOT NULL COMMENT 'The descriptive title for the asset.',
`rules` varchar(5120) NOT NULL COMMENT 'JSON encoded access control.',
PRIMARY KEY (`id`),
UNIQUE KEY `idx_asset_name` (`name`),
KEY `idx_lft_rgt` (`lft`,`rgt`),
KEY `idx_parent_id` (`parent_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=43 ;
:) Enjoy!
Reference: http://dev.mysql.com/doc/refman/5.0/en/use.html
Do you have a database called "j17_jp_investment"?
Otherwise remove that line from the .sql file, go to phpmyadmin, choose the database that joomla is using then go to the import tab and try importing.
The same error was showing up for me. I was trying to import my client's mySQL database to my WAMP server.
I just added these two words at the beginning of my SQL queries:
USE [database_name]
However, before doing so, I made sure that I created a database with the exact same name that I'm using in the code above.

Can't create table (errno: 150) InnoDB adding foreign key constraints

Really hate to use other people's time, but it seems the problem is just not going away.
I considered all recommendations at http://verysimple.com/2006/10/22/mysql-error-number-1005-cant-create-table-mydbsql-328_45frm-errno-150/ and at http://forums.mysql.com/read.php?22,19755,19755#msg-19755 but nothing.
hope that someone points to a stupid mistake.
here are the tables:
CREATE TABLE IF NOT EXISTS `shop`.`category` (
`id` INT(11) NOT NULL AUTO_INCREMENT ,
`category_id` INT(11) NOT NULL ,
`parent_id` INT(11) NULL DEFAULT '0' ,
`lang_id` INT(11) NOT NULL ,
...other columns...
PRIMARY KEY (`id`, `category_id`) )
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_unicode_ci;
CREATE TABLE IF NOT EXISTS `shop`.`product_category` (
`category_id` INT(11) NOT NULL ,
`product_id` INT(11) NOT NULL ,
INDEX `fk_product_category_category1_zxc` (`category_id` ASC) ,
CONSTRAINT `fk_product_category_category1_zxc`
FOREIGN KEY (`category_id` )
REFERENCES `shop`.`category` (`category_id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_unicode_ci;
Error Code: 1005. Can't create table 'shop.product_category' (errno: 150)
You need an index on category_id in the category table (I see it's part of the primary key, but since it's the second column in the index, it can not be used). The field you are referencing in a foreign key always should be indexed.
In my case the issue was more like what was described in the first article you've linked to.
So I just had to make sure that:
Referenced Column is an index,
both Referencing Column and Referenced Column share the same type and length, i.e. e.g. both are INT(10),
both share the same not null, unsigned, zerofill etc. configuration.
both tables are InnoDB!
Here's the query template where Referencing Column is referencing_id and Referenced Column is referenced_id:
ALTER TABLE `db`.`referencing`
ADD CONSTRAINT `my_fk_idx`
FOREIGN KEY (`referencing_id`)
REFERENCES `db`.`referenced`(`referenced_id`)
ON DELETE NO ACTION
ON UPDATE NO ACTION;
Update 2016-03-13: Ran into this problem again, ended up finding my own answer. This time it didn't help though. Turns out the other table was still set to MyISAM, as soon as I changed it to InnoDB everything worked.