I'm looking at different ways of settings up Doctrine 2 to use master/slave connections usng MySQL. The set up would be so that there is one master database with multiple slaves. All SELECT statements should come from a random live slave and any UPDATE, INSERT, DELETE statements would always be delegated to the master connection.
Has anyone set up Doctine 2 in this way or have any ideas on how to approach it?
Doctrine2 now has a MasterSlaveConnection in the \Doctrine\DBAL\Connections namespace.
EDIT:
Dont read below unless this piece doesnt work
Dont need an overloader anymore, the dbal configs will take the slaves by itself. e.g.
connections:
default:
driver: %database_driver%
host: %database_host%
dbname: %database_name%
user: %database_user%
password: %database_password%
slaves:
slave1:
host: %database_slave1%
dbname: %database_name%
user: %database_user%
password: %database_password%
If the above doesnt work, try this
Something simple, just put pipes (|) between each host
default:
driver: %database_driver%
host: %database_host%|%database_slave%|%database_slave2%
port: 3306
dbname: %database_name%
user: %database_user%
password: %database_password%
wrapper_class: \Foo\Bar\Symfony\Doctrine\Connections\MasterSlave
<?php
namespace Foo\Bar\Symfony\Doctrine\Connections;
use \Doctrine\DBAL\Connections\MasterSlaveConnection;
use Doctrine\DBAL\Connection,
Doctrine\DBAL\Driver,
Doctrine\DBAL\Configuration,
Doctrine\Common\EventManager,
Doctrine\DBAL\Event\ConnectionEventArgs,
Doctrine\DBAL\Events,
Doctrine\DBAL\Cache\QueryCacheProfile;
class MasterSlave extends MasterSlaveConnection
{
public function __construct(array $params, Driver $driver, Configuration $config = null, EventManager $eventManager = null)
{
$tempParams = array(
'master' => array()
, 'slaves' => array()
, 'driver' => $params['driver']
);
$hosts = explode('|', $params['host']);
unset($params['host']);
foreach($hosts as $num => $host)
{
$params['host'] = $host;
if($num == 0)
{
$tempParams['master'] = $params;
}
else
{
$tempParams['slaves'][] = $params;
}
}
if(!isset($tempParams['master']['driver']))
$tempParams['master']['driver'] = "pdo_mysql";
foreach($tempParams['slaves'] as $k => $slave)
{
if(!isset($slave['driver']))
$tempParams['slaves'][$k]['driver'] = "pdo_mysql";
}
parent::__construct($tempParams, $driver, $config, $eventManager);
}
public function executeQuery($query, array $params = array(), $types = array(), QueryCacheProfile $qcp = null)
{
try
{
return parent::executeQuery($query, $params, $types, $qcp);
}
catch(\Exception $e)
{
$logger = new \Uelib\Core\Logger();
$message = $e->getMessage() . "\nSql: " . $query . "\nParams: \n" . print_r($params, true);
$logger->log($message);
throw $e;
}
}
}
As far as I know there is not built-in support for this in Doctrine 2.
It really seems as if (at least with mysql), this problem is going to be ultimately solved elsewhere. Either in mysql_proxy, or via some very recent work with mysqlnd (the mysql native driver extension for php)
Since neither of those are ready for primetime (though mysql_proxy might be), your only immediate option is, unfortunately, to start looking at the Doctrine DBAL, and extending the built-in classes to handle things smartly -- meaning it might be a good idea to detect if the current request has done any write operations, and then force any subsequent reads to use the master, avoiding any replication-delay-related issues.
Hopefully, we'll see a more formal solution from the Doctrine team in the next 6-12 months. But if you need it now, it's DIY, AFAIK.
Related
i am able to update the value to the sections "User foo" and "Host *.example.net" by passing the index. If i pass index 1 or 2 the respective value is getting updated.
my code:
$sections = ['Host *.example.net', 'User foo']
$sections.each |String $section| {
sshd_config_match { "${section}":
ensure => present,
}
}
$settings = [['User foo', 'X11Forwarding yes', 'banner none'],['Host *.example.net', 'X11Forwarding no', 'banner none']]
$settings.each |Array $setting| {
$setting_array = split($setting[1],/ /)
sshd_config { "${setting_array[0]} ${setting[0]}":
ensure => present,
key => "${setting_array[0]}",
condition => "${setting[0]}",
value => "${setting_array[1]}",
}
}
current result:
Match Host *.example.net
# Created by Puppet
X11Forwarding no
Match User foo
# Created by Puppet
X11Forwarding yes
Expected results:
Match Host *.example.net
# Created by Puppet
X11Forwarding no
Banner none
Match User foo
# Created by Puppet
X11Forwarding yes
Banner none
i am able to update only one value mentioned in the index but am looking a way to update more or all the values mentioned in the list.
It's not clear what module is providing your sshd_config_match and sshd_config resource types, nor, therefore, exactly what they do. Nevertheless, if we consider this code ...
$settings = [['User foo', 'X11Forwarding yes', 'banner none'],['Host *.example.net', 'X11Forwarding no', 'banner none']]
$settings.each |Array $setting| {
$setting_array = split($setting[1],/ /)
sshd_config { "${setting_array[0]} ${setting[0]}":
ensure => present,
key => "${setting_array[0]}",
condition => "${setting[0]}",
value => "${setting_array[1]}",
}
}
... we can see that each element of $settings is a three-element array, of which the each call accesses only those at indexes 0 and 1. That seems to match up with the result you see, which does not contain anything corresponding to the data from the elements at index 2.
You could iterate over the inner $setting elements, starting at index 1, instead of considering that element only, but I would suggest instead restructuring the data more naturally, and writing code suited to the restructured data. You have data of mixed significance in your arrays, and you are needlessly jamming keys and values together such that you need to spend effort to break them back apart. Structuring the data as a hash of hashes instead of an array of arrays could be a good start:
$settings = {
'User foo' => { 'X11Forwarding' => 'yes', 'banner' => 'none'},
'Host *.example.net' => { 'X11Forwarding' => 'no', 'banner' => 'none'},
}
Not only does that give you much enhanced readability (mostly from formatting), but it also affords much greater usability. To wit, although I'm guessing a bit here, you should be able to do something similar to the following:
$settings.each |String $condition, Hash $properties| {
$properties.each |String $key, String $value| {
sshd_config { "${condition} ${key}":
ensure => 'present',
condition => $condition,
key => $key,
value => $value,
}
}
}
Again, greater readability, this time largely from a helpful choice of names, and along with it greater clarity that something like this is in fact the right structure for the code (supposing that I have correctly inferred enough about the types you are using).
I have a module called appserver in my puppet modules.
In that module manifests I have a params.pp file which is inherited by init.pp file.
In params.pp file I have the following data structure.
$servers = {
appserver-mgr => { axis2 => {subDomain => 'mgt',},
carbon => {subDomain => 'mgt',},
serverOptions => '-Dsetup',
server_home => $carbon_home, },
appserver-wkr => { axis2 => {subDomain => 'worker', members => ['appserver-mgr2-ip']},
carbon => {subDomain => 'worker',},
serverOptions => '-DworkerNode=true',
server_home => $carbon_home, },
}
In my init.pp file I'm filling my templates as follows using the said data structure.
define fill_templates($axis2, $carbon, $clustering, $serverOptions, $server_home) {
$ipAdd = $::ipaddress
$hostName = $::hostname
if $hostName == "${name}" {
notify {"host name match found for $hostName for $ipAdd":}
file { "${server_home}/repository/conf/axis2/axis2.xml":
ensure => file,
content => template('appserver/axis2.xml.erb'),
}
->
file { "${server_home}/repository/conf/carbon.xml":
ensure => file,
content => template('appserver/carbon.xml.erb'),
}
->
file { "${server_home}/repository/conf/tomcat/catalina-server.xml":
ensure => file,
content => template('appserver/catalina-server.xml.erb'),
}
}
}
As per the current method, if a matching node is found (say appserver-mgr) the respective data structure values are retrieved and applied to the templates.
Currently these scripts are working as expected.
Now I want to change it as follows.
I have a cluster containing following nodes.
appserver-mgr-1
appserver-mgr-2
appserver-mgr-3
appserver-wkr-1
appserver-wkr-2
appserver-wkr-3
appserver-wkr-4
appserver-wkr-5
By using the same data structure in params.pp file, how can I apply the appserver-mgr configuration to *.mgr nodes 1-3 and appserver-wkr configuration to *.wkr nodes 1-5?
Can I use regular expressions for this task?
I'm quite sure that it would be possible to bend the Puppet DSL to do what you need here. However, the far better approach to this issue is Hiera.
node /appserver-mgr/ {
$node_class = 'appserver'
$node_subclass = 'manager'
}
node /appserver-wrk/ {
$node_class = 'appserver'
$node_subclass = 'worker'
}
Use the node_class and node_subclass variables in your Hierarchy.
# /etc/puppet/hiera.yaml
---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "%{::clientcert}"
- "class-%{node_class}-%{node_subclass}"
- "class-%{node_class}"
- common
Now you can define your data right in the YAML for Hiera, instead of params.pp.
# /etc/puppet/hieradata/class-appserver-manager.yaml
servers:
axis2:
subDomain: mgt
carbon:
subDomain: mgt
serverOptions: -Dsetup
server_home: %{carbon_home}
and for the worker:
# /etc/puppet/hieradata/class-appserver-worker.yaml
servers:
axis2:
subDomain: worker
members:
- appserver-mgr2-ip
carbon:
subDomain: worker
serverOptions: -DworkerNode=true
server_home: %{carbon_home}
In your params class, the following then suffices:
$servers = hiera('servers')
Or you don't even bother with the params class, and just replace the uses of the $servers variable with hiera calls. But doing just one call in a params style class is a good practice.
Note: Using the variable value %{carbon_home} from Hiera is somewhat dangerous, you might want to hardcode the actual value in the YAML there.
I'm working in some tests for my code and I get my first "STOP" since I don't know how to mover forward on this. See in my setUp() function I load fixtures:
public function setUp() {
static::$kernel = static::createKernel();
static::$kernel->boot();
$this->em = static::$kernel->getContainer()->get('doctrine')->getManager();
$this->user = $this->createUser();
$fix = new MetaDetailGroupFixtures();
$fix->load($this->em);
parent::setUp();
}
But then I have remove that created data since I have a test for the bad case (when not entities are returned):
public function testListMetaDetailGroupFailAction() {
$client = static::createClient();
$this->logIn($client, $this->user);
$route = $client->getContainer()->get('router')->generate('meta-detail-group-list', array('parent_id' => 20000), false);
$client->request("GET", $route);
$decoded = json_decode($client->getResponse()->getContent(), true);
$this->assertCount(0, $decoded['entities']);
$this->assertArrayHasKey('success', $decoded);
$this->assertJsonStringEqualsJsonString(json_encode(array("success" => false, "message" => "No existen grupos de metadetalles de productos creados")), $client->getResponse()->getContent());
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame('application/json', $client->getResponse()->headers->get('Content-Type'));
$this->assertNotEmpty($client->getResponse()->getContent());
}
Since records are created in setup and they remain in DB that test fail. Any advice on this? How did yours solve that?
There is no easy way to do what you ask. What is normally done is truncating the database before and after executing your tests so you have a truly clean and isolated environment.
Quoting from this nice article (http://blog.sznapka.pl/fully-isolated-tests-in-symfony2/:
public function setUp()
{
$kernel = new \AppKernel("test", true);
$kernel->boot();
$this->_application = new \Symfony\Bundle\FrameworkBundle\Console\Application($kernel);
$this->_application->setAutoExit(false);
$this->runConsole("doctrine:schema:drop", array("--force" => true));
$this->runConsole("doctrine:schema:create");
$this->runConsole("doctrine:fixtures:load", array("--fixtures" => __DIR__ . "/../DataFixtures"));
}
As you can see, this solution makes use of Doctrine's Symfony2 commands in order to achieve the isolation state. There is a bundle I like to use which solves exactly this problem and let you use nice ready to use FunctionalTest base classes and many other features. Check it out:
https://github.com/liip/LiipFunctionalTestBundle
I'm looking for a way to test if a connection is working or not with Doctrine 2.
As in my application users can change by themselves the information connections, I want to check if the user has entered the right login and password.
How can I do that?
I tried to put this code into a try/catch block :
try{
$entityManager = $this->getEntityManager() ;
$repository = $entityManager->getRepository('Authentification\Entity\User');
$userToIdentify = $repository->findOneBy(array('login' => $this->_username, 'password' => $this->_password));
}catch(Exception $e){
$code = Result::FAILURE ;
$identity = "unknow" ;
$messages = array(
"message" => "Wrong login/password combination",
) ;
}
The problem is that even if the information connection is correct, I cannot catch the exception.
Otherwise I get the following error :
<b>Fatal error</b>: Uncaught exception 'Zend\View\Exception\RuntimeException'
with message 'Zend\View\Renderer\PhpRenderer::render: Unable to render template
"layout/layout"; resolver could not resolve to a file' in C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\View\Renderer\PhpRenderer.php:451 Stack trace: #0 C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\View\View.php(203): Zend\View\Renderer\PhpRenderer->render(Object(Zend\View\Model\ViewModel)) #1 C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\Mvc\View\Http\DefaultRenderingStrategy.php(128): Zend\View\View->render(Object(Zend\View\Model\ViewModel)) #2 [internal function]: Zend\Mvc\View\Http\DefaultRenderingStrategy->render(Object(Zend\Mvc\MvcEvent))#3 C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\EventManager\EventManager.php(469): call_user_func(Array, Object(Zend\Mvc\MvcEvent))#4 C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\EventManager\EventMa in <b>C:\xampp\htdocs\poemsV3\vendor\zendframework\zendframework\library\Zend\View\Renderer\PhpRenderer.php</b> on line <b>451</b><br />
Do you have any idea in how I could test if the connection works?
Thank you.
Do not use the EntityManager directly. You can instead use following to check the connection parameters:
try {
$entityManager->getConnection()->connect();
} catch (\Exception $e) {
// failed to connect
}
That's sadly the only real way to check if something went wrong, since the exception type changes depending on the driver you use.
For the other exception (the view-related one) you simply have to adjust your view scripts path. I suggest you to keep the skeleton application module enabled so that the default layout is always there: you can override it at any time.
You can use.
$cnx = $this->getDoctrine()->getConnection();
$cnx->isConnected() ?
'Connected' :
'not connected';
I've got a Doctrine Entity defined that maps to a View in my database. All works fine, the Entity relations work fine as expected.
Problem now is that when running orm:schema-manager:update on the CLI a table gets created for this entity which is something I want to prevent. There already is a view for this Entity, no need to create a table for it.
Can I annotate the Entity so that a table won't be created while still keeping access to all Entity related functionality (associations, ...)?
Based on the original alswer of ChrisR inspired in Marco Pivetta's post I'm adding here the solution if you're using Symfony2:
Looks like Symfony2 doesn't use the original Doctrine command at:
\Doctrine\ORM\Tools\Console\Command\SchemaTool\UpdateCommand
Instead it uses the one in the bundle:
\Doctrine\Bundle\DoctrineBundle\Command\Proxy\UpdateSchemaDoctrineCommand
So basically that is the class that must be extended, ending up in having:
src/Acme/CoreBundle/Command/DoctrineUpdateCommand.php:
<?php
namespace App\Command;
use Doctrine\Bundle\DoctrineBundle\Command\Proxy\UpdateSchemaDoctrineCommand;
use Doctrine\ORM\Mapping\ClassMetadata;
use Doctrine\ORM\Tools\SchemaTool;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
class DoctrineUpdateCommand extends UpdateSchemaDoctrineCommand
{
protected function executeSchemaCommand(InputInterface $input, OutputInterface $output, SchemaTool $schemaTool, array $metadatas, SymfonyStyle $ui): ?int
{
$ignoredEntities = [
'App\Entity\EntityToIgnore',
];
$metadatas = array_filter($metadatas, static function (ClassMetadata $classMetadata) use ($ignoredEntities) {
return !in_array($classMetadata->getName(), $ignoredEntities, true);
});
return parent::executeSchemaCommand($input, $output, $schemaTool, $metadatas, $ui);
}
}
Eventually it was fairly simple, I just had to subclass the \Doctrine\ORM\Tools\Console\Command\SchemaTool\UpdateCommand into my own CLI Command. In that subclass filter the $metadatas array that's being passed to executeSchemaCommand() and then pass it on to the parent function.
Just attach this new subclassed command to the ConsoleApplication you are using in your doctrine cli script and done!
Below is the extended command, in production you'll probably want to fetch the $ignoredEntities property from you config or something, this should put you on the way.
<?php
use Doctrine\ORM\Tools\Console\Command\SchemaTool\UpdateCommand;
use Doctrine\ORM\Tools\SchemaTool;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
class My_Doctrine_Tools_UpdateCommand extends UpdateCommand
{
protected $name = 'orm:schema-tool:myupdate';
protected $ignoredEntities = array(
'Entity\Asset\Name'
);
protected function executeSchemaCommand(InputInterface $input, OutputInterface $output, SchemaTool $schemaTool, array $metadatas, SymfonyStyle $ui)
{
/** #var $metadata \Doctrine\ORM\Mapping\ClassMetadata */
$newMetadata = [];
foreach ($metadatas as $metadata) {
if (!in_array($metadata->getName(), $this->ignoredEntities)) {
$newMetadata[] = $metadata;
}
}
return parent::executeSchemaCommand($input, $output, $schemaTool, $newMetadata, $ui);
}
}
PS: credits go to Marco Pivetta for putting me on the right track. https://groups.google.com/forum/?fromgroups=#!topic/doctrine-user/rwWXZ7faPsA
Quite old one but there is also worth nothing solution using Doctrine2: postGenerateSchema event listener - for me it's better than overriding
Doctrine classes:
namespace App\Doctrine\Listener;
use Doctrine\ORM\Tools\Event\GenerateSchemaEventArgs;
/**
* IgnoreTablesListener class
*/
class IgnoreTablesListener
{
private $ignoredTables = [
'table_name_to_ignore',
];
public function postGenerateSchema(GenerateSchemaEventArgs $args)
{
$schema = $args->getSchema();
$tableNames = $schema->getTableNames();
foreach ($tableNames as $tableName) {
if (in_array($tableName, $this->ignoredTables)) {
// remove table from schema
$schema->dropTable($tableName);
}
}
}
}
Also register listener:
# config/services.yaml
services:
ignore_tables_listener:
class: App\Doctrine\Listener\IgnoreTablesListener
tags:
- {name: doctrine.event_listener, event: postGenerateSchema }
No extra hooks is necessary.
In Doctrine 2.7.0 it was introduced the new SchemaIgnoreClasses entity manager config option that basically ignores the configured classes from any schema action.
To use it with Symfony we only need to add the schema_ignore_classes key in the Doctrine entity manager configuration like this:
doctrine:
dbal:
# your dbal configuration
orm:
default_entity_manager: default
entity_managers:
default:
connection: default
mappings:
Main:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity/Main'
prefix: 'App\Entity\Main'
alias: Main
schema_ignore_classes:
- Reference\To\My\Class
- Reference\To\My\OtherClass
$schema->getTableNames() was not working (I don't know why).
So:
<?php
namespace AppBundle\EventListener;
use Doctrine\Bundle\DoctrineBundle\Command\Proxy\UpdateSchemaDoctrineCommand;
use Doctrine\ORM\Tools\Event\GenerateSchemaEventArgs;
class IgnoreTablesListener extends UpdateSchemaDoctrineCommand
{
private $ignoredEntities = [
'YourBundle\Entity\EntityYouWantToIgnore',
];
/**
* Remove ignored tables /entities from Schema
*
* #param GenerateSchemaEventArgs $args
*/
public function postGenerateSchema(GenerateSchemaEventArgs $args)
{
$schema = $args->getSchema();
$em = $args->getEntityManager();
$ignoredTables = [];
foreach ($this->ignoredEntities as $entityName) {
$ignoredTables[] = $em->getClassMetadata($entityName)->getTableName();
}
foreach ($schema->getTables() as $table) {
if (in_array($table->getName(), $ignoredTables, true)) {
// remove table from schema
$schema->dropTable($table->getName());
}
}
}
}
And Register a service
# config/services.yaml
services:
ignore_tables_listener:
class: AppBundle\EventListener\IgnoreTablesListener
tags:
- {name: doctrine.event_listener, event: postGenerateSchema }
Worked fine! ;)
If problem is only with producing errors in db_view, when calling doctrine:schema:update command, why not simplest way:
remove # from #ORM\Entity annotation
execute doctrine:schema:update
add # to ORM\Entity annotation
;-)