Doctrine DTO with native SQL queries - doctrine-orm

I currently have a fairly complex native SQL query which is used for reporting purposes. Given the amount of data it processes this is the only efficient way to handle it is with native SQL.
This works fine and returns an array of arrays from the scalar results.
What I'd like to do, to keep the results consistent with every other result set in the project is use a Data Transfer Object (DTO). Returning an array of simple DTO objects.
These work really well with DQL but I can't see anyway of using them with native SQL. Is this at all possible?

Doctrine can map the results of a raw SQL query to an entity, as shown here:
http://doctrine-orm.readthedocs.org/projects/doctrine-orm/en/latest/reference/native-sql.html
I cannot see support for DTOs unless you are willing to use DQL as well, so a direct solution does not exist. I tried my hand at a simple workaround that works well enough, so here are the DQL and non-DQL ways to achieve your goal.
The examples were built using Laravel and the Laravel Doctrine extension.
The DTO
The below DTO supports both DQL binding and custom mapping so the constructor must be able to work with and without parameters.
<?php namespace App\Dto;
/**
* Date with corresponding statistics for the date.
*/
class DateTotal
{
public $taskLogDate;
public $totalHours;
/**
* DateTotal constructor.
*
* #param $taskLogDate The date for which to return totals
* #param $totalHours The total hours worked on the given date
*/
public function __construct($taskLogDate = null, $totalHours = null)
{
$this->taskLogDate = $taskLogDate;
$this->totalHours = $totalHours;
}
}
Using DQL to fetch results
Here is the standard version, using DQL.
public function findRecentDateTotals($taskId)
{
$fromDate = new DateTime('6 days ago');
$fromDate->setTime(0, 0, 0);
$queryBuilder = $this->getQueryBuilder();
$queryBuilder->select('NEW App\Dto\DateTotal(taskLog.taskLogDate, SUM(taskLog.taskLogHours))')
->from('App\Entities\TaskLog', 'taskLog')
->where($queryBuilder->expr()->orX(
$queryBuilder->expr()->eq('taskLog.taskLogTask', ':taskId'),
$queryBuilder->expr()->eq(0, ':taskId')
))
->andWhere(
$queryBuilder->expr()->gt('taskLog.taskLogDate', ':fromDate')
)
->groupBy('taskLog.taskLogDate')
->orderBy('taskLog.taskLogDate', 'DESC')
->setParameter(':fromDate', $fromDate)
->setParameter(':taskId', $taskId);
$result = $queryBuilder->getQuery()->getResult();
return $result;
}
Support for DTOs with native SQL
Here is a simple helper that can marshal the array results of a raw SQL query into objects. It can be extended to do other stuff as well, perhaps custom updates and so on.
<?php namespace App\Dto;
use Doctrine\ORM\EntityManager;
/**
* Helper class to run raw SQL.
*
* #package App\Dto
*/
class RawSql
{
/**
* Run a raw SQL query.
*
* #param string $sql The raw SQL
* #param array $parameters Array of parameter names mapped to values
* #param string $className The class to pack the results into
* #return Object[] Array of objects mapped from the array results
* #throws \Doctrine\DBAL\DBALException
*/
public static function query($sql, $parameters, $className)
{
/** #var EntityManager $em */
$em = app('em');
$statement = $em->getConnection()->prepare($sql);
$statement->execute($parameters);
$results = $statement->fetchAll();
$return = array();
foreach ($results as $result) {
$resultObject = new $className();
foreach ($result as $key => $value) {
$resultObject->$key = $value;
}
$return[] = $resultObject;
}
return $return;
}
}
Running the raw SQL version
The function is used and called in the same way as other repository methods, and just calls on the above helper to automate the conversion of data to objects.
public function findRecentDateTotals2($taskId)
{
$fromDate = new DateTime('6 days ago');
$sql = "
SELECT
task_log.task_log_date AS taskLogDate,
SUM(task_log.task_log_hours) AS totalHours
FROM task_log task_log
WHERE (task_log.task_log_task = :taskId OR :taskId = 0) AND task_log.task_log_date > :fromDate
GROUP BY task_log_date
ORDER BY task_log_date DESC
";
$return = RawSql::query(
$sql,
array(
'taskId' => $taskId,
'fromDate' => $fromDate->format('Y-m-d')
),
DateTotal::class
);
return $return;
}
Notes
I would not dismiss DQL too quickly as it can perform most kinds of SQL. I have however also recently been involved in building management reports, and in the world of management information the SQL queries can be as large as whole PHP files. In that case I would join you and abandon Doctrine (or any other ORM) as well.

Related

Doctrine2 - Doctrine generating query with associated entity - InvalidFieldNameException

Yep, the title suggests: Doctrine is looking for a fieldname that's not there. That's both true and not true at the same time, though I cannot figure out how to fix it.
The full error:
File: D:\path\to\project\vendor\doctrine\dbal\lib\Doctrine\DBAL\Driver\AbstractMySQLDriver.php:71
Message: An exception occurred while executing 'SELECT DISTINCT id_2
FROM (SELECT p0_.name AS name_0, p0_.code AS code_1, p0_.id AS id_2
FROM product_statuses p0_) dctrn_result ORDER BY p0_.language_id ASC, name_0 ASC LIMIT 25
OFFSET 0':
SQLSTATE[42S22]: Column not found: 1054 Unknown column
'p0_.language_id' in 'order clause'
The query the error is caused by (from error above):
SELECT DISTINCT id_2
FROM (
SELECT p0_.name AS name_0, p0_.code AS code_1, p0_.id AS id_2
FROM product_statuses p0_
) dctrn_result
ORDER BY p0_.language_id ASC, name_0 ASC
LIMIT 25 OFFSET 0
Clearly, that query is not going to work. The ORDER BY should be in the sub-query, or else it should replace p0_ in the ORDER BY with dctrn_result and also get the language_id column in the sub-query to be returned.
The query is build using the QueryBuilder in the indexAction of a Controller in Zend Framework. All is very normal and the same function works perfectly fine when using a the addOrderBy() function for a single ORDER BY statement. In this instance I wish to use 2, first by language, then by name. But the above happens.
If someone knows a full solution to this (or maybe it's a bug?), that would be nice. Else a hint in the right direction to help me solve this issue would be greatly appreciated.
Below additional information - Entity and indexAction()
ProductStatus.php - Entity - Note the presence of language_id column
/**
* #ORM\Table(name="product_statuses")
* #ORM\Entity(repositoryClass="Hzw\Product\Repository\ProductStatusRepository")
*/
class ProductStatus extends AbstractEntity
{
/**
* #var string
* #ORM\Column(name="name", type="string", length=255, nullable=false)
*/
protected $name;
/**
* #var string
* #ORM\Column(name="code", type="string", length=255, nullable=false)
*/
protected $code;
/**
* #var Language
* #ORM\ManyToOne(targetEntity="Hzw\Country\Entity\Language")
* #ORM\JoinColumn(name="language_id", referencedColumnName="id")
*/
protected $language;
/**
* #var ArrayCollection|Product[]
* #ORM\OneToMany(targetEntity="Hzw\Product\Entity\Product", mappedBy="status")
*/
protected $products;
[Getters/Setters]
}
IndexAction - Removed parts not directly related to QueryBuilder. Added in comments showing params as they are.
/** #var QueryBuilder $qb */
$qb = $this->getEntityManager()->createQueryBuilder();
$qb->select($asParam) // 'pro'
->from($emEntity, $asParam); // Hzw\Product\Entity\ProductStatus, 'pro'
if (count($queryParams) > 0 && !is_null($query)) {
// [...] creates WHERE statement, unused in this instance
}
if (isset($orderBy)) {
if (is_array($orderBy)) {
// !!! This else is executed !!! <-----
if (is_array($orderDirection)) { // 'ASC'
// [...] other code
} else {
// $orderBy = ['language', 'name'], $orderDirection = 'ASC'
foreach ($orderBy as $orderParam) {
$qb->addOrderBy($asParam . '.' . $orderParam, $orderDirection);
}
}
} else {
// This works fine. A single $orderBy with a single $orderDirection
$qb->addOrderBy($asParam . '.' . $orderBy, $orderDirection);
}
}
================================================
UPDATE: I found the problem
The above issue is not caused by incorrect mapping or a possible bug. It's that the QueryBuilder does not automatically handle associations between entities when creating queries.
My expectation was that when an entity, such as ProductStatus above, contains the id's of the relation (i.e. language_id column), that it would be possible to use those properties in the QueryBuilder without issues.
Please see my own answer below how I fixed my functionality to be able to have a default handling of a single level of nesting (i.e. ProducStatus#language == Language, be able to use language.name as ORDER BY identifier).
Ok, after more searching around and digging into how and where this goes wrong, I found out that Doctrine does not handle relation type properties of entities during the generation of queries; or maybe does not default to using say, the primary key of an entity if nothing is specified.
In the use case of my question above, the language property is of a #ORM\ManyToOne association to the Language entity.
My use case calls for the ability to handle at lease one level of relations for default actions. So after I realized that this is not handled automatically (or with modifications such as language.id or language.name as identifiers) I decided to write a little function for it.
/**
* Adds order by parameters to QueryBuilder.
*
* Supports single level nesting of associations. For example:
*
* Entity Product
* product#name
* product#language.name
*
* Language being associated entity, but must be ordered by name.
*
* #param QueryBuilder $qb
* #param string $tableKey - short alias (e.g. 'tab' with 'table AS tab') used for the starting table
* #param string|array $orderBy - string for single orderBy, array for multiple
* #param string|array $orderDirection - string for single orderDirection (ASC default), array for multiple. Must be same count as $orderBy.
*/
public function createOrderBy(QueryBuilder $qb, $tableKey, $orderBy, $orderDirection = 'ASC')
{
if (!is_array($orderBy)) {
$orderBy = [$orderBy];
}
if (!is_array($orderDirection)) {
$orderDirection = [$orderDirection];
}
// $orderDirection is an array. We check if it's of equal length with $orderBy, else throw an error.
if (count($orderBy) !== count($orderDirection)) {
throw new \InvalidArgumentException(
$this->getTranslator()->translate(
'If you specify both OrderBy and OrderDirection as arrays, they should be of equal length.'
)
);
}
$queryKeys = [$tableKey];
foreach ($orderBy as $key => $orderParam) {
if (strpos($orderParam, '.')) {
if (substr_count($orderParam, '.') === 1) {
list($entity, $property) = explode('.', $orderParam);
$shortName = strtolower(substr($entity, 0, 3)); // Might not be unique...
$shortKey = $shortName . '_' . (count($queryKeys) + 1); // Now it's unique, use $shortKey when continuing
$queryKeys[] = $shortKey;
$shortName = strtolower(substr($entity, 0, 3));
$qb->join($tableKey . '.' . $entity, $shortName, Join::WITH);
$qb->addOrderBy($shortName . '.' . $property, $orderDirection[$key]);
} else {
throw new \InvalidArgumentException(
$this->getTranslator()->translate(
'Only single join statements are supported. Please write a custom function for deeper nesting.'
)
);
}
} else {
$qb->addOrderBy($tableKey . '.' . $orderParam, $orderDirection[$key]);
}
}
}
It by no means supports everything the QueryBuilder offers and is definitely not a final solution. But it gives a starting point and solid "default functionality" for an abstract function.

What is the best practice for repository?

In my repositories, I have methods with too many arguments (for use in where) :
Example :
class ProchaineOperationRepository extends EntityRepository
{
public function getProchaineOperation(
$id = null, // Search by ID
\DateTime $dateMax = null, // Search by DateMax
\DateTime $dateMin = null, // Search by DateMin
$title = null // Search by title
)
In my controllers, I have differents action ... for get with ID, for get with ID and DateMin, for get ID and Title, ...
My method is too illegible because too many arguments ... and it would be difficult to create many methods because they are almost identical ...
What is the best practice ?
You have two main concerns in your question
You have too many arguments in your repository method which will be used in 'where' condition of the eventual query. You want to organize them in a better way
The repository method should be callable from the controller in a meaningful way because of possible complexity of arguments passed
I suggest you to write a Repository method like:
namespace AcmeBundle\Repository;
/**
* ProchaineOperationRepository
*
*/
class ProchaineOperationRepository extends \Doctrine\ORM\EntityRepository
{
public function search($filters, $sortBy = "id", $orderBy = "DESC")
{
$qb = $this->createQueryBuilder("po");
foreach ($filters as $key => $value){
$qb->andWhere("po.$key='$value'");
}
$qb->addOrderBy("po.$sortBy", $orderBy);
return $qb->getQuery()->getArrayResult();
}
}
The $filters variable here is an array which is supposed to hold the filters you are going to use in 'where' condition. $sortBy and $orderBy should also be useful to get the result in properly sequenced way
Now, you can call the repository method from your controller like:
class ProchaineOperationController extends Controller
{
/**
* #Route("/getById/{id}")
*/
public function getByIdAction($id)
{
$filters = ['id' => $id];
$result = $this->getDoctrine()->getRepository("AcmeBundle:ProchaineOperation")->search($filters);
//process $result
}
/**
* #Route("/getByTitle/{title}")
*/
public function getByTitleAction($title)
{
$filters = ['title' => $title];
$sortBy = 'title';
$result = $this->getDoctrine()->getRepository("AcmeBundle:ProchaineOperation")->search($filters, $sortBy);
//process $result
}
/**
* #Route("/getByIdAndDateMin/{id}/{dateMin}")
*/
public function getByIdAndDateMinAction($id, $dateMin)
{
$filters = ['id' => $id, 'dateMin' => $dateMin];
$sortBy = "dateMin";
$orderBy = "ASC";
$result = $this->getDoctrine()->getRepository("AcmeBundle:ProchaineOperation")->search($filters, $sortBy, $orderBy);
//process $result
}
}
Note that you are calling the same repository method for all controller actions with minor changes according to your parameters. Also note that $sortBy and $orderBy are optionally passed.
Hope it helps!
If your objective is only to query with an AND operator between each properties, the best way could be to use the method proposed by doctrine for that : findBy() cf : this part of the doc
for instance :
$results = $this
->getDoctrine()
->getRepository('AppBundle:ProchaineOperation')
->findBy(array('dateMax' => $myDate, 'title' => 'Hello world');
EDIT : after comment
Then use the same way as Doctrine do : Pass only an array with id, dateMax... as keys if these are set. This should be solve the method signature problem which gives you so much trouble. :)

doctrine2 DQL subquery 1 row limit

I want to do a DQL query like:
$dql = "select p
from AcmeDemoBundle:UserTypeA p
where p.UserTypeB = :id
and (
select top 1 r.boolean
from AcmeDemoBundle:Registry r
)
= true";
But it seems that TOP 1 it's not a valid function in doctrine2.
I can't figure out how can I limit the result of the subquery to one row.
DQL does not support limits on subqueries and neither LIMIT nor OFFSET.
See http://www.doctrine-project.org/jira/browse/DDC-885
Although Doctrine doesn't natively support this, you could implement a custom function named FIRST() to achieve this:
<?php
use Doctrine\ORM\Query\AST\Functions\FunctionNode;
use Doctrine\ORM\Query\AST\Subselect;
use Doctrine\ORM\Query\Lexer;
use Doctrine\ORM\Query\Parser;
use Doctrine\ORM\Query\SqlWalker;
/**
* FirstFunction ::=
* "FIRST" "(" Subselect ")"
*/
class FirstFunction extends FunctionNode
{
/**
* #var Subselect
*/
private $subselect;
/**
* {#inheritdoc}
*/
public function parse(Parser $parser)
{
$parser->match(Lexer::T_IDENTIFIER);
$parser->match(Lexer::T_OPEN_PARENTHESIS);
$this->subselect = $parser->Subselect();
$parser->match(Lexer::T_CLOSE_PARENTHESIS);
}
/**
* {#inheritdoc}
*/
public function getSql(SqlWalker $sqlWalker)
{
return '(' . $this->subselect->dispatch($sqlWalker) . ' LIMIT 1)';
}
}
(More details: https://www.colinodell.com/blog/201703/limiting-subqueries-doctrine-2-dql)
You should really only use this for read-only purposes since Doctrine will not include other related entities in the result (which may become orphaned or lost if you save it).

Doctrine 2 query builder vs entity persist performance

Summary: which is quicker: updating / flushing a list of entities, or running a query builder update on each?
We have the following situation in Doctrine ORM (version 2.3).
We have a table that looks like this
cow
wolf
elephant
koala
and we would like to use this table to sort a report of a fictional farm. The problem is that the user wishes to have a customer ordering of the animals (e.g. Koala, Elephant, Wolf, Cow). Now there exist possibilities using CONCAT, or CASE to add a weight to the DQL (example 0002wolf, 0001elephant). In my experience this is either tricky to build and when I got it working the result set was an array and not a collection.
So, to solve this we added a "weight" field to each record and, before running the select, we assign each one with a weight:
$animals = $em->getRepository('AcmeDemoBundle:Animal')->findAll();
foreach ($animals as $animal) {
if ($animal->getName() == 'koala') {
$animal->setWeight(1);
} else if ($animal->getName() == 'elephant') {
$animal->setWeight(2);
}
// etc
$em->persist($animal);
}
$em->flush();
$query = $em->createQuery(
'SELECT c FROM AcmeDemoBundle:Animal c ORDER BY c.weight'
);
This works perfectly. To avoid race conditions we added this inside a transaction block:
$em->getConnection()->beginTransaction();
// code from above
$em->getConnection()->rollback();
This is a lot more robust as it handles multiple users generating the same report. Alternatively the entities can be weighted like this:
$em->getConnection()->beginTransaction();
$qb = $em->createQueryBuilder();
$q = $qb->update('AcmeDemoBundle:Animal', 'c')
->set('c.weight', $qb->expr()->literal(1))
->where('c.name = ?1')
->setParameter(1, 'koala')
->getQuery();
$p = $q->execute();
$qb = $em->createQueryBuilder();
$q = $qb->update('AcmeDemoBundle:Animal', 'c')
->set('c.weight', $qb->expr()->literal(2))
->where('c.name = ?1')
->setParameter(1, 'elephant')
->getQuery();
$p = $q->execute();
// etc
$query = $em->createQuery(
'SELECT c FROM AcmeDemoBundle:Animal c ORDER BY c.weight'
);
$em->getConnection()->rollback();
Questions:
1) which of the two examples would have better performance?
2) Is there a third or better way to do this bearing in mind we need a collection as a result?
Please remember that this is just an example - sorting the result set in memory is not an option, it must be done on the database level - the real statement is a 10 table join with 5 orderbys.
Initially you could make use of a Doctrine implementation named Logging (\Doctrine\DBAL\LoggingProfiler). I know that it is not the better answer, but at least you can implement it in order to get best result for each example that you have.
namespace Doctrine\DBAL\Logging;
class Profiler implements SQLLogger
{
public $start = null;
public function __construct()
{
}
/**
* {#inheritdoc}
*/
public function startQuery($sql, array $params = null, array $types = null)
{
$this->start = microtime(true);
}
/**
* {#inheritdoc}
*/
public function stopQuery()
{
echo "execution time: " . microtime(true) - $this->start;
}
}
In you main Doctrine configuration you can enable as:
$logger = new \Doctrine\DBAL\Logging\Profiler;
$config->setSQLLogger($logger);

Profiling Doctrine 2.0 in Codeigniter

What's a good way to profile doctrine queries when Doctrine 2.0 has been integrated into codeigniter?
Using the usual CI profiler does not how the queries executed because it's using Doctrine and not the native, active record.
e.g. when you add this code $this->output->enable_profiler(TRUE); it should also show the queries executed.
http://codeigniter.com/user_guide/general/profiling.html
You can add a profiler in the doctrine package
namespace Doctrine\DBAL\Logging;
class Profiler implements SQLLogger
{
public $start = null;
private $ci;
public function __construct()
{
$this->ci =& get_instance();
}
/**
* {#inheritdoc}
*/
public function startQuery($sql, array $params = null, array $types = null)
{
$this->start = microtime(true);
$this->ci->db->queries[] = "/* doctrine */ \n".$sql;
}
/**
* {#inheritdoc}
*/
public function stopQuery()
{
$this->ci->db->query_times[] = microtime(true) - $this->start;
}
}
Then load the profiler as a logger in your main doctrine library (doctrine.php for me)
$logger = new \Doctrine\DBAL\Logging\Profiler;
$config->setSQLLogger($logger);
And the normal profiling will work fine.
Compatible with CodeIgniter..
https://github.com/ahmetkapikiran/CodeIgniter-Doctrine-Profiler