Adding stored procedures to In-Memory DB using SqLite - unit-testing

I am using In-Memory database (using ServiceStack.OrmLite.Sqlite.Windows) for unit testing in servicestack based web api. I want to test the service endpoints which depends on stored Procedures through In-Memory database for which i have gone through the link Servicestack Ormlite SqlServerProviderTests, the unit test class that i am using for the test is as follows,
using System;
using System.Collections.Generic;
using System.Data;
using System.Linq;
using NUnit.Framework;
using ServiceStack.Text;
using ServiceStack.Configuration;
using ServiceStack.Data;
namespace ServiceStack.OrmLite.Tests
{
public class DummyTable
{
public int Id { get; set; }
public string Name { get; set; }
}
[TestFixture]
public class SqlServerProviderTests
{
private IDbConnection db;
protected readonly ServiceStackHost appHost;
public SqlServerProviderTests()
{
appHost = TestHelper.SetUp(appHost).Init();
db = appHost.Container.Resolve<IDbConnectionFactory>().OpenDbConnection("inventoryDb");
if (bool.Parse(System.Configuration.ConfigurationManager.AppSettings["IsMock"]))
TestHelper.CreateInMemoryDB(appHost);
}
[TestFixtureTearDown]
public void TearDown()
{
db.Dispose();
}
[Test]
public void Can_SqlColumn_StoredProc_returning_Column()
{
var sql = #"CREATE PROCEDURE dbo.DummyColumn
#Times integer
AS
BEGIN
SET NOCOUNT ON;
CREATE TABLE #Temp
(
Id integer NOT NULL,
);
declare #i int
set #i=1
WHILE #i < #Times
BEGIN
INSERT INTO #Temp (Id) VALUES (#i)
SET #i = #i + 1
END
SELECT * FROM #Temp;
DROP TABLE #Temp;
END;";
db.ExecuteSql("IF OBJECT_ID('DummyColumn') IS NOT NULL DROP PROC DummyColumn");
db.ExecuteSql(sql);
var expected = 0;
10.Times(i => expected += i);
var results = db.SqlColumn<int>("EXEC DummyColumn #Times", new { Times = 10 });
results.PrintDump();
Assert.That(results.Sum(), Is.EqualTo(expected));
results = db.SqlColumn<int>("EXEC DummyColumn 10");
Assert.That(results.Sum(), Is.EqualTo(expected));
results = db.SqlColumn<int>("EXEC DummyColumn #Times", new Dictionary<string, object> { { "Times", 10 } });
Assert.That(results.Sum(), Is.EqualTo(expected));
}
}
}
when i tried to execute this through Live-DB, it was working fine. but when i tried for In-Memory DB was getting Exceptions as follows,
System.Data.SQLite.SQLiteException : SQL logic error or missing database near "IF": syntax error
near the code line,
db.ExecuteSql("IF OBJECT_ID('DummyColumn') IS NOT NULL DROP PROC DummyColumn");
i commented the above line and executed the test case but still i am getting exception as follows,
System.Data.SQLite.SQLiteException : SQL logic error or missing database near "IF": syntax error
for the code line,
db.ExecuteSql(sql);
the In-Memory DB Created is as follows, and its working fine for remaining cases.
public static void CreateInMemoryDB(ServiceStackHost appHost)
{
using (var db = appHost.Container.Resolve<IDbConnectionFactory>().OpenDbConnection("ConnectionString"))
{
db.DropAndCreateTable<DummyData>();
TestDataReader<TableList>("Reservation.json", "InMemoryInput").Reservation.ForEach(x => db.Insert(x));
db.DropAndCreateTable<DummyTable>();
}
}
why we are facing this exception is there any other way to add and run stored Procedure in In-Memory DB with Sqlite??

The error is because you're trying to run SQL Server-specific queries with TSQL against an in memory version of Sqlite - i.e. a completely different, embeddable database. As the name suggests SqlServerProviderTests only works on SQL Server, I'm confused why you would try to run this against Sqlite?
SQLite doesn't support Stored Procedures, TSQL, etc so trying to execute SQL Server TSQL statements will always result in an error. The only thing you can do is fake it with a custom Exec Filter, where you can catch the exception and return whatever custom result you like, e.g:
public class MockStoredProcExecFilter : OrmLiteExecFilter
{
public override T Exec<T>(IDbConnection dbConn, Func<IDbCommand, T> filter)
{
try
{
return base.Exec(dbConn, filter);
}
catch (Exception ex)
{
if (dbConn.GetLastSql() == "exec sp_name #firstName, #age")
return (T)(object)new Person { FirstName = "Mocked" };
throw;
}
}
}
OrmLiteConfig.ExecFilter = new MockStoredProcExecFilter();

Related

org.apache.calcite.sql.validate.SqlValidatorException

I'm using Apache Calcite to parse a simple SQL statement and return its relational tree. I obtain a database schema using a JDBC connection to a simple SQLite database. The schema is then added using FrameworkConfig. The parser configuration is then modified to handle identifier quoting and case (not sensitive). However the SQL validator is unable to find the quoted table identifier in the SQL statement. Somehow the parser ignore the configuration settings and converts the table to UPPER CASE. A SqlValidatorException is raised, stating the the table name is not found. I suspect, the configuration is not being updated correctly? I have already validated that the table name is correctly included in the schema's meta-data.
public class ParseSQL {
public static void main(String[] args) {
try {
// register the JDBC driver
String sDriverName = "org.sqlite.JDBC";
Class.forName(sDriverName);
JsonObjectBuilder builder = Json.createObjectBuilder();
builder.add("jdbcDriver", "org.sqlite.JDBC")
.add("jdbcUrl",
"jdbc:sqlite://calcite/students.db")
.add("jdbcUser", "root")
.add("jdbcPassword", "root");
Map<String, JsonValue> JsonObject = builder.build();
//argument for JdbcSchema.Factory().create(....)
Map<String, Object> operand = new HashMap<String, Object>();
//explicitly extract JsonString(s) and load into operand map
for(String key : JsonObject.keySet()) {
JsonString value = (JsonString) JsonObject.get(key);
operand.put(key, value.getString());
}
final SchemaPlus rootSchema = Frameworks.createRootSchema(true);
Schema schema = new JdbcSchema.Factory().create(rootSchema, "students", operand);
rootSchema.add("students", schema);
//build a FrameworkConfig using defaults where values aren't required
Frameworks.ConfigBuilder configBuilder = Frameworks.newConfigBuilder();
//set defaultSchema
configBuilder.defaultSchema(rootSchema);
//build configuration
FrameworkConfig frameworkdConfig = configBuilder.build();
//use SQL parser config builder to ignore case of quoted identifier
SqlParser.configBuilder(frameworkdConfig.getParserConfig()).setQuotedCasing(Casing.UNCHANGED).build();
//use SQL parser config builder to set SQL case sensitive = false
SqlParser.configBuilder(frameworkdConfig.getParserConfig()).setCaseSensitive(false).build();
//get planner
Planner planner = Frameworks.getPlanner(frameworkdConfig);
//parse SQL statement
SqlNode sql_node = planner.parse("SELECT * FROM \"Students\" WHERE age > 15.0");
System.out.println("\n" + sql_node.toString());
//validate SQL
SqlNode sql_validated = planner.validate(sql_node);
//get associated relational expression
RelRoot relationalExpression = planner.rel(sql_validated);
relationalExpression.toString();
} catch (SqlParseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (RelConversionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ValidationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end main
} // end class
***** ERROR MESSAGE ******
Jan 20, 2016 8:54:51 PM org.apache.calcite.sql.validate.SqlValidatorException
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 'Students' not found
This is a case-sensitivity issue, similar to table not found with apache calcite. Because you enclosed the table name in quotes in your SQL statement, the validator is looking for a table called "Students", and the error message attests to this. If your table is called "Students", I am surprised that Calcite can't find it.
There is a problem with how you are using the SqlParser.ConfigBuilder. When you call build(), you are not using the SqlParser.Config object that it creates. If you passed that object to Frameworks.ConfigBuilder.parserConfig, I think you would get the behavior you want.

Same Instances header ( arff ) for all my database queries

I am using InstanceQuery , SQL queries, to construct my Instances. But my query results does not come in the same order always as it is normal in SQL.
Beacuse of this Instances constucted from different SQL has different headers. A simple example can be seen below. I suspect my results changes because of this behavior.
Header 1
#attribute duration numeric
#attribute protocol_type {tcp,udp}
#attribute service {http,domain_u}
#attribute flag {SF}
Header 2
#attribute duration numeric
#attribute protocol_type {tcp}
#attribute service {pm_dump,pop_2,pop_3}
#attribute flag {SF,S0,SH}
My question is : How can I give correct header information to Instance construction.
Is something like below workflow is possible?
get pre-prepared header information from arff file or another place.
give instance construction this header information
call sql function and get Instances (header + data)
I am using following sql function to get instances from database.
public static Instances getInstanceDataFromDatabase(String pSql
,String pInstanceRelationName){
try {
DatabaseUtils utils = new DatabaseUtils();
InstanceQuery query = new InstanceQuery();
query.setUsername(username);
query.setPassword(password);
query.setQuery(pSql);
Instances data = query.retrieveInstances();
data.setRelationName(pInstanceRelationName);
if (data.classIndex() == -1)
{
data.setClassIndex(data.numAttributes() - 1);
}
return data;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
I tried various approaches to my problem. But it seems that weka internal API does not allow solution to this problem right now. I modified weka.core.Instances append command line code for my purposes. This code is also given in this answer
According to this, here is my solution. I created a SampleWithKnownHeader.arff file , which contains correct header values. I read this file with following code.
public static Instances getSampleInstances() {
Instances data = null;
try {
BufferedReader reader = new BufferedReader(new FileReader(
"datas\\SampleWithKnownHeader.arff"));
data = new Instances(reader);
reader.close();
// setting class attribute
data.setClassIndex(data.numAttributes() - 1);
}
catch (Exception e) {
throw new RuntimeException(e);
}
return data;
}
After that , I use following code to create instances. I had to use StringBuilder and string values of instance, then I save corresponding string to file.
public static void main(String[] args) {
Instances SampleInstance = MyUtilsForWeka.getSampleInstances();
DataSource source1 = new DataSource(SampleInstance);
Instances data2 = InstancesFromDatabase
.getInstanceDataFromDatabase(DatabaseQueries.WEKALIST_QUESTION1);
MyUtilsForWeka.saveInstancesToFile(data2, "fromDatabase.arff");
DataSource source2 = new DataSource(data2);
Instances structure1;
Instances structure2;
StringBuilder sb = new StringBuilder();
try {
structure1 = source1.getStructure();
sb.append(structure1);
structure2 = source2.getStructure();
while (source2.hasMoreElements(structure2)) {
String elementAsString = source2.nextElement(structure2)
.toString();
sb.append(elementAsString);
sb.append("\n");
}
} catch (Exception ex) {
throw new RuntimeException(ex);
}
MyUtilsForWeka.saveInstancesToFile(sb.toString(), "combined.arff");
}
My save instances to file code is as below.
public static void saveInstancesToFile(String contents,String filename) {
FileWriter fstream;
try {
fstream = new FileWriter(filename);
BufferedWriter out = new BufferedWriter(fstream);
out.write(contents);
out.close();
} catch (Exception ex) {
throw new RuntimeException(ex);
}
This solves my problem but I wonder if more elegant solution exists.
I solved a similar problem with the Add filter that allows adding attributes to Instances. You need to add a correct Attibute with proper list of values to both datasets (in my case - to test dataset only):
Load train and test data:
/* "train" contains labels and data */
/* "test" contains data only */
CSVLoader csvLoader = new CSVLoader();
csvLoader.setFile(new File(trainFile));
Instances training = csvLoader.getDataSet();
csvLoader.reset();
csvLoader.setFile(new File(predictFile));
Instances test = csvLoader.getDataSet();
Set a new attribute with Add filter:
Add add = new Add();
/* the name of the attribute must be the same as in "train"*/
add.setAttributeName(training.attribute(0).name());
/* getValues returns a String with comma-separated values of the attribute */
add.setNominalLabels(getValues(training.attribute(0)));
/* put the new attribute to the 1st position, the same as in "train"*/
add.setAttributeIndex("1");
add.setInputFormat(test);
/* result - a compatible with "train" dataset */
test = Filter.useFilter(test, add);
As a result, the headers of both "train" and "test" are the same (compatible for Weka machine learning)

How do I export NHibernate schema for SQLite if I have dots in the schema name?

I'm trying to set up SQLite for unit testing with Fluent NHibernate as shown here but the table names isn't being generated as I expected.
Some tables have schemas with dots inside which seems to break the generation. (Dots works perfectly well with Microsoft SQL Server which I have in my production environment.)
Example:
[Foo.Bar.Schema].[TableName]
Result:
TestFixture failed: System.Data.SQLite.SQLiteException : SQLite error
unknown database Foo
How do I instruct SQLite to translate the dots to underscores or something so I can run my unit tests?
(I've tried adding brackets to the schema names with no success)
You can use a convention
http://wiki.fluentnhibernate.org/Conventions
*UPDATED
public static class PrivatePropertyHelper
{
// from http://stackoverflow.com/questions/1565734/is-it-possible-to-set-private-property-via-reflection
public static T GetPrivatePropertyValue<T>(this object obj, string propName)
{
if (obj == null) throw new ArgumentNullException("obj");
PropertyInfo pi = obj.GetType().GetProperty(propName, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
if (pi == null) throw new ArgumentOutOfRangeException("propName", string.Format("Property {0} was not found in Type {1}", propName, obj.GetType().FullName));
return (T)pi.GetValue(obj, null);
}
}
public class CustomTableNameConvention : IClassConvention
{
// Use this to set schema to specific value
public void Apply(FluentNHibernate.Conventions.Instances.IClassInstance instance)
{
instance.Schema("My_NEw_Schema");
instance.Table(instance.EntityType.Name.CamelToUnderscoreLower());
}
// Use this to alter the existing schema value.
// note that Schema is a private property and you need reflection to get it
public void Apply(FluentNHibernate.Conventions.Instances.IClassInstance instance)
{
instance.Schema(instance.GetPrivatePropertyValue<string>("Schema").Replace(".", "_"));
instance.Table(instance.EntityType.Name.CamelToUnderscoreLower());
}
}
You must use only one of he Apply methods.
*UPDATE 2
I don't know I would recommend this but if you like to experiment this seems to work. Even more reflection :)
public static void SetSchemaValue(this object obj, string schema)
{
var mapping_ref = obj.GetType().GetField("mapping", BindingFlags.Instance | BindingFlags.IgnoreCase | BindingFlags.GetField | BindingFlags.NonPublic).GetValue(obj);
var mapping = mapping_ref as ClassMapping;
if (mapping != null)
{
mapping.Schema = schema;
}
}
public void Apply(FluentNHibernate.Conventions.Instances.IClassInstance instance)
{
var schema = instance.GetPrivatePropertyValue<string>("Schema");
if (schema == null)
{
instance.Schema("My_New_Schema");
}
else
{
instance.SetSchemaValue("My_New_Schema");
}
}

Testing NHibernate with SQLite "No Such Table" - schema is generated

I'm trying to use an in-memory SQLite database to test my data layer which is provided by NHibernate.
I've read a load of blogs and articles about getting this setup but I'm now very confused as to why it isn't working.
The problem - when I run a unit test I get the error 'no such table: Student'. The articles I've read suggest this is because the schema isn't getting generated, or, the connection is being closed between my SchemaExport and query. I've checked everywhere I can think of and can't see how either of these scenarios are occuring.
My test output log looks like this:
OPEN CONNECTION
drop table if exists "Student"
drop table if exists "Tutor"
create table "Student" (
ID integer,
Name TEXT,
DoB DATETIME,
TutorId INTEGER,
primary key (ID)
)
create table "Tutor" (
ID integer,
Name TEXT,
primary key (ID)
)
NHibernate: INSERT INTO "Student" (Name, DoB, TutorId) VALUES (#p0, #p1, #p2); select last_insert_rowid();#p0 = 'Text1', #p1 = 01/12/2010 14:55:05, #p2 = NULL
14:55:05,750 ERROR [TestRunnerThread] AbstractBatcher [(null)]- Could not execute query: INSERT INTO "Student" (Name, DoB, TutorId) VALUES (#p0, #p1, #p2); select last_insert_rowid()
System.Data.SQLite.SQLiteException (0x80004005): SQLite error
no such table: Student
at System.Data.SQLite.SQLite3.Prepare(String strSql, SQLiteStatement previous, String& strRemain)
at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
at System.Data.SQLite.SQLiteCommand.GetStatement(Int32 index)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
14:55:05,781 ERROR [TestRunnerThread] ADOExceptionReporter [(null)]- SQLite error
no such table: Student
DISPOSE
CLOSING CONNECTION
Originally I was using my own code for the connection/session management but have moved to the code in this blog post translated to C# and with a couple changes to the DBConfig method and some debug statements to show the state of the connection.
private FluentNHibernate.Cfg.Db.IPersistenceConfigurer GetDBConfig()
{
return SQLiteConfiguration.Standard
.ConnectionString((ConnectionStringBuilder cs) => cs.Is(CONNECTION_STRING))
.ProxyFactoryFactory("NHibernate.ByteCode.LinFu.ProxyFactoryFactory, NHibernate.ByteCode.LinFu")
.Raw("connection.release_mode", "on_close");
}
I added the on_close after reading this
My test code looks like this:
[Test]
public void CanGetStudentById()
{
using (var scope = new SQLiteDatabaseScope<StudentMapping>())
{
using (ISession sess = scope.OpenSession())
{
// Arrange
var repo = new StudentRepository();
repo.Save(new Student() { Name = "Text1", DoB = DateTime.Now });
// Act
var student = repo.GetById(1);
// Assert
Assert.IsNotNull(student);
Assert.AreEqual("Text1", student.Name);
}
}
}
What have I overlooked here?
Update: I created a copy of the class that connects to an SQLite file DB and it worked fine. So it has to be something to do with the connection being closed.
If you change your test method to the following, does it work?
[Test]
public void CanGetStudentById()
{
using (var scope = new SQLiteDatabaseScope<StudentMapping>())
{
using (ISession sess = scope.OpenSession())
{
// Arrange
sess.Save(new Student() { Name = "Text1", DoB = DateTime.Now });
// Act
var student = sess.Get<Student>(1);
// Assert
Assert.IsNotNull(student);
Assert.AreEqual("Text1", student.Name);
}
}
}
I would hazard to guess that your StudentRepository is opening its own session and hence doesn't see the table.

Server architecture and SubSonic

Im trying to develop an server /client application. The server will be a bunch of webservices, the idea was to expose methods like:
Company GetNewCompany(); //Creates an new Company Object
Save(Company C);
CompanyCollection GetCompany(Query q);
Where Query object is part of Subsonic 2.1. But the problem is that SubSonic is not built for this, Have I missed something here? or is it just impossible to to use subsonic query language over SOAP?
This would have been great feature, becuase then it is really easy to make an application server using subsonic.
Br
Soren.
If you want to use subsonic v3 you can look at this issue that talks about IUpdatable:
http://code.google.com/p/subsonicthree/issues/detail?id=30
This will let you use ado data services somewhat painlessly. You use a DB constructor that take a URI argument. This probably won't be a part of v3 but you could make changes like this yourself.
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using WcfClientTest.NorthwindService;
namespace WcfClientTest
{
/// <summary>
/// Summary description for WcfTest
/// To run these tests, load this project, and somehow get a server running at the URI.
/// This can be done by updating the service reference to start the development server.
/// </summary>
[TestClass]
public class WcfTest
{
private string baseURI = "http://127.0.0.1:49649/Northwind.svc";
private DB ctx;
/// <summary>
/// Sets up test.
/// </summary>
[TestInitialize]
public void SetUp()
{
ctx = new DB(new Uri(baseURI));
}
[TestCleanup]
public void Cleanup()
{
}
[TestMethod]
public void Select_Simple_With_Variable()
{
int categoryID = 5;
IQueryable<Product> result = from p in ctx.Products
where p.CategoryID == categoryID
select p;
List<Product> products = result.ToList();
Assert.AreEqual(7, products.Count());
}
[TestMethod]
public void TestAddNew()
{
// add customer
var c = new Customer
{
CustomerID = "XXXXX",
ContactTitle = "Prez",
Country = "USA",
ContactName = "Big Guy",
CompanyName = "Big Guy Company"
};
ctx.AddToCustomers(c);
ctx.SaveChanges();
IQueryable<Customer> qCustomer = from cust in ctx.Customers
where cust.CustomerID == "XXXXX"
select cust;
Customer c2 = qCustomer.FirstOrDefault();
Assert.AreEqual("XXXXX", c2.CustomerID);
if (c2 != null)
{
ctx.DeleteObject(c2);
}
ctx.SaveChanges();
IQueryable<Customer> qCustomer2 = from cust in ctx.Customers
where cust.ContactName == "Big Guy"
select cust;
// Returns null if the row isn't found.
Customer c3 = qCustomer2.SingleOrDefault();
Assert.AreEqual(null, c3);
}
}
}
And this is all there is to the service:
using System.Data.Services;
using Northwind;
namespace NorthwindService
{
[System.ServiceModel.ServiceBehavior(IncludeExceptionDetailInFaults=false)]
public class Northwind: DataService<DB>
{
// This method is called only once to initialize service-wide policies.
public static void InitializeService(IDataServiceConfiguration config)
{
config.SetEntitySetAccessRule("*", EntitySetRights.All);
config.UseVerboseErrors = true;
}
}
}
And for web.config:
<system.serviceModel>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
</system.serviceModel>