We are using aggregators to keep a count for exceptions during processing
public class BigTableWriter extends DoFn<String, Void> {
private Aggregator<Integer, Integer> errorAggregator;
public BigTableWriter(CloudBigtableOptions options) {
errorAggregator = createAggregator("errors",new Sum.SumIntegerFn());
}
#Override
public void processElement(DoFn<String, Void>.ProcessContext c){
try {
....do work here
}
catch(Exception ex){
errorAggregator.addValue(1);
}
}
}
We'd like to make this more granular rather than keeping a single aggregator to collect the errors. The documentation says that aggregator is typically created in the constructor. Is it possible to create an aggregator for each exception type inside of our catch block? For instance, we want to do something like.
public class BigTableWriter extends DoFn<String, Void> {
private Map<String, Aggregator<Integer, Integer> aggregatorMap;
public BigTableWriter(CloudBigtableOptions options) {
aggregatorMap = new HashMap<>();
}
#Override
public void processElement(DoFn<String, Void>.ProcessContext c){
try {
....do work here
}
catch(Exception ex){
aggregateException(ex.getCause().getMessage());
}
}
public void aggregateException(String exceptionMessage) {
Aggregator<Integer, Integer> aggregator = null;
if(!aggregatorMap.containsKey(exceptionMessage){
aggregator = createAggregator(exceptionMessage,new Sum.SumIntegerFn());
}
else {
aggregator = aggregatorMap.get(exceptionMessage);
}
aggregator.put(exceptionMessage, aggregator);
}
}
Unfortunately, no. The current logic for initializing aggregators requires that they are known at graph construction time (aka. created during the DoFn construction). It's a good feature request though. Created an issue to track it here: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/55
Related
Is it possible in DryIoc container to figure out whether some singleton has been instantiated?
For instance
var container = new Container();
container.Register<IApplicationContext, ApplicationContext>( Reuse.Singleton );
// var context = container.Resolve<IApplicationContext>();
if ( container.IsInstantiated<IApplicationContext>() ) // Apparently this does not compile
{
// ...
}
// OR
if ( container.IsInstantiated<ApplicationContext>() )
{
// ...
}
There is no way at the moment and no such feature planned. You may create an issue to request this.
But I am wandering why it is needed. Cause singleton provides a guarantee to be created only once, so you may not worry or check for double creation.
Is it for something else?
Update
OK, in DryIoc you may register a "decorator" to control and provide information about decoratee creation, here is more on decorators:
[TestFixture]
public class SO_IsInstantiatedViaDecorator
{
[Test]
public void Test()
{
var c = new Container();
c.Register<IService, X>(Reuse.Singleton);
c.Register<XProvider>(Reuse.Singleton);
c.Register<IService>(
Made.Of(_ => ServiceInfo.Of<XProvider>(), p => p.Create(Arg.Of<Func<IService>>())),
Reuse.Singleton,
Setup.Decorator);
c.Register<A>();
var x = c.Resolve<XProvider>();
Assert.IsFalse(x.IsCreated);
c.Resolve<A>();
Assert.IsTrue(x.IsCreated);
}
public interface IService { }
public class X : IService { }
public class A
{
public A(IService service) { }
}
public class XProvider
{
public bool IsCreated { get; private set; }
public IService Create(Func<IService> factory)
{
IsCreated = true;
return factory();
}
}
}
This example also illustrates how powerful is composition of DryIoc decorators and factory methods.
I am using the [AutoNSubstituteData] attribute, which was posted here:
AutoFixture, xUnit.net, and Auto Mocking
I would like to combine this with the [PropertyData("")] attribute from xunit extensions.
This is my test:
public static IEnumerable<string[]> InvalidInvariant
{
get
{
yield return new string[] { null };
yield return new [] { string.Empty };
yield return new [] { " " };
}
}
[Theory, AutoNSubstituteData, PropertyData("InvalidInvariant")]
public void TestThatGuardsAreTriggeredWhenConnectionStringArgumentIsInvalid(
IDeal deal,
IDbConnection conn,
IDb db,
ISender sender,
string invalidConnString,
string query)
{
deal.Init.Group.Returns(Group.A);
deal.Aggr.Group.Returns(Group.A);
deal.Product.Commodity.Returns(Product.Commodity.E);
var sut = new Handler(db, sender);
Assert.Throws<ArgumentException>(() =>
sut.HandleDeal(deal, conn, invalidConnString, query));
}
Is there a way to combine these attributes or to get the desired functionality (mock everything, except for invalidConnstring, which should be filled with the property-data)?
There are two ways to do this:
Option 1 - Using AutoFixture.Xunit and the CompositeDataAttribute class:
internal class AutoNSubstituteDataAttribute : AutoDataAttribute
{
internal AutoNSubstituteDataAttribute()
: base(new Fixture().Customize(new AutoNSubstituteCustomization()))
{
}
}
internal class AutoNSubstitutePropertyDataAttribute : CompositeDataAttribute
{
internal AutoNSubstitutePropertyDataAttribute(string propertyName)
: base(
new DataAttribute[] {
new PropertyDataAttribute(propertyName),
new AutoNSubstituteDataAttribute() })
{
}
}
Define the test cases as below:
public class Scenario
{
public static IEnumerable<object[]> InvalidInvariantCase1
{
get
{
yield return new string[] { null };
}
}
public static IEnumerable<object[]> InvalidInvariantCase2
{
get
{
yield return new string[] { string.Empty };
}
}
public static IEnumerable<object[]> InvalidInvariantCase3
{
get
{
yield return new string[] { " " };
}
}
}
Then declare the parameterized test as:
public class Scenarios
{
[Theory]
[AutoNSubstitutePropertyData("InvalidInvariantCase1")]
[AutoNSubstitutePropertyData("InvalidInvariantCase2")]
[AutoNSubstitutePropertyData("InvalidInvariantCase3")]
public void AParameterizedTest(
string invalidConnString,
IDeal deal,
IDbConnection conn,
IDb db,
ISender sender,
string query)
{
}
}
Please note that the parameterized parameter invalidConnString have to be declared before the other parameters.
Option 2 - Using Exude:
public class Scenario
{
public void AParameterizedTest(
IDeal deal,
IDbConnection conn,
IDb db,
ISender sender,
string invalidConnString,
string query)
{
}
[FirstClassTests]
public static TestCase<Scenario>[] RunAParameterizedTest()
{
var testCases = new []
{
new
{
invalidConnString = (string)null
},
new
{
invalidConnString = string.Empty
},
new
{
invalidConnString = " "
}
};
var fixture = new Fixture()
.Customize(new AutoNSubstituteCustomization());
return testCases
.Select(tc =>
new TestCase<Scenario>(
s => s.AParameterizedTest(
fixture.Create<IDeal>(),
fixture.Create<IDbConnection>(),
fixture.Create<IDb>(),
fixture.Create<ISender>(),
tc.invalidConnString,
fixture.Create<string>())))
.ToArray();
}
}
The [Theory] attribute works by looking for one or more 'data source attributes'; for example
[InlineData]
[PropertyData]
[ClassData]
etc.
The [AutoData] attribute is just another such attribute, as is your derived [AutoNSubstituteData] attribute.
It's possible to add more than one 'data source attribute' to the same [Theory], as witnessed by the idiomatic use of the [InlineData] attribute:
[Theory]
[InlineData("foo")]
[InlineData("bar")]
[InlineData("baz")]
public void MyTest(string text)
This produces three test cases.
It's also possible to combine [PropertyData] and [AutoData], but it probably doesn't do what you want it to do. This:
[Theory]
[AutoNSubstituteData]
[PropertyData("InvalidInvariant")]
public void MyTest(/* parameters go here */)
will result in 1 + n test cases:
1 test case from [AutoNSubstituteData]
n test cases from the InvalidInvariant property
These two attributes know nothing about each other, so you can't combine them in the sense that they're aware of each other.
However, when you're implementing a property, you can write whatever code you'd like, including using a Fixture instance, so why not just do this?
public static IEnumerable<string[]> InvalidInvariant
{
get
{
var fixture = new Fixture().Customize(new MyConventions());
// use fixture to yield values...,
// using the occasional hard-coded test value
}
}
Another option is to use derive from the InlineAutoDataAttribute, which would enable you to write your test cases like this:
[Theory]
[MyInlineAutoData("foo")]
[MyInlineAutoData("bar")]
[MyInlineAutoData("baz")]
public void MyTest(string text, string someOtherText, int number, Guid id)
This would cause the first argument (text) to be populated with the constants from the attributes, while the remaining parameters are populated by AutoFixture.
Theoretically, you may also be able to combine the [AutoData] and [PropertyData] attributes using the CompositeDataAttribute, but it may not work the way you'd like.
Finally, you could consider using Exude for true first-class parameterized tests.
I have implemented an AutoPropertyDataAttribute that combines xUnit's PropertyDataAttribute with AutoFixture's AutoDataAttribute. I posted it as an answer here.
In your case you will need to inherit from the attribute in the same way as you would from an AutoDataAttribute, with the exception that you pass a fixture creation function instead of an instance:
public class AutoNSubPropertyDataAttribute : AutoPropertyDataAttribute
{
public AutoNSubPropertyDataAttribute(string propertyName)
: base(propertyName, () => new Fixture().Customize(new AutoNSubstituteCustomization()))
{
}
}
Supposing that there are two repository interface :
interface IFooRepository
{
void Delete(int id);
}
interface IBarRepository
{
void Delete(int id);
}
And an IUnitOfWork interface like :
interface IUnitOfWork : IDisposable
{
void Commit();
void Rollback();
}
what is the best practices of implementing those interface using ServiceStack.ORMLite so that user can use them like
MyFooRepository.Delete(4);
// if an Exception throws here, Bar won't be deleted
MyBarRepository.Delete(7);
Or
using (var uow = CreateUnitOfWork())
{
MyFooRepository.Delete(4);
MyBarRepository.Delete(7);
uow.Commit(); //now they are in an transaction
}
Not sure of your need for Repository + UnitOfWork patterns but I think there are some alternative solutions in ServiceStack + OrmLite that keep your code 'DRY' before you need to introduce any patterns (especially if you're mainly seeking Transaction/Rollback support). Something like below is where I would start.
public class Foo //POCO for data access
{
//Add Attributes for Ormlite
public int Id { get; set; }
}
public class Bar //POCO for data access
{
//Add Attributes for Ormlite
public int Id { get; set; }
}
//your request class which is passed to your service
public class DeleteById
{
public int Id { get; set; }
}
public class FooBarService : MyServiceBase //MyServiceBase has resusable method for handling transactions.
{
public object Post(DeleteById request)
{
DbExec(dbConn =>
{
dbConn.DeleteById<Foo>(request.Id);
dbConn.DeleteById<Bar>(request.Id);
});
return null;
}
}
public class MyServiceBase : Service
{
public IDbConnectionFactory DbFactory { get; set; }
protected void DbExec(Action<IDbConnection> actions)
{
using (var dbConn = DbFactory.OpenDbConnection())
{
using (var trans = dbConn.OpenTransaction())
{
try
{
actions(dbConn);
trans.Commit();
}
catch (Exception ex)
{
trans.Rollback();
throw ex;
}
}
}
}
}
Some references...
https://github.com/ServiceStack/ServiceStack.RedisWebServices - The above code is modified from this example
https://groups.google.com/forum/#!msg/servicestack/1pA41E33QII/R-trWwzYgjEJ - discussion about layers in ServiceStack
http://ayende.com/blog/3955/repository-is-the-new-singleton - Ayende Rahien (NHibernate core contributor) on Repository pattern
I am facing a problem that, for some business processes the sequence of invoking business objects and methods may change frequently. So I came up with something similar to the below:(Sorry somehow I can't post image..., I tried to express them in the below text)
Business Objects:
Object1, Object2
Methods: M1, M2, M3, M4
Processes: P1 (M1 > M2 > M3), P2 (M2 > M3 > if M3 return true then M4 else end)
In this case I am using .NET 3.5. I create some classes to represent processes, which contains those sequences I mentioned. It works. But the problem is I need to compile every time when process changed. It would be much better if I could configure it by some sort of XML.
I have heard about jBPM for Java, Workflow Foundation for .NET but not sure if they fit my needs, or would they be overkill. I even don't what keyword to search in Google. Could anyone advice what technology I should use to solve this issue? Or just point me to some websites or books? Thanks in advance.
A common way to decouple software layers is by using interfaces as stated by Dependency Inversion Principle. In you case you could abstract the process concept using an interface and implement the logic in the implementation of that interface.
when you need change the logic of the process you can create a new implementation of that interface. You can use any IoC framework to inject what implementation you want to use
below is showed just a simple way to do that:
public interface IMethod
{
void M1();
string M2();
void M3();
void M4();
}
public interface IProcess
{
IMethod Method { get; set; }
void P1();
void P2();
}
public class Process : IProcess
{
public IMethod Method
{
get { throw new NotImplementedException(); }
set { throw new NotImplementedException(); }
}
public void P1()
{
Method.M1();
Method.M2();
}
public void P2()
{
if(Method.M2()==string.Empty)
{
Method.M3();
}
}
}
public class AnotherProcess : IProcess
{
public IMethod Method
{
get { throw new NotImplementedException(); }
set { throw new NotImplementedException(); }
}
public void P1()
{
Method.M4();
}
public void P2()
{
Method.M2();
Method.M4();
}
}
public class UseProcess
{
private IProcess _process;
//you can inject the process dependency if you need use a different implementation
public UseProcess(IProcess process)
{
_process = process;
}
public void DoSomething()
{
_process.P1();
}
}
I have been given the task to evaluate codeFirst and possible to use for all our future projects.
The evaluation is based on using codeFirst with an existing database.
Wondering if it's possible to mock the repository using codeFirst 4.1.(no fakes)
The idea is to inject a repository into a service and moq the repository.
I have been looking on the net but I have only found an example using fakes.I dont want to use fakes I want to use moq.
I think my problem is in the architecture of the DAL.(I would like to use unitOfWork etc.. by I need to show a working moq example)
Below is my attempt(Failed miserably) due to lack of knowledge on Code first 4.1.
I have also uploaded a solution just in case somebody is in good mood and would like to change it.
http://cid-9db5ae91a2948485.office.live.com/browse.aspx/Public%20Folder?uc=1
I am open to suggestions and total modification to my Dal.Ideally using Unity etc.. but I will worry about later.
Most importantly I need to be able to mock it. Without ability to use MOQ we will bin the project using EF 4.1
Failed attempt
//CodeFirst.Tests Project
[TestClass]
public class StudentTests
{
[TestMethod]
public void Should_be_able_to_verify_that_get_all_has_been_called()
{
//todo redo test once i can make a simple one work
//Arrange
var repository = new Mock<IStudentRepository>();
var expectedStudents = new List<Student>();
repository.Setup(x => x.GetAll()).Returns(expectedStudents);
//act
var studentService = new StudentService(repository.Object);
studentService.GetAll();
//assert
repository.Verify(x => x.GetAll(), Times.AtLeastOnce());
}
}
//CodeFirst.Common Project
public class Student
{
public int StudentId { get; set; }
public string Name { get; set; }
public string Surname { get; set; }
}
public interface IStudentService
{
IEnumerable<Student> GetAll();
}
//CodeFirst.Service Project
public class StudentService:IStudentService
{
private IStudentRepository _studentRepository;
public StudentService()
{
}
public StudentService(IStudentRepository studentRepository)
{
_studentRepository = studentRepository;
}
public IEnumerable<Student> GetAll()
{
//TODO when mocking using moq this will actually call the db as we need a separate class.
using (var ctx = new SchoolContext("SchoolDB"))
{
_studentRepository = new StudentRepository(ctx);
var students = _studentRepository.GetAll().ToList();
return students;
}
}
}
//CodeFirst.Dal Project
public interface IRepository<T> where T : class
{
T GetOne(Expression<Func<T, bool>> predicate);
IEnumerable<T> GetAll();
IEnumerable<T> Find(Expression<Func<T, bool>> predicate);
void Add(T entity);
void Delete(T entity);
T Single(Func<T, bool> predicate);
T First(Func<T, bool> predicate);
}
public class RepositoryBase<T> : IRepository<T> where T : class
{
private readonly IDbSet<T> _dbSet;
public RepositoryBase(DbContext dbContext)
{
_dbSet = dbContext.Set<T>();
if (_dbSet == null) throw new InvalidOperationException("Cannot create dbSet ");
}
protected virtual IDbSet<T> Query
{
get { return _dbSet; }
}
public T GetOne(Expression<Func<T, bool>> predicate)
{
return Query.Where(predicate).FirstOrDefault();
}
public IEnumerable<T> GetAll()
{
return Query.ToArray();
}
public IEnumerable<T> Find(Expression<Func<T, bool>> predicate)
{
return Query.Where(predicate).ToArray();
}
public void Add(T entity)
{
_dbSet.Add(entity);
}
public void Delete(T entity)
{
_dbSet.Remove(entity);
}
public T Single(Func<T, bool> predicate)
{
return Query.Where(predicate).SingleOrDefault();
}
public T First(Func<T, bool> predicate)
{
return Query.Where(predicate).FirstOrDefault();
}
}
public class SchoolContext:DbContext
{
public SchoolContext(string connectionString):base(connectionString)
{
Database.SetInitializer<SchoolContext>(null);
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
//Not sure why I have to do this.Without this when using integration testing
//as opposed to UnitTests it does not work.
modelBuilder.Entity<Student>().ToTable("Student"); }
public DbSet<Student> Students { get; set; }
}
public interface IStudentRepository:IRepository<Student>
{
}
public class StudentRepository : RepositoryBase<Student>, IStudentRepository
{
public StudentRepository(DbContext dbContext)
: base(dbContext)
{
}
public IEnumerable<Student> GetStudents()
{
return GetAll();
}
}
Again feel free to modify or whatever is needed to help me to get something together.
Thanks a lot for your help
When I started with repository and unit of work patterns I used the implementation similar to this (it is for ObjectContext API but converting it to DbContext API is simple). We used that implementation with MOQ and Unity without any problems. By the time implementations of repository and unit of work have evolve as well as the approach of injecting. Later on we found that whole this approach has serious pitfalls but that was alredy discussed in other questions I referenced here (I highly recommend you to go through these links).
It is very surprising that you are evaluating the EFv4.1 with high emphasis on mocking and unit testing and in the same time you defined service method which is not unit-testable (with mocking) at all. The main problem of you service method is that you are not passing repository/context as dependency and because of that you can't mock it. The only way to test your service and don't use the real repository is using some very advanced approach = replacing mocking and MOQ with detouring (for example Moles framework).
First what you must do is replacing your service code with:
public class StudentService : IStudentService
{
private readonly IStudentRepository _studentRepository;
public StudentService(IStudentRepository studentRepository)
{
_studentRepository = studentRepository;
}
public IEnumerable<Student> GetAll()
{
return _studentRepository.GetAll().ToList();
}
}
Btw. this is absolutely useless code and example of silly layering which doesn't offer any useful functionality. Just wrapping the call to repository only shows that service is not needed at all as well as unit testing this method is not needed. The main point here is integration test for GetAll method.
Anyway if you want to unit thest such method with MOQ you will do:
[TestClass]
public class StudentsServiveTest
{
private Mock<IRespository<Student>> _repo;
[TestInitialize]
public void Init()
{
_repo = new Mock<IRepository<Student>>();
_repo.Setup(r => r.GetAll()).Returns(() => new Student[]
{
new Student { StudentId = 1, Name = "A", Surname = "B" },
new Student { StudentId = 2, Name = "B", Surname = "C" }
});
}
[TestMethod]
public void ShouldReturnAllStudents()
{
var service = new StudentsService(_repo.Object);
var data = service.GetAll();
_repo.Verify(r => r.GetAll(), Times.Once());
Assert.IsNotNull(data);
Assert.AreEqual(2, data.Count);
}
}
The issue from what I can see is that you are throwing away the mock object and newing up a new instance
_studentRepository = new StudentRepository(ctx);
Perhaps add a method on the interface to add the context object and reuse the same instance that was injected in the constructor.
using (var ctx = new SchoolContext("SchoolDB"))
{
_studentRepository.Context = ctx;
var students = _studentRepository.GetAll().ToList();
return students;
}
}