Why does WebSharingAppDemo-CEProviderEndToEnd sample still need a client db connection after scope creation to perform sync - microsoft-sync-framework

I'm researching a way to build an n-tierd sync solution. From the WebSharingAppDemo-CEProviderEndToEnd sample it seems almost feasable however for some reason, the app will only sync if the client has a live SQL db connection. Can some one explain what I'm missing and how to sync without exposing SQL to the internet?
The problem I'm experiencing is that when I provide a Relational sync provider that has an open SQL connection from the client, then it works fine but when I provide a Relational sync provider that has a closed but configured connection string, as in the example, I get an error from the WCF stating that the server did not receive the batch file. So what am I doing wrong?
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = hostName;
builder.IntegratedSecurity = true;
builder.InitialCatalog = "mydbname";
builder.ConnectTimeout = 1;
provider.Connection = new SqlConnection(builder.ToString());
// provider.Connection.Open(); **** un-commenting this causes the code to work**
//create anew scope description and add the appropriate tables to this scope
DbSyncScopeDescription scopeDesc = new DbSyncScopeDescription(SyncUtils.ScopeName);
//class to be used to provision the scope defined above
SqlSyncScopeProvisioning serverConfig = new SqlSyncScopeProvisioning();
....
The error I get occurs in this part of the WCF code:
public SyncSessionStatistics ApplyChanges(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges, object changeData)
{
Log("ProcessChangeBatch: {0}", this.peerProvider.Connection.ConnectionString);
DbSyncContext dataRetriever = changeData as DbSyncContext;
if (dataRetriever != null && dataRetriever.IsDataBatched)
{
string remotePeerId = dataRetriever.MadeWithKnowledge.ReplicaId.ToString();
//Data is batched. The client should have uploaded this file to us prior to calling ApplyChanges.
//So look for it.
//The Id would be the DbSyncContext.BatchFileName which is just the batch file name without the complete path
string localBatchFileName = null;
if (!this.batchIdToFileMapper.TryGetValue(dataRetriever.BatchFileName, out localBatchFileName))
{
//Service has not received this file. Throw exception
throw new FaultException<WebSyncFaultException>(new WebSyncFaultException("No batch file uploaded for id " + dataRetriever.BatchFileName, null));
}
dataRetriever.BatchFileName = localBatchFileName;
}
Any ideas?

For the Batch file not available issue, remove the IsOneWay=true setting from IRelationalSyncContract.UploadBatchFile. When the Batch file size is big, ApplyChanges will be called even before fully completing the previous UploadBatchfile.
// Replace
[OperationContract(IsOneWay = true)]
// with
[OperationContract]
void UploadBatchFile(string batchFileid, byte[] batchFile, string remotePeer1

I suppose it's simply a stupid example. It exposes "some" technique but assumes you have to arrange it in proper order by yourself.
http://msdn.microsoft.com/en-us/library/cc807255.aspx

Related

Using Spanner within Apache Beam Dataflow

I am trying to add a Spanner connection within an Apache Beam ParDo(DoFn). I need to lookup some rows as part of the ParDo. The dataflow creates a number of workers (usually 4 max) and I use the startBundle and finishBundle methods to open and close the spanner connections for the workers lifetime. Then within the processElement method I perform the lookup for each item passing the DatabaseClient and using a singleUseReadOnlyTransaction.
I should add this is running as a dataflow under GCP
Some code to illustrate this.
private static CustomDoFn<String, TransactionImport> processRow = new CustomDoFn<String, TransactionImport>(){
private static final long serialVersionUID = 1L;
private Spanner spanner = null;
private DatabaseClient dbClient = null;
#StartBundle
public void startBundle(StartBundleContext c){
TransactionFileOptions options = c.getPipelineOptions().as(TransactionFileOptions.class);
com.google.cloud.spanner.SpannerOptions spannerOptions = com.google.cloud.spanner.SpannerOptions.newBuilder().build();
spanner = spannerOptions.getService();
String spannerProjectID = options.getSpannerProjectId();
String spannerInstanceID = options.getSpannerInstanceId();
String spannerDatabaseID = options.getSpannerDatabaseId();
DatabaseId db = DatabaseId.of(spannerProjectID, spannerInstanceID, spannerDatabaseID);
dbClient = spanner.getDatabaseClient(db);
}
#FinishBundle
public void finishBundle(FinishBundleContext c){
spanner.close();
}
#ProcessElement
public void processElement(DoFn<String, TransactionImport>.ProcessContext c) throws Exception {
TransactionImport import = new TransactionImport();
Statement statement = Statement.newBuilder("SELECT * FROM Table1 WHERE Name= #Name")
.bind("Name").to( text)
.build();
ResultSet resultSet = dbClient.singleUseReadOnlyTransaction().executeQuery(statement);
// set some value on import dependant on retrieved value
c.output(import);
}
This always results in the dataflow not completing and when I check the log I see:
Processing stuck in step Process Rows for at least 05m00s without outputting or completing in state process
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at com.google.common.util.concurrent.Uninterruptibles.takeUninterruptibly(Uninterruptibles.java:233)
at com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:411)
at com.google.cloud.spanner.SessionPool$Waiter.access$3300(SessionPool.java:399)
at com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:754)
at com.google.cloud.spanner.DatabaseClientImpl.singleUseReadOnlyTransaction(DatabaseClientImpl.java:52)
at com.mycompany.pt.SpannerDataAccess.getBinDetails(SpannerDataAccess.java:197)
at com.mycompany.pt.transactionFiles.TransactionFileDataflow$1.processLine(TransactionFileDataflow.java:411)
at com.mycompany.pt.transactionFiles.TransactionFileDataflow$1.processElement(TransactionFileDataflow.java:336)
at com.mycompany.pt.transactionFiles.TransactionFileDataflow$1$DoFnInvoker.invokeProcessElement(Unknown Source)
`
Does anyone have any experience using Spanner like this within a ParDo?
I'm not a spanner expert, but maybe I can help:
You should use #Setup/#Teardown to connect & disconnect from spanner. #{Start,Finish}Bundle gets called multiple times over the lifetime of a worker. See here for more details: https://beam.apache.org/documentation/execution-model/#bundling-and-persistence
Does your processElement method ever emit an element using
c.output(...)? If not, beam will think your pipeline is stuck

Testing camel-sql route with in-memory database not fetching results

I have written the code using camel-sql which is working fine. Now I have to write test cases for the same. I have used in-memory database H2. I have initialized the database and assigned the datasource to sqlComponent.
// Setup code
#Override
protected JndiRegistry createRegistry() throws Exception {
JndiRegistry jndi = super.createRegistry();
// this is the database we create with some initial data for our unit test
database = new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2).addScript("createTableAndInsert.sql").build();
jndi.bind("myDataSource", database);
return jndi;
}
// Testcase code
#SuppressWarnings("unchecked")
#Test
public void testRoute() throws Exception {
Exchange receivedExchange = template.send("direct:myRoute", ExchangePattern.InOut ,exchange -> {
exchange.getIn().setHeader("ID", new Integer(1));
});
camelContext.start();
MyClass updatedEntity = (MyClass)jdbcTemplate.queryForObject("select * from MY_TABLE where id=?", new Long[] { 1l } ,
new RouteTest.CustomerRowMapper() );
// Here I can get the updatedEntity from jdbcTemplate
assertNotNull(receivedExchange);
assertNotNull(updatedEntity);
}
// Main code
from("direct:myRoute")
.routeId("pollDbRoute")
.transacted()
.to("sql:select * from MY_TABLE msg where msg.id = :#"+ID+"?dataSource=#myDataSource&outputType=SelectOne&outputClass=com.entity.MyClass")
.log(LoggingLevel.INFO,"Polled message from DB");
The problem is, as soon as the test case starts, it is saying
No bean could be found in the registry for: myDataSource of type: javax.sql.DataSource
I looked into camel-SQL component test cases and doing the same thing but the code is not able to find dataSource. Please help. Thanks in advance.
After spending a lot of time on this issue, I identified that H2 database was using JDBCUtils to fetch records and It was throwing ClassNotFoundException. I was getting it nowhere in Camel exception hierarchy because this exception was being suppressed and all I was getting a generic exception message. Here is the exception:
ClassNotFoundException: com.vividsolutions.jts.geom.Geometry
After searching for the issue I found out that It requires one more dependency. So I added it and it resolved the issue.
Issue URL: https://github.com/elastic/elasticsearch/issues/9891
Dependency: https://mvnrepository.com/artifact/com.vividsolutions/jts-core/1.14.0

Sync Framework - Conflict Resolution Triggers Change, Resulting in Unnecessary Downloads

I'm using Sync Framework v2.1 configured in a hub <--> spoke fashion.
Hub: SQL Server 2012 using SqlSyncProvider.
Spokes: LocalDb 2012 using SqlSyncProvider. Each spokes' database begins as a restored backup from the server, after which PostRestoreFixup is executed against it. In investigating this, I've also tried starting with an empty spoke database whose schema and data are created through provisioning and an initial, download-only sync.
Assume two spokes (A & B) and a central hub (let's call it H). They each have one table with one record and they're all in sync.
Spoke A changes the record and syncs, leaving A & H with identical records.
Spoke B changes the same record and syncs, resulting in a conflict with the change made in step #1. B's record is overwritten with H's, and H's record remains as-is. This is the expected/desired result. However, the SyncOperationStatistics returned by the orchestrator suggest changes are made at H. I've tried both SyncDirectionOrder directions, with these results:
- DownloadAndUpload (H's local_update_peer_timestamp and last_change_datetime are updated) -->
* Download changes total: 1
* Download changes applied: 1
* Download changed failed: 0
* Upload changes total: 1
* Upload changes applied: 1
* Upload changed failed: 0
- UploadAndDownload (H's local_update_peer_timestamp is updated)-->
* Upload changes total: 1
* Upload changes applied: 1
* Upload changed failed: 0
* Download changes total: 1
* Download changes applied: 1
* Download changed failed: 0
And, indeed, when Spoke A syncs again the record is downloaded from H, even though H's record hasn't changed. Why?
The problem arising from this is, for example, if Spoke A makes another change to the record between steps #2 and 3, that change will (falsely) be flagged as a conflict and will be overwritten at step #3.
Here's the pared-down code demonstrating the issue or, rather, my question. Note that I've implemented the provider's ApplyChangeFailed handlers such that the server wins, regardless of the SyncDirectionOrder:
private const string ScopeName = "TestScope";
private const string TestTable = "TestTable";
public static SyncOperationStatistics Synchronize(SyncEndpoint local,SyncEndpoint remote, EventHandler<DbSyncProgressEventArgs> eventHandler)
{
using (var localConn = new SqlConnection(local.ConnectionString))
using (var remoteConn = new SqlConnection(remote.ConnectionString))
{
// provision the remote server if necessary
//
var serverProvision = new SqlSyncScopeProvisioning(remoteConn);
if (!serverProvision.ScopeExists(ScopeName))
{
var serverScopeDesc = new DbSyncScopeDescription(ScopeName);
var serverTableDesc = SqlSyncDescriptionBuilder.GetDescriptionForTable(TestTable, remoteConn);
serverScopeDesc.Tables.Add(serverTableDesc);
serverProvision.PopulateFromScopeDescription(serverScopeDesc);
serverProvision.Apply();
}
// provision locally (localDb), if necessary, bringing down the server's scope
//
var clientProvision = new SqlSyncScopeProvisioning(localConn);
if (!clientProvision.ScopeExists(ScopeName))
{
var scopeDesc = SqlSyncDescriptionBuilder.GetDescriptionForScope(ScopeName, remoteConn);
clientProvision.PopulateFromScopeDescription(scopeDesc);
clientProvision.Apply();
}
// create\initialize the sync providers and go for it...
//
using (var localProvider = new SqlSyncProvider(ScopeName, localConn))
using (var remoteProvider = new SqlSyncProvider(ScopeName, remoteConn))
{
localProvider.SyncProviderPosition = SyncProviderPosition.Local;
localProvider.SyncProgress += eventHandler;
localProvider.ApplyChangeFailed += LocalProviderOnApplyChangeFailed;
remoteProvider.SyncProviderPosition = SyncProviderPosition.Remote;
remoteProvider.SyncProgress += eventHandler;
remoteProvider.ApplyChangeFailed += RemoteProviderOnApplyChangeFailed;
var syncOrchestrator = new SyncOrchestrator
{
LocalProvider = localProvider,
RemoteProvider = remoteProvider,
Direction = SyncDirectionOrder.UploadAndDownload // also issue with DownloadAndUpload
};
return syncOrchestrator.Synchronize();
}
}
}
private static void RemoteProviderOnApplyChangeFailed(object sender, DbApplyChangeFailedEventArgs e)
{
// ignore conflicts at the server
//
e.Action = ApplyAction.Continue;
}
private static void LocalProviderOnApplyChangeFailed(object sender, DbApplyChangeFailedEventArgs e)
{
// server wins, force write at each client
//
e.Action = ApplyAction.RetryWithForceWrite;
}
To reiterate, using this code along w/the configuration described at the outset, conflicting rows are, as expected, overwritten on the spoke containing the conflict and the server's version of that row remains as-is (unchanged). However, I'm seeing that each conflict results in an update to the server's xxx_tracking table, specifically the local_update_peer_timestamp and last_change_datetime fields. This, I'm guessing, results in a download to every other spoke even though the server's data hasn't really changed. This seems unnecessary and is, to me, counter-intuitive.

Parsing XML webservice and storing the data for presentation on a windows phone 7 device

I'm working on an app that requires extracting data from an xml web service, then I want to store that data (images+titles+datetime ...) to display it on my app then select an item and navigate to another page that displays more info about this item.
Is there a detailed tutorial that explains the parsing and storing process clearly (with the threads) because I'm gonna need it a lot for my app.Thanks!
I usually use this method, but didn't always get me what i want:
var doc = XDocument.Load(new StringReader(e.Result));
var items = from c in doc.Descendants("item")
select new RSSitem()
{
Title = c.Element("title").Value,
Photo = c.Element("img").Attribute("src").Value,
Description = c.Element("description").Value,
Link = c.Element("link").Value,
};
ListBoxNews.ItemsSource = items;
Sounds like you are in over your head (based on the vague nature of your question). So I'm offering my advise to get up to speed, so you can get started and ask a question that we can help give a definitive answer to.
With WP7 and .NET you shouldn't really have to do much manual parsing of Web Services. You should be able to add a Service Reference and generate a proxy which will handle this for you. This will also generate business objects for the data returned by your service.
Once you have that done, you can look into Windows Phone Navigation which should help you transition between pages in your application.
To consume web services:
String baseUri = “your service URI";
WebClient wc = new WebClient();
public MainPage()
{
InitializeComponent();
wc.DownloadStringCompleted += new DownloadStringCompletedEventHandler(wc_downloadstringcompleted);
// event handler that will handle the ‘downloadstringsompleted’ event
wc.DownloadStringAsync(new Uri(baseUri));
// this method will download your string URI asynchronously
}
void wc_downloadstringcompleted(Object sender, DownloadStringCompletedEventArgs e)
{
// method will get fired after URI download completes
// writes your every code here
}
To parse the data:
using (XmlReader reader = XmlReader.Create(new StringReader(xmlString)))
{
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Element:
break;
case XmlNodeType.Text:
break;
case XmlNodeType.EndElement:
break;
}
}
}
}
To store in isolated storage: http://msdn.microsoft.com/en-us/library/system.io.isolatedstorage.isolatedstoragesettings%28v=vs.95%29.aspx
For navigation:
NavigationService.Navigate(new Uri("/SecondPage.xaml?msg=" + navigationstring, UriKind.Relative));

Sitecore Clear Cache Programmatically

I am trying to publish programmatically in Sitecore. Publishing works fine. But doing so programmatically doesn't clear the sitecore cache. What is the best way to clear the cache programmatically?
I am trying to use the webservice that comes with the staging module. But I am getting a Bad request exception(Exception: The remote server returned an unexpected response: (400) Bad Request.). I tried to increase the service receivetimeout and sendtimeout on the client side config file but that didn't fix the problem. Any pointers would be greatly appreciated?
I am using the following code:
CacheClearService.StagingWebServiceSoapClient client = new CacheClearService.StagingWebServiceSoapClient();
CacheClearService.StagingCredentials credentials = new CacheClearService.StagingCredentials();
credentials.Username = "sitecore\adminuser";
credentials.Password = "***********";
credentials.isEncrypted = false;
bool s = client.ClearCache(true, dt, credentials);
I am using following code to do publish.
Database master = Sitecore.Configuration.Factory.GetDatabase("master");
Database web = Sitecore.Configuration.Factory.GetDatabase("web");
string userName = "default\adminuser";
Sitecore.Security.Accounts.User user = Sitecore.Security.Accounts.User.FromName(userName, true);
user.RuntimeSettings.IsAdministrator = true;
using (new Sitecore.Security.Accounts.UserSwitcher(user))
{
Sitecore.Publishing.PublishOptions options = new Sitecore.Publishing.PublishOptions(master, web,
Sitecore.Publishing.PublishMode.Full, Sitecore.Data.Managers.LanguageManager.DefaultLanguage, DateTime.Now);
options.RootItem = master.Items["/sitecore/content/"];
options.Deep = true;
options.CompareRevisions = true;
options.RepublishAll = true;
options.FromDate = DateTime.Now.AddMonths(-1);
Sitecore.Publishing.Publisher publisher = new Sitecore.Publishing.Publisher(options);
publisher.Publish();
}
In Sitecore 6, the CacheManager class has a static method that will clear all caches. The ClearAll() method is obsolete.
Sitecore.Caching.CacheManager.ClearAllCaches();
Just a quick note, in Sitecore 6.3, that is not needed anymore. Caches are being cleared automatically after a change happens on a remote server.
Also, if you are on previous releases, instead of clearing all caches, you can do partial cache clearing.
There is a free shared source component called Stager that does that.
http://trac.sitecore.net/SitecoreStager
If you need a custom solution, you can simply extract the source code from there.
I got this from Sitecore support. It clears all caches:
Sitecore.Context.Database = this.WebContext.Database;
Sitecore.Context.Database.Engines.TemplateEngine.Reset();
Sitecore.Context.ClientData.RemoveAll();
Sitecore.Caching.CacheManager.ClearAllCaches();
Sitecore.Context.Database = this.ShellContext.Database;
Sitecore.Context.Database.Engines.TemplateEngine.Reset();
Sitecore.Caching.CacheManager.ClearAllCaches();
Sitecore.Context.ClientData.RemoveAll();
Out of the box solution provided by Sitecore to clean caches (ALL of them) is utilized by the following page: http://sitecore_instance_here/sitecore/admin/cache.aspx and code behind looks like the following snippet:
foreach (var cache in Sitecore.Caching.CacheManager.GetAllCaches())
cache.Clear();
Via the SDN:
HtmlCache cache = CacheManager.GetHtmlCache(Context.Site);
if (cache != null) {
cache.Clear();
}