I have a custom form component that captures lead information and then gives the visitor a download. I have set up an A/B test on the form with different fields and layout to determine which form is the most effective at capturing leads. Basically our A case is the full form and the B case has only the required fields.
Until now both our A and B cases have a score of 0. How do I measure success for a case of the multivariate test? I imagine I need to trigger something in the code-behind that indicates a success or complete state but I don't have anything to go on. What am I missing either in the Page Editor/CMS side or on the code side? Or is this even possible?
Sitecore CMS 6.5.0 and DMS 2.0.0 rev.111230 (6.5.0 Update-3) - Not using WFM
Sounds like you might not have 'deployed' your test? You need to go into the Marketing Center > Test Lab > find your test and then in the Review Tab there will be a 'Deploy' option.
When you're done with your test, you come back here and click 'End Test' in the same place.
Check out the marketing operations cookbook for more detail.
http://sdn.sitecore.net/upload/sitecore6/65/marketing_operations_cookbook_sc65-usletter.pdf
Related
In Ember.js, I currently want to test a UI feature present. Essentially, once a model variable changes, I expect to see a UI element appear (a checkmark). I have tried creating a model within the acceptance test but this unfortunately did not work as I did.
I just wanted to know which function to use to set model variables.
A model typically would involve unit tests, but like you've said you're testing the visual result of something being set on a model. I would recommend an integration test. If you are able to refactor (or maybe this is the case already), the part of the template into a component then you can create an ember test for the component and pass in the model set up perfectly how you would like.
If this test really does depend on the model being setup a specific way I would look at how your application sets up that model to begin with and try to replicate those actions with click and fillIn helpers. Another way is say, your application wants to setup a user but relies on a network request to do this, then you could use pretender.js and fake the response to that request so that the application's inputs are setup from the network in the way you're after.
I would really try to do this an acceptance test though, the composable nature of components allows them to be tested in stricter isolation, these tests will run faster, and you're worrying less about side effects.
I am working on the Sitecore A/B testing and I would like to test my A version with my product section on the top, while in B version, my whole product section is on the bottom?
I have tried to figure it out in the Sitecore Jetstream demo site first (see the diagram in below). I want to move the whole bottom section upwards in my version B, while version A keep as it is. However, I cannot do so but only able to change the component within a section.
Can someone help me on this? Thanks in advance.
There are 2 types of Experience Optimization multivar, A/B tests in Sitecore, the component test and the page test. In your case which kind of test you need depends on the jet stream demo possibilities. With the page test you can test to a new page as long as it uses the same template. The component test is just for one component.
Create a new page where you move the hole section, and use a Sitecore page test.
(If it is not possible to move the section, first change the template to make it possible)
See Create and Run Page Tests
I'm trying to create a A/B component test in Sitecore on a existing condition on a component, example:
A certain banner image is only displayed if a request comes from New York city, for all the other requests the image will not be displayed. Can I now create a A/B component test (new Variant) on top of this condition to display two different images randomly?
When I create the variant on the city condition I no longer have access to the default condition (no image in case of city different of New York), meaning that I cannot switch between conditions in the component menu, from that moment on I can only switch between test variants.
Before creating the variant:
After creating the variant:
Does anyone know if this configuration is possible in Sitecore (8.0 150812), or did anyone had to do something similar, and if so how was it achieved?
Thanks in advance.
The behavior is changed in more recent versions of Sitecore, (tested in Sitecore 8.1 update 2) with this version the edit variations button is directly greyed out when there is personalization rule. so not possible anymore to create personalizate variant and test variants.
Not the way to go, try page testing if you use a personalized component.
I have a test profile in my google analytics account, and ultimately I'd like that profile to include all of the following:
all requests to the test.mydomain.com subdomain
all requests that include "test.mydomain.com" somewhere in the URI
all events that include "test.mydomain.com" somewhere in the category
I had originally tried creating 3 filters on that profile (one for each of the above), but it appears that google-analytics profiles only include things that match ALL filters.
I tried using an advanced filter to combine the above 3 items together into one field, so that I could then just search that field using a regex, but I'm not having much luck... Here's what I have so far:
Filter 1:
which should result in "|test|test" being placed into Custom Field 1 (assuming I had "test.mydomain.com" somewhere in the event category and "test.mydomain.com" as the Hostname)
Filter 2:
which should result in "|test|test|test|" being placed into Custom Field 2 (assuming I had "test.mydomain.com" somewhere in the Request URI).
Filter 3:
which should make it so my profile includes all items where "|test|" appears somewhere in Custom Field 2.
Unfortunately, that's not working, as my test profile is always empty. My "include everything" profile is showing that my requests are coming through and that the do include "test.mydomain.com". I've tried other variations of the above filters, such as setting my advanced-filter regular expressions to ".*", so that they simply concatenate the full Category, Hostname, and Request URI together into Custom Field 2 and then adjusting filter 3 accordingly, but still nothing comes through on that profile. I understand that changes made to profiles can sometimes take a couple of hours to start showing up (https://support.google.com/analytics/answer/1638635?hl=en), but my test filters have been set up for days, so Google should have applied them by now.
Does anyone have an idea what I might be doing wrong? Am I just misunderstanding how the advanced filter works?
Does anyone know of a way to inspect the value of "Custom Field 1" and/or "Custom Field 2" so I can see if those are being built correctly?
You can accomplish this with 4 filters (3 advanced, 1 include). If any of the requirements are satisfied, write "ok" in one of the custom fields. Then do an include on that custom field that matches "ok".
filter 1 - advanced
field a - hostname
extract a - test.mydomain.com
output to - custom field 1
constructor - ok
Use the above concept for the remaining 2 conditions. Then do an include on custom field 1 = ok
I found the issue. The technique that I outlined in my original question, or the technique suggested by Andy should work fine. The problem in my specific example is that google-analytics filters do not have full support for regular expressions. In particular, they do not support the positive lookahead in my example. The other part of my issue was that I sometimes wasn't waiting long enough before testing my profile changes (google can sometimes take a few hours before applying the changes).
To see what regex options google does support, see https://support.google.com/analytics/answer/1034324?hl=en. I originally thought that page was a "getting started with regular expressions" page, but, apparently, that might be all of the regex language that google supports.
If you are unsure if there are parts of your regex that google doesn't support, you can test it on the google-analytics site directly, and in realtime on live data (without waiting for google to apply your profile changes), by navigating to Content > Site Content > All Pages in your Profile, then click on "Advanced" at the top of the "Primary Dimension" table, then change the match type to "Matching RegExp" (see https://support.google.com/analytics/answer/2936903?hl=en). There you can enter in your regex, and if google doesn't support part of your regex, it will tell you immediately when you click "Apply" (not sure why they don't include that regex validation on their profile filter page...). You will also be able to see it immediately filter the content of that table.
I've found absolutely nothing on Google with regard to A/B testing with a client-side framework such as ember.js.
The goal is to serve up adjusted content (different nav items, header phrasing etc.) in order to A/B test our UI/UX. I should note that nothing significant (i.e. sitemap) is changing, just some minor presentational aspects.
There are several possible approaches, namely using different view templates / helper snippets, or serving up a different stylesheet. Both have advantages and challenges, and ideally the same visitor would always be served the same version. Results would be fed through a service like Mixpanel.
I fear I may have to roll my own solution here, but would love to hear any suggestions / pointers.
At their root, most A/B javascript testing frameworks cookie a user as being in the "A" or "B" group, give you a way to ask if a user is "A" or "B" and report "results" back to a service to measure. This can plug into Ember or other client-side frameworks in a way that is fairly orthogonal to the framework.
I would recommend exposing the "A"- or "B"-ness of the user as a property on your user (in Ember, probably your UserController). Then you can use your framework's standard branching or conditionals to render the "A" UI or the "B" UI.
I have actually built a pretty robust A/B Testing tool using Ember for my startup. We are actually thinking of open sourcing it if there was a demand for it. I can let you know the basic idea of how it works for now though.
I have landingPage objects, that can then have a bunch of A/B tests associated with the, When a user comes to the landing page, they are assigned a cookie, and for each A/B test that are assigned either A or B.
I have used two different approaches within jade to handle A/B testing.
For styling type things, I use something like this
and set the .css property in the view to either test-a or test-b
or if it is for text I will do something like this
{{view view.landingPageText}}
and the landingPageText would be set to either the text for A or the test for B.
This thing also dynamically sets up mixpanel, mailchimp, and uses parse.com and node. You can see the code in action here.
http://golf.nextstudioapps.com/