web service pattern that supports lazy loading of all properties - web-services

Question

I am trying to design an endpoint template for a web service. My main requirement is that the caller is able to specify which properties should be populated in the returned result set.
My service returns large lists (up to 1M records) of partial objects as well as individual full objects such as (rough example XML, sorry it's a little verbose)
List:
<items>
<item>
<a>aaa</a>
<b>bbb</b>
</item>
<item>
<a>aaaA</a>
<b>bbbB</b>
</item>
</items>
Detail:
<item>
<a>aaa</a>
<b>bbb</b>
<c>ccc</c>
...
<w>
<x>xxx</x>
<y>yyy</y>
</w>
<z>zzz</z>
</item>
I have considered the following ideas:
Returning the full detail items in the list
Creating a 'list' item type that is shorter
passing a string array of property names that the caller wants to be returned
I am leaning towards the 3rd option but I want something different to that it doesn't support sub objects, I have considered passing the xml schema that you want returned instead of an array.
I would like the API to support lazy loading which is why the 3rd way seems viable as well.
Here's an example of what a function for 3. would look like:
public User GetUser(long ID, string[] properties)
And then the caller could just go:
User.Email = GetUser(User.ID, "Email").Email
Through extensive use of default values and hiding nulls, the returned XML for that would be:
<User>
<ID>123</ID>
<Email>example#example.com</Email>
</User>
Now the problem as mentioned above is trying to make it play nice with things like <w> far above, which itself has sub items as well as the possibility for lists to have sub items.
As I have far too many properties, I cannot have just a ws method for each property.
I am considering option 3. but using an xml schema instead of a string[].. But I can't think of an easy way to define this, I would also like to not have to use String names for properties such as "Email".
The final plan is to have a series of pre-defined schemas that are used commonly and only in advanced cases would we need to actually define the requested properties. But I have no idea of all the systems that will be talking to my API, let alone what properties they might each want (it's not going to be feasible for us to tailor the API for every caller).
Or am I over complicating everything too much?

Answer 1

I found the documentation for the Google APIs on Partial Responses and Partial Updates:
http://googlecode.blogspot.com/2011/07/lightning-fast-performance-tips-for.html
This seems to answer my question.